Rounding weirdness with scientific notation formatString | Will Holtz | 2016-03-02 15:17 | |||||||||||||||||||||
Status: Closed | |||||||||||||||||||||||
I have the query: SELECT 0.00055555 AS a, 0.55555 AS b, 555.55 AS c; and associated metadata: <?xml version="1.0" encoding="UTF-8"?> <query xmlns="http://labkey.org/data/xml/query"> <metadata> <tables xmlns="http://labkey.org/data/xml"> <table tableName="TestFormat" tableDbType="NOT_IN_DB"> <columns> <column columnName="a"> <formatString>##0.#E0</formatString> </column> <column columnName="b"> <formatString>##0.#E0</formatString> </column> <column columnName="c"> <formatString>##0.#E0</formatString> </column> </columns> </table> </tables> </metadata> </query> By viewing the query output in the schema browser I get the following: A B C 555.6E-6 555.5E-3 555.5E0 I would have expected the final digit of each of these to get rounded up to 6, but it only happened for A. This is with Postgres v9.5 and Labkey revision 41716. -Will |
|||||||||||||||||||||||
| |||||||||||||||||||||||