GeezerVisonThe World Wide Web, restored to 800 pixels of glory. |
| Home|About|Gaming cabinet|Archives|How it works|Starter sites |
GeezerVison is showing github.com — [fxp] half-even rounding gives 0 instead of ±1 when the truncated quotient is 0 · Issue #7717 · feldera/feldera Closed Copy link Assignees Description
Every half-even path in fxp goes through this function: SQL effect: CREATE TABLE t (s VARCHAR, d DECIMAL(10, 4)); CREATE VIEW v AS SELECT CAST(s AS DECIMAL(5, 0)) AS a, CAST(d AS TIMESTAMP) AS b FROM t; -- With the row ('0.6', 0.0006) in t: -- a = 0, expected 1 (cast_to_SqlDecimal_s parses with Fixed::from_str) -- b = 1970-01-01 00:00:00, expected 1970-01-01 00:00:00.000001 -- (cast_to_Timestamp_SqlDecimal rescales with convert_round_even::<38, 3>) Repro in fxp alone; each assert fails on main 462ef2e, and the comment shows the actual value: assert_eq!(Fixed::<5, 0>::from_str("0.6").unwrap().to_string(), "1"); // "0" assert_eq!(Fixed::<5, 0>::from_str("-0.6").unwrap().to_string(), "-1"); // "0" assert_eq!(Fixed::<5, 1>::from_str("0.06").unwrap().to_string(), "0.1"); // "0" let ms = Fixed::<10, 4>::from_str("0.0006").unwrap(); assert_eq!(ms.convert_round_even::<38, 3>().unwrap().to_string(), "0.001"); // "0" Fix: step by the sign of the value instead of the quotient: Activity
|
| You are visitor 01402520 — GeezerVison does not store the pages you look up. Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required. |