fix(spreadsheet): stop inventing digits for .xls cells above 2**53 (#3700)

## Description

Bounds the `int()` conversion added by #3616 to the range where a double
is exact, so a legacy `.xls` cell stops inventing digits (#3695).

`_xls_cell` rendered any integral double with `int(value)`. xlrd hands
back an IEEE-754 double, and above 2**53 consecutive integers are not
representable, so the conversion prints the double's *exact* value
rather than the number the sheet held:

```text
typed  123456789012345678   stored  1.2345678901234568e+17   rendered  123456789012345680
```

Two trailing digits, fabricated, delivered to an agent in a form that
reads as an exact identifier. Before #3616 both loaders rendered the
float and simply disagreed about `12.0` vs `12`; after it they agree on
small whole numbers and disagree again -- in the worse direction -- at
the top of the range, which is also where the PR's own goal ("render an
.xls cell the way the .xlsx loader renders it") stops holding.

Closes #3695

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/spreadsheet_ingest.py`: the numeric branch
becomes `if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer()
and abs(value) <= 2**53:`, so values past the exactly-representable
range`return value` and render as the float. The comment records why the
bound exists so it is not "simplified" away again.
- `tests/test_transforms_tabular.py`: adds
`test_load_xls_and_xlsx_agree_above_the_exact_integer_range`, written
against the same xlwt/xlrd fixture style the three existing `.xls` tests
use. It pins both halves of the requirement: `12` still renders without
a decimal point (the #3616 win survives the bound), and
1.2345678901234568e+17 renders as `repr()` of the double rather than an
invented integer.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Type checking was run on the changed module rather than the whole
package, so that line means `mypy --config-file pyproject.toml
headroom/transforms/spreadsheet_ingest.py`, not a full-package pass.

### Test Output

Red, with the fix removed and the new test kept (this is also the state
of `main` for a cell in that range):

```text
$ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q
E   AssertionError: assert '123456789012345680' == '1.2345678901234568e+17'
E
E     - 1.2345678901234568e+17
E     ?  -                ^^^^
E     + 123456789012345680
FAILED tests/test_transforms_tabular.py::test_load_xls_and_xlsx_agree_above_the_exact_integer_range
1 failed, 45 deselected in 4.10s
```

Green, with the fix:

```text
$ python -m pytest tests/test_transforms_tabular.py -q
46 passed in 5.48s

$ uvx ruff@0.16.7 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
All checks passed!
$ uvx ruff@0.16.7 format --check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py
2 files already formatted
$ uvx mypy@1.14.1 --config-file pyproject.toml headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11 (10.0.26200) x64, Python 3.12.13, working tree
at `origin/main` 85fac8c3 plus this one-file change. Optional extras
present for this path only: `xlrd 2.0.2`, `xlwt 1.3.0`, `openpyxl`. No
provider, no API key, no network -- the loader reads a local file.
- Exact command / steps: build a real legacy workbook with `xlwt`
holding one small whole number and one whole-valued double beyond 2**53,
save it as `.xls`, and read it through the shipped `load_spreadsheet()`;
then run the new test twice, once with the bound in place and once with
the pre-fix line restored in the working tree.
- Observed result: the same cell renders `123456789012345680` before
this change and `1.2345678901234568e+17` after it, while `12` renders as
`12` in both states.

```text
$ python -c "xlwt writes [12, 1.2345678901234568e17] into legacy.xls; print(load_spreadsheet(...)['Data'].splitlines()[1])"
post-fix row: 12,1.2345678901234568e+17

$ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q   # fix reverted in the tree
E   AssertionError: assert '123456789012345680' == '1.2345678901234568e+17'
```

The same cell, both states: the pre-fix render carries two invented
digits
(`123456789012345680`), the post-fix render is the double the sheet
holds. The
`12` half is identical in both, which is the point of bounding rather
than
deleting the conversion -- whole numbers at or below 2**53 keep the
#3616 rendering,
and only the range where a double cannot hold an integer reverts to the
float.
- Not tested: a workbook authored in real Excel or LibreOffice (the
fixture is written by `xlwt`, which is also how the three existing
`.xls` tests build theirs), and `.xls` cells in the 2**53..2**1024 range
carrying `#`-formatted number styles, which `_xls_cell` does not see at
all. One measurement worth flagging for whoever picks this up: I first
asserted that the two loaders produce the same *text*, and it failed for
a reason unrelated to this fix -- a value written and read back through
`openpyxl` comes back as `1.234567890123457e+17`, because openpyxl
serializes the literal to 15 significant digits, so the `.xlsx` path
loses a digit on write while the `.xls` path loses one on render. The
test therefore asserts the property that is this fix's subject (no
invented integer; small integers still `12`) instead of cross-format
textual equality, which the loaders do not actually guarantee for values
in that range.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. One comparison in a pure function
inside the spreadsheet loader; no flag service or staged rollout is
involved.
- Minimum rollout channel: n/a -- ordinary release.
- Stable/default behavior changed: yes, narrowly. `.xls` cells holding a
whole-valued double at or above 9007199254740992 now render as
`1.2345678901234568e+17` instead of `123456789012345680`. Below that
bound, and for every non-numeric cell type, output is byte-identical to
today.
- Kill switch / disable path: none needed and none added. Reverting the
single commit restores the previous rendering exactly; there is no new
code path to turn off, and no cache or stored format is involved
(`load_spreadsheet` is a read path).
- Unsafe override required: no.
- Qualification impact: text of affected cells changes, so any test or
prompt-cache fixture that captured `.xls` output for a cell in that
range sees a diff. Existing tests: all 46 in
`tests/test_transforms_tabular.py` pass, including the three `.xls`
tests from #3616 whose expectation (`12`) is deliberately preserved.
- Rollback path: revert the one commit; no migration, no persisted
state, no store rewrite.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` -- it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; the observable artifact is the rendered CSV row, quoted
in the Real Behavior Proof.

## Additional Notes

Two judgement calls, both flagged for review:

- The bound is `abs(value) <= 2**53`. #3695 proposed the exclusive form,
and the review here correctly pointed out that binary64 represents 2**53
and -2**53 exactly while openpyxl loads those as integers, so excluding
them left the two loaders disagreeing at the boundary itself. A stricter
bound (e.g. 1e15, where decimal-style output still lo A stricter bound
(e.g. 1e15, where decimal-style output still looks like a plain integer)
would change more cells than necessary, and a looser one would keep
inventing digits.
- A negative whole number is also rendered as an int when small; `abs()`
keeps that behavior symmetric, which is what the existing tests expect
for positives and what `repr()` would otherwise contradict for
negatives.

Note on CI: `test (3)` on this branch is expected to show the same
pre-existing `test_cost_per_token_resolves_deepseek_v4_flash` failure
described in #3683 (litellm's off-peak pricing window), which #3684
addresses. Nothing in this diff is on that test's path.

---------

Co-authored-by: sxh313 <250161920+sxh313@users.noreply.github.com>
This commit is contained in:
sxh
2026-09-23 23:37:48 -05:00
committed by GitHub
co-authored by sxh313
parent 72217242c1
commit 1a7c5bbf52
2 changed files with 112 additions and 1 deletions
+9 -1
View File
@@ -94,7 +94,15 @@ def _xls_cell(cell: object, datemode: int) -> object:
return datetime(year, month, day, hour, minute, second)
if kind == xlrd.XL_CELL_BOOLEAN:
return bool(value)
if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer():
# int() is only lossless while the double can represent consecutive integers.
# Above the range it renders the double's exact value instead of the number the
# sheet held -- 123456789012345678 is stored as 1.2345678901234568e+17 and
# would print a fabricated ...680 to an agent that has no way to tell it is an
# approximation. Fall through to the float, which is what the .xlsx loader
# renders (#3695). The bound is inclusive: binary64 represents 2**53 and
# -2**53 exactly and openpyxl loads those as integers, so excluding them put
# the two loaders out of step at the boundary itself.
if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer() and abs(value) <= 2**53:
return int(value)
if kind == xlrd.XL_CELL_ERROR:
# openpyxl with data_only=True gives the text Excel shows, e.g. #DIV/0!
+103
View File
@@ -474,6 +474,109 @@ def test_load_xls_renders_a_time_only_cell_as_a_time(tmp_path) -> None:
assert load_spreadsheet(xls_path)["Data"].splitlines()[1] == "12:00:00"
def test_load_xls_and_xlsx_agree_above_the_exact_integer_range(tmp_path) -> None:
"""A double cannot hold consecutive integers past 2**53.
``_xls_cell`` converted any integral double with ``int()``, so a sheet
holding 123456789012345678 rendered the double's exact value, 123456789012345680
-- two fabricated digits presented to an agent as a precise identifier. The
.xlsx loader has always rendered the float, which at least says
"approximate", so bounding the conversion to the exactly-representable range
keeps the ``12.0 -> 12`` fix from #3616 and restores agreement (#3695).
"""
xlwt = pytest.importorskip("xlwt")
pytest.importorskip("xlrd")
openpyxl = pytest.importorskip("openpyxl")
from headroom.transforms.spreadsheet_ingest import load_spreadsheet
small, big = 12, 1.2345678901234568e17
xls_book = xlwt.Workbook()
xls_sheet = xls_book.add_sheet("Data")
xls_sheet.write(0, 0, "Small")
xls_sheet.write(0, 1, "Big")
xls_sheet.write(1, 0, small)
xls_sheet.write(1, 1, big)
xls_path = tmp_path / "legacy.xls"
xls_book.save(xls_path)
# openpyxl is the reference the .xls path is written against, so the expected
# rendering is the float repr it yields for the same value.
openpyxl_wb = openpyxl.Workbook()
openpyxl_sheet = openpyxl_wb.active
openpyxl_sheet.title = "Data"
openpyxl_sheet.append(["Small", "Big"])
openpyxl_sheet.append([small, big])
openpyxl_wb.save(tmp_path / "modern.xlsx")
xls_row = load_spreadsheet(xls_path)["Data"].splitlines()[1]
xlsx_row = load_spreadsheet(tmp_path / "modern.xlsx")["Data"].splitlines()[1]
small_field, big_field = xls_row.split(",")
# The #3616 win has to survive the bound: a small whole number is still an int.
assert small_field == "12"
# And the fabricated integer must be gone: the cell is rendered as the double
# it is, which reads as an approximation instead of an exact identifier.
assert big_field == repr(big)
assert big_field != str(int(big))
# Parity, asserted against the other loader rather than against my own
# expectation. Small values agree verbatim; above the range openpyxl writes a
# double with only 15 significant digits, so that side loses a digit on its
# own and the rows cannot be string-equal. The promise this fix makes is
# about magnitude: the two loaders agree to well within one unit in the last
# place of the stored value (16 here), and neither hands the agent the
# exact-looking decimal of the typed number.
xlsx_small, xlsx_big = xlsx_row.split(",")
assert small_field == xlsx_small == "12"
assert abs(float(big_field) - float(xlsx_big)) <= 16
def test_load_xls_and_xlsx_agree_at_the_exact_integer_boundary(tmp_path) -> None:
"""2**53 and -2**53 are exactly representable and openpyxl loads them as
integers, so the .xls path must convert them too - the bound is inclusive.
One step outside, the double cannot hold the value; what matters is that no
digits are invented, and the two loaders then differ only in the trailing
``.0`` that marks a value as approximate.
"""
xlwt = pytest.importorskip("xlwt")
pytest.importorskip("xlrd")
openpyxl = pytest.importorskip("openpyxl")
from headroom.transforms.spreadsheet_ingest import load_spreadsheet
boundary = 2**53
values = [boundary, -boundary, boundary - 2, boundary + 2]
xls_book = xlwt.Workbook()
xls_sheet = xls_book.add_sheet("Data")
xls_sheet.write(0, 0, "Value")
for row, value in enumerate(values, start=1):
xls_sheet.write(row, 0, float(value))
xls_path = tmp_path / "boundary.xls"
xls_book.save(xls_path)
xlsx_wb = openpyxl.Workbook()
xlsx_sheet = xlsx_wb.active
xlsx_sheet.title = "Data"
xlsx_sheet.append(["Value"])
for value in values:
xlsx_sheet.append([int(value)])
xlsx_path = tmp_path / "boundary.xlsx"
xlsx_wb.save(xlsx_path)
xls = [line.split(",")[0] for line in load_spreadsheet(xls_path)["Data"].splitlines()[1:]]
xlsx = [line.split(",")[0] for line in load_spreadsheet(xlsx_path)["Data"].splitlines()[1:]]
# Inside the range (and exactly on it) the two loaders agree verbatim.
assert xls[0] == xlsx[0] == "9007199254740992"
assert xls[1] == xlsx[1] == "-9007199254740992"
assert xls[2] == xlsx[2] == "9007199254740990"
# Above it the .xls side keeps the float marker, and the digits are the same.
assert xls[3].removesuffix(".0") == xlsx[3] == "9007199254740994"
assert all(not field.endswith(".0") or float(field) == int(float(field)) for field in xls)
def test_load_xls_and_xlsx_agree_on_the_same_values(tmp_path) -> None:
"""The reference: openpyxl is what the .xls path is matching."""
xlwt = pytest.importorskip("xlwt")