Description

squiddy

opened on Dec 23, 2022

Issue body actions

Feature

PEP 263 specifies that

To aid with platforms such as Windows, which add Unicode BOM marks to the beginning of Unicode files, the UTF-8 signature \xef\xbb\xbf will be interpreted as ‘utf-8’ encoding as well (even if no magic encoding comment is given).

However, RustPython currently fails on files that include a BOM.

$ printf '\xEF\xBB\xBF' > f.py
$ cargo run f.py
SyntaxError: Got unexpected token  at line 1 column 1

^

Python Documentation

Not familiar with CPython source code, but some pointers:

https://github.com/python/cpython/blob/3.11/Lib/tokenize.py#L299 (detect_encoding)
https://github.com/python/cpython/blob/3.11/Lib/test/test_importlib/source/test_source_encoding.py

Activity

1. fanninpm commented on Dec 24, 2022

   

   fanninpm

   on Dec 24, 2022

   Contributor

   More actions

   I remember encountering something like this before, in #3919 (comment).
2. yt2b commented on Dec 24, 2022

   

   yt2b

   on Dec 24, 2022

   Contributor

   More actions

   Checking whether BOM exists is needed.

   But in Rust, UTF-8 BOM(EF BB BF) is recognized as FE FF when converted from bytes to String.

   let utf8_bom = vec![0xef, 0xbb, 0xbf];
   println!("{:?}", String::from_utf8(utf8_bom)); // Ok("\u{feff}")

   Reacted by DimitrisJim
3. squiddy

   added a commit that references this issue on Dec 26, 2022

   Update rust python to handle files with BOM

   c1b9777
4. DimitrisJim commented on Dec 26, 2022

   

   DimitrisJim

   on Dec 26, 2022

   Member

   More actions

   Thanks for reporting this @squiddy! #4359 fixed the erroneous SyntaxError, handling of the mismatch between it and an encoding declaration that differs is not handled (and silently ignored atm) so keep that in mind.
5. DimitrisJim

   closed this as completedon Dec 26, 2022

Issue actions