Description

illia-v

opened on May 27, 2022

Issue body actions

Bug report

A value of the Content-length header returned by urllib.request.FileHandler.open_local_file may mismatch the length of data on Linux.

This happens when a file from a special file system (e.g., procfs or sysfs) is requested.

open_local_file relies on st_size; st_size is equal to zero for pseudo files on Linux.

cpython/Lib/urllib/request.py

Lines 1506 to 1511 in 8a0d9a6

	size = stats.st_size
	modified = email.utils.formatdate(stats.st_mtime, usegmt=True)
	mtype = mimetypes.guess_type(filename)[0]
	headers = email.message_from_string(
	'Content-type: %s\nContent-length: %d\nLast-modified: %s\n' %
	(mtype or 'text/plain', size, modified))

Example

>>> import urllib.request
>>> url = "file:///proc/cpuinfo"
>>> handler = urllib.request.FileHandler()
>>> response = handler.file_open(urllib.request.Request(url))
>>> data = response.read()
>>> headers = response.info()
>>> assert int(headers["Content-length"]) == len(data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError
>>> print(headers["Content-length"])
0
>>> print(len(data))
18294

Your environment

• CPython versions tested on: 3.12.0 alpha 0
• Operating system and architecture: Linux

Activity

1. illia-v

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on May 27, 2022
2. illia-v

   changed the title [-]`urllib.request.FileHandler.open_local_file` sets `Content-length: 0` for non-empty pseudo files[/-] [+]urllib sets wrong `Content-Length` for pseudo files on Linux[/+] on May 27, 2022
3. AA-Turner

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   on May 27, 2022
4. illia-v

   added a commit that references this issue on May 27, 2022

   pythongh-93296: test that urllib sets wrong `Content-Length` for pseu…

   1b21033
5. carlbordum commented on Jun 1, 2022

   

   carlbordum

   on Jun 1, 2022

   Contributor

   More actions

   I'm not sure this really is a bug. What would you expect?
6. illia-v commented on Jun 1, 2022

   

   illia-v

   on Jun 1, 2022

   ContributorAuthor

   More actions

   I'm not sure this really is a bug. What would you expect?

   I would expect a value of Content-Length to be equal to the size of the HTTP message body regardless of the peculiarities of a file in this case.
7. serhiy-storchaka commented on Feb 21, 2024

   

   serhiy-storchaka

   on Feb 21, 2024

   Member

   More actions

   curl returns the same result:

   $ curl --dump-header /dev/stdout file:///proc/cpuinfo
   Content-Length: 0
   Accept-ranges: bytes
   Last-Modified: Fri, 16 Feb 2024 14:54:57 GMT
   
   processor       : 0
   ...
   

   Header Content-Length: 0 followed by non-empty body.
8. encukou commented on Mar 26, 2024

   

   encukou

   on Mar 26, 2024

   Member

   More actions

   AFAIK, a similar discrepancy can happen if the file changes while it's being prepared/sent.

   urllib asks the system for size, mtime and contents, and gives them to you.
   I don't think it's urllib's job to second-guess the system.

   The situation is the same as when a HTTP server sends a mismatched Content-Length, except here it's the kernel, not a server. It's up to you to handle it.

   IMO, this is not an issue. If there are no objections, I'll close in a month.
9. illia-v commented on Mar 27, 2024

   

   illia-v

   on Mar 27, 2024

   ContributorAuthor

   More actions

   curl returns the same result:

   $ curl --dump-header /dev/stdout file:///proc/cpuinfo
   Content-Length: 0
   Accept-ranges: bytes
   Last-Modified: Fri, 16 Feb 2024 14:54:57 GMT
   
   processor       : 0
   ...
   

   Header Content-Length: 0 followed by non-empty body.

   This is interesting, it used to return no content, but that was changed in 2016 curl/curl#681 (comment)

   The situation is the same as when a HTTP server sends a mismatched Content-Length, except here it's the kernel, not a server. It's up to you to handle it.

   IMO, this is not an issue. If there are no objections, I'll close in a month.

   As a user I'd expect a server to handle known cases, but I don't insist on keeping the issue open.
10. serhiy-storchaka

    closed this as not plannedon Mar 27, 2024

Issue actions