GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — UBSan: Calling a function through pointer to incorrect function type is undefined behavior · Issue #111178 · python/cpython

UBSan: Calling a function through pointer to incorrect function type is undefined behavior #111178

New issue
Copy link
New issue
Copy link
Closed
Closed
UBSan: Calling a function through pointer to incorrect function type is undefined behavior#111178
Copy link
Labels
topic-C-APItype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

Description

@chrstphrchvz
chrstphrchvz
opened on Oct 22, 2023
Issue body actions

Bug report

Bug description:

UBSan (UndefinedBehaviorSanitizer) in LLVM.org Clang 17 makes -fsanitize=function available for C; previously, it was only for C++. (So it may also be made available in future Apple Xcode clang and GCC.) By default, it is implied by -fsanitize=undefined (which is what ./configure --with-undefined-behavior-sanitizer uses), but it can be disabled using -fno-sanitize=function.

For a project such as CPython, which has long relied on function pointers for callbacks, yet seems to have only required that callbacks behave as expected under typical ABI calling conventions, rather than more strictly be declared/defined as a type compatible with the function pointer they will be called as, this leads to numerous errors from UBSan.

Examples when starting Python REPL:
% ./python.exe           
Objects/object.c:2731:5: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:2731:5 in 
Objects/object.c:878:16: runtime error: call to function long_hash through pointer to incorrect function type 'long (*)(struct _object *)'
longobject.c:3295: note: long_hash defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:878:16 in 
Include/internal/pycore_object.h:365:43: runtime error: call to function type_is_gc through pointer to incorrect function type 'int (*)(struct _object *)'
typeobject.c:5347: note: type_is_gc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_object.h:365:43 in 
Objects/abstract.c:157:26: runtime error: call to function dict_subscript through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
dictobject.c:2511: note: dict_subscript defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:157:26 in 
Objects/abstract.c:2954:14: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2954:14 in 
Objects/abstract.c:236:19: runtime error: call to function dict_ass_sub through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
dictobject.c:2546: note: dict_ass_sub defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:236:19 in 
Objects/call.c:242:18: runtime error: call to function type_call through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:1647: note: type_call defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/call.c:242:18 in 
Objects/typeobject.c:10309:24: runtime error: call to function classmethod_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:94: note: classmethod_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:10309:24 in 
Modules/gcmodule.c:493:16: runtime error: call to function list_traverse through pointer to incorrect function type 'int (*)(struct _object *, int (*)(struct _object *, void *), void *)'
listobject.c:2704: note: list_traverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:493:16 in 
Modules/gcmodule.c:605:20: runtime error: call to function list_traverse through pointer to incorrect function type 'int (*)(struct _object *, int (*)(struct _object *, void *), void *)'
listobject.c:2704: note: list_traverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:605:20 in 
Objects/dictobject.c:3569:17: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (*)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:3569:17 in 
Objects/descrobject.c:694:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (*)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:694:5 in 
[…]
Objects/typeobject.c:4892:19: runtime error: call to function classmethod_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:94: note: classmethod_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4892:19 in 
Python/generated_cases.c.h:3859:17: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3859:17 in 
Objects/object.c:1442:19: runtime error: call to function member_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:160: note: member_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1442:19 in 
Objects/object.c:1503:15: runtime error: call to function method_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1503:15 in 
Python/generated_cases.c.h:3857:13: runtime error: call to function meth_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
methodobject.c:160: note: meth_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3857:13 in 
Objects/typeobject.c:2212:19: runtime error: call to function method_get through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:2212:19 in 
Objects/descrobject.c:188:16: runtime error: call to function func_get_name through pointer to incorrect function type 'struct _object *(*)(struct _object *, void *)'
funcobject.c:582: note: func_get_name defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:188:16 in 
Python/generated_cases.c.h:3248:20: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3248:20 in 
Objects/object.c:1564:19: runtime error: call to function member_set through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
descrobject.c:227: note: member_set defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1564:19 in 
Objects/descrobject.c:241:16: runtime error: call to function func_set_name through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, void *)'
funcobject.c:588: note: func_set_name defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:241:16 in 
Python/generated_cases.c.h:3364:21: runtime error: call to function tupleiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:984: note: tupleiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3364:21 in 
Python/ceval.c:591:5: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:591:5 in 
Objects/object.c:1031:18: runtime error: call to function _Py_module_getattro through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
moduleobject.c:877: note: _Py_module_getattro defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1031:18 in 
Objects/abstract.c:2895:25: runtime error: call to function dictitems_iter through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
dictobject.c:5180: note: dictitems_iter defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2895:25 in 
Python/generated_cases.c.h:3193:13: runtime error: call to function dictview_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:4587: note: dictview_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3193:13 in 
Python/ceval.c:1929:13: runtime error: call to function tupleiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:984: note: tupleiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1929:13 in 
Objects/abstract.c:2334:19: runtime error: call to function tuplecontains through pointer to incorrect function type 'int (*)(struct _object *, struct _object *)'
tupleobject.c:354: note: tuplecontains defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:2334:19 in 
Objects/typeobject.c:1698:19: runtime error: call to function AttributeError_init through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
exceptions.c:2279: note: AttributeError_init defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:1698:19 in 
Python/generated_cases.c.h:1351:13: runtime error: call to function AttributeError_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
exceptions.c:2315: note: AttributeError_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1351:13 in 
Python/generated_cases.c.h:623:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:623:13 in 
Python/generated_cases.c.h:749:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:749:13 in 
Python/generated_cases.c.h:3809:17: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3809:17 in 
Objects/descrobject.c:393:24: runtime error: call to function dict_pop through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *const *, long)'
dictobject.c.h:138: note: dict_pop defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:393:24 in 
Python/generated_cases.c.h:3260:17: runtime error: call to function dictiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:4047: note: dictiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3260:17 in 
Objects/object.c:1699:15: runtime error: call to function long_bool through pointer to incorrect function type 'int (*)(struct _object *)'
longobject.c:4890: note: long_bool defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1699:15 in 
Python/generated_cases.c.h:4816:13: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4816:13 in 
Python/generated_cases.c.h:3896:17: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3896:17 in 
Python/generated_cases.c.h:4625:19: runtime error: call to function dict_pop through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *const *, long)'
dictobject.c.h:138: note: dict_pop defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4625:19 in 
Objects/descrobject.c:467:24: runtime error: call to function list_append through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
listobject.c:842: note: list_append defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:467:24 in 
Include/internal/pycore_object.h:142:9: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (*)(struct _object *)'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_object.h:142:9 in 
Objects/call.c:361:18: runtime error: call to function type_call through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:1647: note: type_call defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/call.c:361:18 in 
Python/generated_cases.c.h:2931:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2931:13 in 
Python/generated_cases.c.h:3362:25: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3362:25 in 
Python/generated_cases.c.h:928:13: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:928:13 in 
Python/ceval.c:1604:5: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1604:5 in 
Python/bltinmodule.c:378:16: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object *(*)(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:378:16 in 
Objects/object.c:1702:15: runtime error: call to function list_length through pointer to incorrect function type 'long (*)(struct _object *)'
listobject.c:438: note: list_length defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1702:15 in 
Python/generated_cases.c.h:3519:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3519:13 in 
Objects/methodobject.c:540:18: runtime error: call to function rlock_acquire through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *, struct _object *)'
_threadmodule.c:314: note: rlock_acquire defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:540:18 in 
Python/generated_cases.c.h:2767:13: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2767:13 in 
Objects/methodobject.c:551:18: runtime error: call to function rlock_release through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
_threadmodule.c:364: note: rlock_release defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:551:18 in 
Objects/typeobject.c:2096:5: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:2096:5 in 
Python/generated_cases.c.h:130:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:130:13 in 
Objects/object.c:1181:15: runtime error: call to function type_setattro through pointer to incorrect function type 'int (*)(struct _object *, struct _object *, struct _object *)'
typeobject.c:4938: note: type_setattro defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:1181:15 in 
Python/generated_cases.c.h:1510:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (*)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1510:13 in 
Python/generated_cases.c.h:164:13: runtime error: call to function listiter_dealloc through pointer to incorrect function type 'void (*)(struct _object *)'
listobject.c:3222: note: listiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:164:13 in 
[…]

Modules/gcmodule.c:1033:24: runtime error: call to function _list_clear through pointer to incorrect function type 'int (*)(struct _object )'
listobject.c:597: note: _list_clear defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/gcmodule.c:1033:24 in
Objects/abstract.c:62:26: runtime error: call to function list_length through pointer to incorrect function type 'long (
)(struct _object )'
listobject.c:438: note: list_length defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:62:26 in
Objects/abstract.c:270:19: runtime error: call to function dict_ass_sub through pointer to incorrect function type 'int (
)(struct _object *, struct _object *, struct _object *)'
dictobject.c:2546: note: dict_ass_sub defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:270:19 in
Python/bltinmodule.c:329:16: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object ()(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:329:16 in
Objects/abstract.c:1141:18: runtime error: call to function tupleconcat through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
tupleobject.c:443: note: tupleconcat defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:1141:18 in
Objects/abstract.c:440:15: runtime error: call to function bytes_buffer_getbuffer through pointer to incorrect function type 'int (
)(struct _object *, Py_buffer *, int)'
bytesobject.c:1663: note: bytes_buffer_getbuffer defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:440:15 in
Objects/methodobject.c:441:24: runtime error: call to function int_from_bytes through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *const *, long, struct _object )'
longobject.c.h:396: note: int_from_bytes defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:441:24 in
[…]
Python/generated_cases.c.h:4814:13: runtime error: call to function method_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
classobject.c:236: note: method_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4814:13 in
Python/generated_cases.c.h:4815:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4815:13 in
Python/generated_cases.c.h:1529:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1529:13 in
Python/generated_cases.c.h:648:17: runtime error: call to function slice_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
sliceobject.c:359: note: slice_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:648:17 in
Objects/abstract.c:953:13: runtime error: call to function long_add through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
longobject.c:3473: note: long_add defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:953:13 in
Python/generated_cases.c.h:3322:21: runtime error: call to function listiter_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
listobject.c:3222: note: listiter_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3322:21 in
Python/generated_cases.c.h:2821:13: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (
)(struct _object *)'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2821:13 in
Objects/listobject.c:946:26: runtime error: call to function gen_iternext through pointer to incorrect function type 'struct _object ()(struct _object *)'
genobject.c:609: note: gen_iternext defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/listobject.c:946:26 in
Python/generated_cases.c.h:4503:19: runtime error: call to function list_extend through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object )'
listobject.c:861: note: list_extend defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4503:19 in
Python/generated_cases.c.h:4507:13: runtime error: call to function gen_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
genobject.c:128: note: gen_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4507:13 in
Python/generated_cases.c.h:2378:17: runtime error: call to function structseq_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
structseq.c:119: note: structseq_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2378:17 in
Python/ceval.c:1605:5: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/ceval.c:1605:5 in
Python/generated_cases.c.h:1548:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object *)'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1548:13 in
Objects/descrobject.c:439:24: runtime error: call to function _io_FileIO_isatty through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *)'
fileio.c.h:532: note: _io_FileIO_isatty defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:439:24 in
Objects/methodobject.c:484:24: runtime error: call to function _io_FileIO_readall through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *)'
fileio.c.h:284: note: _io_FileIO_readall defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/methodobject.c:484:24 in
Objects/descrobject.c:372:24: runtime error: call to function _io_FileIO_close through pointer to incorrect function type 'struct _object ()(struct _object *, struct _typeobject *, struct _object *const *, unsigned long, struct _object *)'
fileio.c.h:29: note: _io_FileIO_close defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/descrobject.c:372:24 in
Objects/object.c:783:15: runtime error: call to function bytes_richcompare through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object , int)'
bytesobject.c:1524: note: bytes_richcompare defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/object.c:783:15 in
Python/generated_cases.c.h:650:13: runtime error: call to function memory_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
memoryobject.c:1141: note: memory_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:650:13 in
Objects/abstract.c:804:9: runtime error: call to function memory_releasebuf through pointer to incorrect function type 'void (
)(struct _object *, Py_buffer *)'
memoryobject.c:1593: note: memory_releasebuf defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:804:9 in
Objects/abstract.c:1950:25: runtime error: call to function list_item through pointer to incorrect function type 'struct _object ()(struct _object , long)'
listobject.c:460: note: list_item defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/abstract.c:1950:25 in
Modules/timemodule.c:2087:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (
)(struct _object *, void )'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/timemodule.c:2087:5 in
Python/generated_cases.c.h:2660:21: runtime error: call to function set_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
setobject.c:489: note: set_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2660:21 in
Python/generated_cases.c.h:4440:13: runtime error: call to function tupledealloc through pointer to incorrect function type 'void (
)(struct _object )'
tupleobject.c:187: note: tupledealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4440:13 in
Python/generated_cases.c.h:2822:13: runtime error: call to function PyObject_Free through pointer to incorrect function type 'void (
)(struct _object )'
obmalloc.c:829: note: PyObject_Free defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2822:13 in
Python/generated_cases.c.h:2157:13: runtime error: call to function dict_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
dictobject.c:2373: note: dict_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2157:13 in
Objects/typeobject.c:4872:19: runtime error: call to function getset_get through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *, struct _object )'
descrobject.c:180: note: getset_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4872:19 in
Python/generated_cases.c.h:4730:17: runtime error: call to function func_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
funcobject.c:913: note: func_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:4730:17 in
Objects/typeobject.c:4905:15: runtime error: call to function method_get through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *, struct _object )'
descrobject.c:136: note: method_get defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:4905:15 in
Objects/typeobject.c:1863:16: runtime error: call to function tupletraverse through pointer to incorrect function type 'int (
)(struct _object , int ()(struct _object *, void *), void *)'
tupleobject.c:606: note: tupletraverse defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/typeobject.c:1863:16 in
[…]
Include/internal/pycore_call.h:187:11: runtime error: call to function range_vectorcall through pointer to incorrect function type 'struct _object ()(struct _object *, struct _object *const *, unsigned long, struct _object )'
rangeobject.c:148: note: range_vectorcall defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/internal/pycore_call.h:187:11 in
Python/generated_cases.c.h:2878:13: runtime error: call to function mappingproxy_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
descrobject.c:1160: note: mappingproxy_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2878:13 in
Python/generated_cases.c.h:3320:25: runtime error: call to function list_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
listobject.c:347: note: list_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:3320:25 in
Python/generated_cases.c.h:2766:13: runtime error: call to function set_dealloc through pointer to incorrect function type 'void (
)(struct _object )'
setobject.c:489: note: set_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:2766:13 in
Modules/_abc.c:52:5: runtime error: call to function visit_reachable through pointer to incorrect function type 'int (
)(struct _object *, void *)'
gcmodule.c:502: note: visit_reachable defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/_abc.c:52:5 in
Python/bltinmodule.c:1408:25: runtime error: call to function tupleiter_next through pointer to incorrect function type 'struct _object ()(struct _object )'
tupleobject.c:999: note: tupleiter_next defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/bltinmodule.c:1408:25 in
Python/generated_cases.c.h:673:17: runtime error: call to function slice_dealloc through pointer to incorrect function type 'void (
)(struct _object *)'
sliceobject.c:359: note: slice_dealloc defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:673:17 in


(Omitting remaining errors due to GitHub comment length limit.)


Python 3.13.0a1+ (heads/patch-103194-dirty:0a6e69f9a2, Oct 21 2023, 14:45:09) [Clang 17.0.3 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Example workaround for the first error and likely many others, where instead of casting functions to incompatible pointers, the functions use compatible signatures and cast their parameter(s):

--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -343,8 +343,9 @@ PyList_Append(PyObject *op, PyObject *newitem)
 /* Methods */
 
 static void
-list_dealloc(PyListObject *op)
+list_dealloc(PyObject *self)
 {
+    PyListObject *op = (PyListObject *)self;
     Py_ssize_t i;
     PyObject_GC_UnTrack(op);
     Py_TRASHCAN_BEGIN(op, list_dealloc)
@@ -3104,7 +3105,7 @@ PyTypeObject PyList_Type = {
     "list",
     sizeof(PyListObject),
     0,
-    (destructor)list_dealloc,                   /* tp_dealloc */
+    list_dealloc,                               /* tp_dealloc */
     0,                                          /* tp_vectorcall_offset */
     0,                                          /* tp_getattr */
     0,                                          /* tp_setattr */

In other cases, it may be less disruptive to introduce a wrapper function with the correct signature:

@@ -615,6 +617,13 @@ _list_clear(PyListObject *a)
     return 0;
 }
 
+static int
+_list_clear_wrap(PyObject *self)
+{
+    PyListObject *a = (PyListObject *)self;
+    return _list_clear(a);
+}
+
 /* a[ilow:ihigh] = v if v != NULL.
  * del a[ilow:ihigh] if v == NULL.
  *
@@ -3123,7 +3133,7 @@ PyTypeObject PyList_Type = {
         _Py_TPFLAGS_MATCH_SELF | Py_TPFLAGS_SEQUENCE,  /* tp_flags */
     list___init____doc__,                       /* tp_doc */
     (traverseproc)list_traverse,                /* tp_traverse */
-    (inquiry)_list_clear,                       /* tp_clear */
+    _list_clear_wrap,                           /* tp_clear */
     list_richcompare,                           /* tp_richcompare */
     0,                                          /* tp_weaklistoffset */
     list_iter,                                  /* tp_iter */

Likely instances of this can be found at compile time using e.g. -Wcast-function-type (although this emits false positives for when the function pointer is cast back to the correct type before called, and this warning is suppressed by intermediate casts through (void *)):

Objects/listobject.c:3162:5: warning: cast from 'void (*)(PyListObject *)' to 'destructor' (aka 'void (*)(struct _object *)') converts to incompatible function type [-Wcast-function-type-strict]
 3162 |     (destructor)list_dealloc,                   /* tp_dealloc */
      |     ^~~~~~~~~~~~~~~~~~~~~~~~

I would be interested in combing through and replacing similar instances. But I would not be surprised if sooner or later I encounter an instance that is won’t-fix because it involves a stable API, or if I am told that this problem should be ignored because fixing it is too disruptive or requires disproportionate review effort. I am not aware how immediate any danger is from optimizing compilers exploiting this type of undefined behavior.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS


Written by @picnixz:

See #111178 (comment) for the remaining files to fix.

For detecting the UBSan failures, contributors may configure Python with:

./configure                                                 \
    --with-pydebug                                          \
    --prefix="$(pwd)/build"                                 \
    CC=clang LD=clang                                       \
    CFLAGS="-fsanitize=function     -fsanitize-recover"     \
    LDFLAGS="-fsanitize=function    -fsanitize-recover"

The complete list of failures can be retrieved as follows:

PAT='runtime error: call to function (\w+) through pointer' && \
make -j12 2>&1 >/dev/null | \
    grep -E "$PAT" | \
    sed -r "s#^(.+): $PAT.+#\1@\2#g" | \
    sort -k1,2 -t@ -u | \
    awk 'BEGIN {
            PROCINFO["sorted_in"]="@ind_num_asc";
            FS=SUBSEP="@"
        } {
            A[$1][length(A[$1])+1]=$2
        } END {
            for (m in A) {
                for (i in A[m]) {
                    if (i == 1) printf "%s\n", m;
                    print "#", A[m][i]
                }
                print ""
            }
        }' | \
    sed -r "s#"$(pwd)"/?(\./)?(.+)#\2#g"

Note that different builds should be configured in order to hunt all UBSan failures (e.g., --with-trace-refs or --disable-gil to expose conditional compiled code guarded by macros).


Note by @encukou:

We use the macro _Py_NO_SANITIZE_UNDEFINED to disable the UB sanitizer in some hard-to-fix cases, so that we can get a stable, regression-monitoring checker sooner. This issue should only be closed after the macro is removed.


Linked PRs

  • gh-111178: Define visitproc callback functions properly and remove unnecessary casts in gcmodule.c #112687
  • gh-111178: Make slot functions in typeobject.c have compatible types #112752
  • gh-111178: Avoid calling functions from incompatible pointer types #112782
  • gh-111178: Docs: fix traverseproc, inquiry, and destructor parameters in slot typedefs table #112742
  • [3.12] gh-111178: Docs: fix traverseproc, inquiry, and destructor parameters in slot typedefs table (GH-112742) #112792
  • [3.11] gh-111178: Docs: fix traverseproc, inquiry, and destructor parameters in slot typedefs table (GH-112742) #112793
  • gh-111178: Avoid calling functions from incompatible pointer types in listobject.c #112820
  • gh-111178: Avoid calling functions from incompatible pointer types in descrobject.c #112861
  • gh-111178: Avoid calling functions from incompatible pointer types in memoryobject.c #112863
  • gh-111178: Avoid calling functions from incompatible pointer types in dictobject.c #112892
  • gh-111178: Avoid calling functions from incompatible pointer types in _tkinter.c #112893
  • gh-111178: Avoid calling functions from incompatible pointer types in longobject.c #122972
  • gh-111178: fix some USAN failures #123004
  • gh-111178: fix USAN failures for partialobject #124733
  • gh-111178: add UBSan suppression file #124763
  • gh-111178: Fix function signatures in tupleobject.c #124804
  • gh-111178: Fix function signatures in bytesobject.c #124806
  • gh-111178: Fix function signatures in setobject.c #124888
  • gh-111178: Fix function signatures in longobject.c #124895
  • gh-111178: Fix locale_free() signature #124896
  • gh-111178: Fix function signatures in moduleobject.c #124900
  • gh-111178: Fix function signatures in methodobject.c #124902
  • gh-111178: Fix function signatures in weakrefobject.c #124903
  • gh-111178: Fix function signatures in funcobject.c #124908
  • gh-111178: Fix function signatures in bytearrayobject.c #124940
  • gh-111178: Fix function signatures in Python-ast.c #124942
  • gh-111178: Fix function signatures in classobject.c #124943
  • gh-111178: Fix function signatures in _threadmodule.c #124964
  • gh-111178: Fix function signatures in genobject.c #124970
  • gh-111178: Fix function signatures in fileio.c #125043
  • gh-111178: Fix function signatures in codeobject.c #125180
  • gh-111178: Fix function signatures in cellobject.c #125182
  • gh-111178: fix UBSan failures in _elementtree.c #127982
  • gh-111178: fix UBSan failures in Objects/exceptions.c #128154
  • gh-111178: fix UBSan failures for gdbmobject #128178
  • gh-111178: fix UBSan failures in Python/bltinmodule.c #128235
  • gh-111178: fix UBSan failures in Objects/bytearrayobject.c #128236
  • gh-111178: fix UBSan failures in Objects/bytesobject.c #128237
  • gh-111178: fix UBSan failures in Modules/_bz2module.c #128238
  • gh-111178: fix UBSan failures in Objects/capsule.c #128239
  • gh-111178: fix UBSan failures in Objects/codeobject.c #128240
  • gh-111178: fix UBSan failures in Objects/complexobject.c #128241
  • gh-111178: fix UBSan failures in Python/context.c #128242
  • gh-111178: fix UBSan failures in Modules/_csv.c #128243
  • gh-111178: fix UBSan failures in Modules/curses*.c #128244
  • gh-111178: fix UBSan failures in Objects/descrobject.c #128245
  • gh-111178: fix UBSan failures in Objects/enumobject.c #128246
  • gh-111178: fix UBSan failures in Python/hamt.c #128247
  • gh-111178: fix UBSan failures in Modules/{blake2,md5,sha1,sha2,sha3}module.c #128248
  • gh-111178: fix UBSan failures in Modules/socketmodule.c #128249
  • gh-111178: fix UBSan failures in Modules/_sre/sre.c #128250
  • gh-111178: fix UBSan failures in Objects/tupleobject.c #128251
  • gh-111178: fix UBSan failures in Modules/zlibmodule.c #128252
  • gh-111178: fix UBSan failures in Modules/_abc.c #128253
  • gh-111178: fix UBSan failures in Python/traceback.c #128259
  • gh-111178: Generate correct signature for most self converters #128447
  • gh-111178: Regen clinic and fix exceptions.c post gh-128447 #129060
  • gh-111178: fix UBSan failures in Modules/_ctypes #129071
  • gh-111178: fix UBSan failures in Modules/_decimal #129074
  • gh-111178: fix UBSan failures in Modules/_io/*.c #129083
  • gh-111178: fix UBSan failures in Modules/_multiprocessing/semaphore.c #129084
  • gh-111178: fix UBSan failures in Modules/_sqlite #129087
  • gh-111178: fix UBSan failures in Modules/_ssl/cert.c #129088
  • gh-111178: fix UBSan failures in Modules/cjkcodecs/multibytecodec.c #129090
  • [3.13] gh-111178: fix UBSan failures in Modules/_multiprocessing/semaphore.c (GH-129084) #129100
  • [3.12] gh-111178: fix UBSan failures in Modules/_multiprocessing/semaphore.c (GH-129084) #129101
  • gh-111178: fix UBSan failures in Modules/arraymodule.c #129772
  • gh-111178: fix UBSan failures in Modules/_collectionsmodule.c #129773
  • gh-111178: fix UBSan failures in Modules/_datetimemodule.c #129774
  • gh-111178: fix UBSan failures in Modules/_dbmmodule.c #129775
  • gh-111178: fix UBSan failures in Objects/floatobject.c #129776
  • gh-111178: fix UBSan failures in Objects/frameobject.c #129777
  • gh-111178: fix UBSan failures in Modules/_functoolsmodule.c #129778
  • gh-111178: fix UBSan failures in Modules/_interp*module.c #129779
  • gh-111178: fix UBSan failures in Modules/itertoolsmodule.c #129780
  • gh-111178: fix UBSan failures in Modules/_jsonmodule.c #129781
  • gh-111178: fix UBSan failures in Modules/_lsprof.c #129782
  • gh-111178: fix UBSan failures in Modules/_lzmamodule.c #129783
  • gh-111178: fix UBSan failures in Modules/mmapmodule.c #129784
  • gh-111178: fix UBSan failures in Modules/_operator.c #129785
  • gh-111178: fix UBSan failures in Modules/overlapped.c #129786
  • gh-111178: fix UBSan failures in Modules/_pickle.c #129787
  • gh-111178: fix UBSan failures in Modules/posixmodule.c #129788
  • gh-111178: fix UBSan failures in Modules/pyexpat.c #129789
  • gh-111178: fix UBSan failures in Modules/_queuemodule.c #129790
  • gh-111178: fix UBSan failures in Modules/_randommodule.c #129791
  • gh-111178: fix UBSan failures in Modules/selectmodule.c #129792
  • gh-111178: fix UBSan failures in Modules/_struct.c #129793
  • gh-111178: fix UBSan failures in Modules/_threadmodule.c #129794
  • gh-111178: fix UBSan failures in Modules/_tkinter.c #129795
  • gh-111178: fix UBSan failures in Modules/_winapi.c #129796
  • gh-111178: fix UBSan failures in Modules/xx*.c #129797
  • gh-111178: fix UBSan failures in Modules/_zoneinfo.c #129798
  • gh-111178: fix UBSan failures in Objects/typeobject.c #129799
  • gh-111178: fix UBSan failures in Objects/typevarobject.c #129800
  • gh-111178: fix UBSan failures in Modules/unicodedata.c #129801
  • gh-111178: fix UBSan failures in Modules/_hashopenssl.c #129802
  • gh-111178: fix clang-cl compilation of Modules/mmapmodule.c #130446
  • gh-111178: Fix function signatures in sliceobject.c #130575
  • gh-111178: Fix function signatures in symtable.c #130589
  • gh-111178: Fix function signatures in namespaceobject.c #130590
  • gh-111178: Fix function signatures in instruction_sequence.c #130591
  • gh-111178: Change Argument Clinic signature for METH_O #130682
  • gh-111178: Fix function signatures in structseq.c #130683
  • gh-111178: Fix function signatures of unicodeiter #130684
  • gh-111178: fix UBSan failures in Modules/_ssl.c #130719
  • gh-111178: Fix function signatures in rangeobject.c #131101
  • gh-111178: Change Argument Clinic signature for METH_METHOD #131102
  • gh-111178: Fix PyCMethod API #131135
  • gh-111178: Change Argument Clinic signature for @classmethod #131157
  • gh-111178: Change Argument Clinic signature for @staticmethod (#131157) #131159
  • gh-111178: Fix function signatures in odictobject.c #131160
  • gh-111178: Fix function signatures in rangeobject.c #131161
  • gh-111178: Fix PyRangeIter_Type deallocator #131162
  • gh-111178: Fix function signatures in iterobject.c #131163
  • gh-111178: Fix function signatures in misc files #131180
  • gh-111178: Fix function signatures to fix undefined behavior #131191
  • gh-111178: Fix function signatures for PyStdPrinter #131192
  • gh-111178: Fix _PyLegacyEventHandler_Type deallocator #131193
  • gh-111178: Fix function signatures for test_os #131227
  • gh-111178: Fix function signatures in _asynciomodule.c #131228
  • gh-111178: Fix function signatures for test_types #131455
  • gh-111178: Fix function signatures for test_iter #131456
  • gh-111178: Fix function signatures in _testbuffer.c #131463
  • gh-111178: Fix function signatures in legacy_tracing.c #131464
  • gh-111178: Fix function signatures for multiple tests #131496
  • gh-111178: fix UBSan failures for TaskStepMethWrapper #131602
  • gh-111178: fix UBSan failures for bytesobject.c #131603
  • gh-132097: allow AC to disable fastcall convention to avoid UBSan failures #131605
  • gh-111178: fix UBSan for custom.c examples #131606
  • gh-111178: fix UBSan failures for PyStdPrinter_Object #131607
  • gh-111178: fix UBSan failures for Python/instrumentation.c #131608
  • gh-111178: fix UBSan failures for anextawaitableobject #131609
  • gh-111178: fix UBSan failures for _PyExecutorObject #131610
  • gh-111178: fix UBSan failures for Python/legacy_tracing.c #131611
  • gh-111178: fix UBSan failures for Modules/_testbuffer.c #131612
  • gh-111178: fix UBSan failures for Modules/_testcapi/*.c #131613
  • gh-111178: fix UBSan failures for Modules/_testcapimodule.c #131614
  • gh-111178: fix UBSan failures for Modules/_testmultiphase.c #131615
  • gh-111178: fix UBSan failures for PyBufferWrapper #131616
  • gh-111178: Fix function signatures for test_capi #131659
  • gh-111178: Fix function signatures for test_ctypes #131660
  • gh-111178: Fix function signature for test_threading #131663
  • gh-111178: Fix function signatures for test_zoneinfo #131664
  • gh-132097: remove unnecessary clinic casts to PyCFunction and others #131665
  • gh-111178: Fix function signatures for test_socket #131667
  • gh-111178: Fix getsockaddrarg() undefined behavior #131668
  • gh-111178: Skip tests which require deep stack if UBsan #131669
  • gh-111178: remove redundant casts for functions with correct signatures #131673
  • gh-111178: Fix function signature in pyexpat.c #131674
  • gh-111178: Skip undefined behavior checks in _PyPegen_lookahead() #131714
  • [3.13] gh-111178: Fix getsockaddrarg() undefined behavior (#131668) #131977
  • [3.12] gh-111178: Fix getsockaddrarg() undefined behavior (GH-131668) (GH-131977) #131979
  • gh-111178: Skip test_perf_profiler on function sanitizer #132020
  • gh-111178: fix incorrect function signatures in docs #132395
  • gh-111178: fix incorrect function signatures for Windows builds #133072
  • gh-111178: fix UBSan failures for RemoteUnwinderObject #135539
  • [3.14] gh-111178: fix UBSan failures for RemoteUnwinderObject (GH-135539) #135547
  • gh-111178: Set function=False by default in support.check_sanitizer() #161640
  • [3.15] gh-111178: Set function=False by default in support.check_sanitizer() (#161640) #161713
  • [3.14] gh-111178: Set function=False by default in support.check_sanitizer() (GH-161640) (GH-161713) #161720
Reacted by Sam James, Nedko Arnaudov, Kirill Podoprigora and Erlend E. Aasland

Activity

Next
  1. chrstphrchvz
    added
    type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
    on Oct 22, 2023
  2. thesamesam commented on Nov 5, 2023

    @thesamesam
    thesamesam
    on Nov 5, 2023
    Contributor
    More actions

    See also http://maskray.me/blog/2022-12-18-control-flow-integrity#fsanitizefunction by @MaskRay for more background.

    Reacted by Christopher Chavez
  3. gpshead
    added
    topic-C-API
    on Nov 5, 2023
  4. gpshead commented on Nov 5, 2023

    @gpshead
    gpshead
    on Nov 5, 2023
    Member
    More actions

    This is one example where the recent trend of thinking we should use specific types on our C API is not compatible with parts of our design. Things like your proposed change are likely good. I'd like to avoid gaining new -f flags to the compiler if possible. ABI considerations are something that could force the compiler flag issue.

    In this case it's odd that this counts as undefined behavior. Reality: A pointer is always going to be a pointer regardless of what type it points to - one machine word. The standard is deficient.

  5. ArsenArsen commented on Nov 5, 2023

    @ArsenArsen
    ArsenArsen
    on Nov 5, 2023
    More actions

    A pointer is always going to be a pointer regardless of what type it points to - one machine word. The standard is deficient.

    note that the standard requires that all pointers are of the same finite size, including FPs

  6. encukou commented on Nov 5, 2023

    @encukou
    encukou
    on Nov 5, 2023
    Member
    More actions

    the recent trend of thinking we should use specific types on our C API

    That trend, as I understand it, is unrelated, or only tangentially related. It doesn't involve casting function types; and the discussions around it didn't involve type definition. Please don't conflate it with this issue.

  7. gpshead commented on Nov 5, 2023

    @gpshead
    gpshead
    on Nov 5, 2023
    Member
    More actions

    My comment is more about seeing a function using the specific type. I didn't go looking to see if it has always been that way or was a recent change. Per the source history, this one has always been defined using PyListObject.

    That it causes UBSAN issues is just something to be aware of if we do want to use specific types in more places.

  8. gpshead commented on Nov 5, 2023

    @gpshead
    gpshead
    on Nov 5, 2023
    Member
    More actions

    A pointer is always going to be a pointer regardless of what type it points to - one machine word. The standard is deficient.

    note that the standard requires that all pointers are of the same finite size, including FPs

    Regardless of pointers all being the same size, the compiler behavior concern might be more to do with the structure aliasing concept rather than the pointer itself. I'd have to ask llvm folks. The answer likely wouldn't change what we need to deal with.

  9. gpshead commented on Nov 5, 2023

    @gpshead
    gpshead
    on Nov 5, 2023
    Member
    More actions

    Poking around, we've got quite a few of these PyTypeObject function pointers that are cast in order to construct the structures with functions taking more specific types. The cleanup for clang-17's new UBSAN check will touch many files. (but is otherwise a pretty mechanical change - in most cases we can just change the parameter type to be generic and add a casting assignment to a local variable of the original name within the function to "launder" the type).

  10. encukou commented on Nov 5, 2023

    @encukou
    encukou
    on Nov 5, 2023
    Member
    More actions

    Struct casting & aliasing rules say that you can safely cast PyListObject * to PyObject * and back, and that you can safely mutate the object using both resulting pointers.
    But when calling a function through an incompatible function pointer, there is no cast. PyListObject * and PyObject * are simply not “compatible”.

    When compilers start punishing this, it'll be bad news. Pretty much all types defined in C use this pattern – not just in CPython core, but in third-party extensions as well. It's even in the tutorial: https://docs.python.org/3/extending/newtypes_tutorial.html#adding-data-and-methods-to-the-basic-example

    I would not be surprised if sooner or later I encounter an instance that is won’t-fix because it involves a stable API, or if I am told that this problem should be ignored because fixing it is too disruptive or requires disproportionate review effort

    The overwhelming majority of the fixes should be simple. It's the sheer number of them that will require disproportionate review effort.

    Reacted by Christopher Chavez
  11. gpshead commented on Dec 6, 2023

    @gpshead
    gpshead
    on Dec 6, 2023
    Member
    More actions

    I updated our Clang UBSAN buildbot last night, clang-17. So it fails the build on this issue now. =) https://buildbot.python.org/all/#/workers/2

  12. vstinner commented on Dec 6, 2023

    @vstinner
    vstinner
    on Dec 6, 2023
    Member
    More actions

    See also http://maskray.me/blog/2022-12-18-control-flow-integrity#fsanitizefunction by @MaskRay for more background.

    Oh no, this issue is related to Control Flow Integrity (CFI). I think that this problem should be treated seriously, more and more companies are enforcing CFI. It's better if Python can be adapted to respect CFI as soon as possible. CFI even started to be implemented at the hardware (CPU) level. The Linux kernel does its best to implement CFI to harden the kernel.

    Recently, I added the _PyCFunction_CAST() macro with a long comment:

    // Cast an function to the PyCFunction type to use it with PyMethodDef.
    //
    // This macro can be used to prevent compiler warnings if the first parameter
    // uses a different pointer type than PyObject* (ex: METH_VARARGS and METH_O
    // calling conventions).
    //
    // The macro can also be used for METH_FASTCALL and METH_VARARGS|METH_KEYWORDS
    // calling conventions to avoid compiler warnings because the function has more
    // than 2 parameters. The macro first casts the function to the
    // "void func(void)" type to prevent compiler warnings.
    //
    // If a function is declared with the METH_NOARGS calling convention, it must
    // have 2 parameters. Since the second parameter is unused, Py_UNUSED() can be
    // used to prevent a compiler warning. If the function has a single parameter,
    // it triggers an undefined behavior when Python calls it with 2 parameters
    // (bpo-33012).
    #define _PyCFunction_CAST(func) \
        _Py_CAST(PyCFunction, _Py_CAST(void(*)(void), (func)))

    The problem is that PyMethodDef.ml_meth member type is:

    typedef PyObject *(*PyCFunction)(PyObject *, PyObject *);

    whereas Python uses calling convention with more or less parameters than PyCFunction or with other parameter types :-(

    I think that we should decide a global strategy for this problem, rather than trying the whack-a-mole game with various strategies. It's a problem as bad a Python 2 PyObject structure inheritance which was not compatible with strict aliasing. This problem was solved with a new API, especially this macro:

    #define PyObject_HEAD PyObject ob_base;
    Reacted by Gregory P. Smith and Sam James
  13. vstinner commented on Dec 6, 2023

    @vstinner
    vstinner
    on Dec 6, 2023
    Member
    More actions

    @encukou:

    Struct casting & aliasing rules say that you can safely cast PyListObject * to PyObject * and back, and that you can safely mutate the object using both resulting pointers.

    I have a different experience with casting from/to PyObject*. I discovered "type punning" issue the hard way in #98724

  14. chrstphrchvz commented on Dec 6, 2023

    @chrstphrchvz
    chrstphrchvz
    on Dec 6, 2023
    ContributorAuthor
    More actions

    I have opened a couple of PRs for a few examples of this issue. I intended to separate them by codeowner(s), if that is helpful.

    Some instances of this issue lead to far more errors than others; visit_reachable() seems to be the most common.

    I am not aware how instances involving generated code should be handled:

    cpython/Objects/listobject.c

    Lines 1013 to 1017 in d384813

    [clinic start generated code]*/
    static PyObject *
    py_list_extend(PyListObject *self, PyObject *iterable)
    /*[clinic end generated code: output=b8e0bff0ceae2abd input=9a8376a8633ed3ba]*/

    Objects/listobject.c:2911:5: warning: cast from 'PyObject *(*)(PyListObject *, PyObject *)' (aka 'struct _object *(*)(PyListObject *, struct _object *)') to 'PyCFunction' (aka 'struct _object *(*)(struct _object *, struct _object *)') converts to incompatible function type [-Wcast-function-type-strict]
     2911 |     PY_LIST_EXTEND_METHODDEF
          |     ^~~~~~~~~~~~~~~~~~~~~~~~
    Objects/clinic/listobject.c.h:105:16: note: expanded from macro 'PY_LIST_EXTEND_METHODDEF'
      105 |     {"extend", (PyCFunction)py_list_extend, METH_O, py_list_extend__doc__},
          |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    Python/generated_cases.c.h:1626:19: runtime error: call to function py_list_extend through pointer to incorrect function type 'struct _object *(*)(struct _object *, struct _object *)'
    listobject.c:1025: note: py_list_extend defined here
    SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/generated_cases.c.h:1626:19 in 
    
  15. chrstphrchvz commented on Dec 6, 2023

    @chrstphrchvz
    chrstphrchvz
    on Dec 6, 2023
    ContributorAuthor
    More actions

    I think that we should decide a global strategy for this problem, rather than trying the whack-a-mole game with various strategies.

    Related to this point: how likely is it that instances of this issue will keep being introduced or reintroduced, at least without greater awareness and acceptance of the relevant C rules? How much does having CI run with -fsanitize=function or even -Werror=cast-function-type help prevent this?

  16. 322 remaining items

    Load more actions
  17. diegorusso
    added 5 commits that reference this issue on Apr 1, 2025

    pythongh-111178: fix UBSan failures for `anextawaitableobject` (pytho…

    e4c3acd

    pythongh-111178: Fix function signatures for test_socket (python#131667)

    f001126

    pythongh-111178: fix UBSan for example code in `extending/newtypes_tu…

    af2d5f9

    pythongh-111178: Fix function signature in pyexpat.c (python#131674)

    1498409

    pythongh-111178: Skip undefined behavior checks in _PyPegen_lookahead…

    673773a
  18. vstinner
    added 3 commits that reference this issue on Apr 1, 2025

    gh-111178: Fix getsockaddrarg() undefined behavior (#131668)

    8cd29c2

    pythongh-111178: Fix getsockaddrarg() undefined behavior (python#131668)

    ad606fc

    [3.13] gh-111178: Fix getsockaddrarg() undefined behavior (#131668) (#…

    c318a03
  19. miss-islington
    added a commit that references this issue on Apr 1, 2025

    [3.13] pythongh-111178: Fix getsockaddrarg() undefined behavior (pyth…

    a0fefae
  20. vstinner
    added a commit that references this issue on Apr 1, 2025

    [3.12] gh-111178: Fix getsockaddrarg() undefined behavior (GH-131668) (…

    afb3f33
  21. picnixz
    added a commit that references this issue on Apr 1, 2025

    gh-111178: remove redundant casts for functions with correct signatur…

    edbf7fb
  22. vstinner commented on Apr 3, 2025

    @vstinner
    vstinner
    on Apr 3, 2025
    Member
    More actions

    The whole test suite now pass on Linux on Python built with -fsanitize=function. I close the issue.

    Thanks everyone who was involved in this big change and nice enhancement!

    Reacted by GalaxySnail, Sam James and Chris Eibl
  23. encukou commented on Apr 24, 2025

    @encukou
    encukou
    on Apr 24, 2025
    Member
    More actions

    Hooray! Thank you for the effort everyone!

    But, as noted in the issue:

    We use the macro _Py_NO_SANITIZE_UNDEFINED to disable the UB sanitizer in some hard-to-fix cases, so that we can get a stable, regression-monitoring checker sooner. This issue should only be closed after the macro is removed.

  24. picnixz commented on Apr 24, 2025

    @picnixz
    picnixz
    on Apr 24, 2025
    Member
    More actions

    We use the macro _Py_NO_SANITIZE_UNDEFINED to disable the UB sanitizer in some hard-to-fix cases, so that we can get a stable, regression-monitoring checker sooner. This issue should only be closed after the macro is removed.

    Can we track this in two separate issues instead? I think it'll be easier (I've opened other issues for specific issues because otherwise this issue becomes unreadable)

  25. encukou commented on Apr 29, 2025

    @encukou
    encukou
    on Apr 29, 2025
    Member
    More actions

    Sure: #133157
    (Edit: Well, it's one issue; for separate ones we'd ideally need three -- pyexpat, pegen, and making sure it can't be added elsewhere.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-C-APItype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      You are visitor 01402503 — GeezerVison does not store the pages you look up.
      Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.