GeezerVison

Is there another module that behaves like this?

import sys

print(type(sys.monitoring))

import sys.monitoring

produces:

<class 'module'>
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/../lab/pep669.py", line 5, in <module>
    import sys.monitoring
ModuleNotFoundError: No module named  'sys.monitoring'; 'sys' is not a package

It's a module but I can't import it directly.

But does it matter? There is no promise that sys.monitoring is a module. It's just a namespace ("let's do more of those!"). You can write import sys; sys.monitoring.set_events(...) or you can use from sys import modules; modules.set_events(...). I'd say having it be a module is better than having it be a class full of static methods. :-)

Is there another module that behaves like this?

import sys

print(type(sys.monitoring))

import sys.monitoring

produces:

<class 'module'>
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/../lab/pep669.py", line 5, in <module>
    import sys.monitoring
ModuleNotFoundError: No module named  'sys.monitoring'; 'sys' is not a package

It's a module but I can't import it directly.

This happens basically whenever a module imports another module:

>>> import ast
>>> type(ast.sys)
<class 'module'>
>>> import ast.sys
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named  'ast.sys'; 'ast' is not a package

🙃

Perhaps a good mental model for sys.monitoring is that the top line of the sys module is import _something_super_secret as monitoring.

It is a bit bit weird that from sys import monitoring as m doesn't work.
We should probably fix that.

I expect to (hopefully before 3.12) use some sort of lazy loading to avoid creating the monitoring object if it isn't needed.

@fabioz Thanks for the reproducer.

OK. I'm merging this.

I think this is stable enough to merge, and we can probably get better bug reports with this merged than on a branch.

@markshannon from the fix you seem to have put there, it'd still fail if the user set the tracing to None and then back to the actual trace function...

I think that is the current behavior, is it not?

If you restart tracing, then you want a line event for the current line.
At least, pdb seems to want that. If I don't clear the "last traced line", when setting f_trace, then the pdb tests fail.

TBH, it is all an undocumented black box, so some guess work is required.
It seems the best we can do is add test cases, so if you have any more tests I'd be grateful.

Darn, I was just about to complete my second review.

$ ./python.exe -m test -R : test_monitoring  # <= fails

I think that is the current behavior, is it not?

If you restart tracing, then you want a line event for the current line. At least, pdb seems to want that. If I don't clear the "last traced line", when setting f_trace, then the pdb tests fail.

TBH, it is all an undocumented black box, so some guess work is required. It seems the best we can do is add test cases, so if you have any more tests I'd be grateful.

Ok, I'll try another round of the debugger tests to see if there's more breakage -- that previous issue didn't really let me get further, so, I'll check and report back -- with bugs in the tracker if that's the case I guess ;)

@markshannon I just checked and changing the tracing shouldn't duplicate line events in the new tracer (it should only report when the line actually changes).

I created #103471 with a test case for this (which works in Python 3.11 and fails in the current master).

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.x has failed when building commit 411b169.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/484/builds/3091) and take a look at the build logs.
4. Check if the failure is related to this commit (411b169) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/484/builds/3091

Summary of the results of the build (if available):

Note: switching to '411b1692811b2ecac59cb0df0f920861c7cf179a'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 411b169281 GH-103082: Implementation of PEP 669: Low Impact Monitoring for CPython (GH-103083)
Switched to and reset branch 'main'

In file included from Python/instrumentation.c:9:
./Include/internal/pycore_object.h:20:35: warning: initialization of ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
   20 | #define _PyObject_IMMORTAL_REFCNT 999999999
      |                                   ^~~~~~~~~
Python/instrumentation.c:19:5: note: in expansion of macro ‘_PyObject_IMMORTAL_REFCNT’
   19 |     _PyObject_IMMORTAL_REFCNT,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
./Include/internal/pycore_object.h:20:35: note: (near initialization for ‘DISABLE._ob_next’)
   20 | #define _PyObject_IMMORTAL_REFCNT 999999999
      |                                   ^~~~~~~~~
Python/instrumentation.c:19:5: note: in expansion of macro ‘_PyObject_IMMORTAL_REFCNT’
   19 |     _PyObject_IMMORTAL_REFCNT,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
Python/instrumentation.c:20:5: warning: initialization of ‘PyObject *’ {aka ‘struct _object *’} from incompatible pointer type ‘PyTypeObject *’ {aka ‘struct _typeobject *’} [-Wincompatible-pointer-types]
   20 |     &PyBaseObject_Type
      |     ^
Python/instrumentation.c:20:5: note: (near initialization for ‘DISABLE._ob_prev’)
./Include/internal/pycore_object.h:20:35: warning: initialization of ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
   20 | #define _PyObject_IMMORTAL_REFCNT 999999999
      |                                   ^~~~~~~~~
Python/instrumentation.c:25:5: note: in expansion of macro ‘_PyObject_IMMORTAL_REFCNT’
   25 |     _PyObject_IMMORTAL_REFCNT,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
./Include/internal/pycore_object.h:20:35: note: (near initialization for ‘_PyInstrumentation_MISSING._ob_next’)
   20 | #define _PyObject_IMMORTAL_REFCNT 999999999
      |                                   ^~~~~~~~~
Python/instrumentation.c:25:5: note: in expansion of macro ‘_PyObject_IMMORTAL_REFCNT’
   25 |     _PyObject_IMMORTAL_REFCNT,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
Python/instrumentation.c:26:5: warning: initialization of ‘PyObject *’ {aka ‘struct _object *’} from incompatible pointer type ‘PyTypeObject *’ {aka ‘struct _typeobject *’} [-Wincompatible-pointer-types]
   26 |     &PyBaseObject_Type
      |     ^
Python/instrumentation.c:26:5: note: (near initialization for ‘_PyInstrumentation_MISSING._ob_prev’)
Modules/gcmodule.c:450: visit_decref: Assertion "!_PyObject_IsFreed(op)" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f159fa02740
object refcount : 1
object type     : 0x563902ad8980
object type name: dict
object repr     : make: *** [Makefile:1282: Python/frozen_modules/abc.h] Segmentation fault (core dumped)

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [Makefile:2676: clean-retain-profile] Error 1 (ignored)