GeezerVison

Copy link

Copy Markdown

Member

$ cp  build/lib.linux-x86_64-3.7/_sysconfigdata_m_linux_x86_64-linux-gnu.py sysconfigdata
$ ./python -c 'import sysconfigdata'  # create pyc file
$ ./python -m timeit -s 'import sysconfigdata, importlib' -- 'importlib.reload(sysconfigdata)'
1000 loops, best of 5: 269 usec per loop

methane force-pushed the optimize-site-startup branch from 4f58b0c to 483769e Compare February 16, 2017 14:35

ned-deily previously requested changes Feb 16, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner requested changes Feb 17, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

methane force-pushed the optimize-site-startup branch from dfeced1 to 45f5e9a Compare February 18, 2017 05:04

methane force-pushed the optimize-site-startup branch from 45f5e9a to 07369f1 Compare February 18, 2017 05:10

methane force-pushed the optimize-site-startup branch from 75b4c6c to ff0d05c Compare June 28, 2017 10:29

haypo@selma$ ./python -m perf command --inherit=PYTHONPATH -v -o pr136.json -- ./python -c pass  
...
haypo@selma$ ./python -m perf command --inherit=PYTHONPATH -v -o ref.json -- ./python -c pass
...
haypo@selma$ ./python -m perf compare_to ref.json pr136.json 
Mean +- std dev: [ref] 17.4 ms +- 0.8 ms -> [pr136] 16.6 ms +- 1.1 ms: 1.05x faster (-5%)

        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations.
            from sysconfig import get_config_var
            framework = get_config_var("PYTHONFRAMEWORK")
            if framework:
                sitepackages.append(
                        os.path.join("/Library", framework,
                            '%d.%d' % sys.version_info[:2], "site-packages"))

diff --git a/Lib/site.py b/Lib/site.py
index 500c59b..929252d 100644
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -334,15 +334,12 @@ def getsitepackages(prefixes=None):
         else:
             sitepackages.append(prefix)
             sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
-        if sys.platform == "darwin":
-            # for framework builds *only* we add the standard Apple
-            # locations.
-            from sysconfig import get_config_var
-            framework = get_config_var("PYTHONFRAMEWORK")
-            if framework:
-                sitepackages.append(
-                        os.path.join("/Library", framework,
-                            '%d.%d' % sys.version_info[:2], "site-packages"))
+        # for framework builds *only* we add the standard Apple
+        # locations.
+        if sys.platform == "darwin" and sys._framework:
+            sitepackages.append(
+                    os.path.join("/Library", sys._framework,
+                        '%d.%d' % sys.version_info[:2], "site-packages"))
     return sitepackages
 
 def addsitepackages(known_paths, prefixes=None):

macbook:master haypo$ ./python.exe -m perf command --inherit=PYTHONPATH -v -o ref.json -- ./python.exe -c pass 
...
macbook:master haypo$ ./python.exe -m perf command --inherit=PYTHONPATH -v -o pr136.json -- ./python.exe -c pass
...
macbook:master haypo$ ./python.exe -m perf compare_to ref.json pr136.json 
Mean +- std dev: [ref] 35.4 ms +- 1.7 ms -> [pr136] 22.0 ms +- 1.9 ms: 1.61x faster (-38%)

vstinner requested changes Jun 28, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner dismissed ned-deily’s stale review June 28, 2017 12:19

$ ./python.exe -m perf compare_to ref.json pr136.json
Mean +- std dev: [ref] 30.4 ms +- 0.9 ms -> [pr136] 21.6 ms +- 4.0 ms: 1.40x faster (-29%)

vstinner reviewed Jun 28, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

vstinner approved these changes Jun 28, 2017

View reviewed changes

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

methane deleted the optimize-site-startup branch June 28, 2017 15:31

     2>..\Python\sysmodule.c(1968): error C2065: 'PYTHONFRAMEWORK': undeclared identifier [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]

This reverts commit a8f8d5b.

Python 3 has no xrange.

…_inline_except_opcode_array_c

Fixes THREE boundary-domain bugs in build_inline_except_opcode_array_c
(introduced W27c #2a 7135d94), found across two HIR-diff Phase 0 cycles
of the W-2B-RECONVERT investigation.

CONVENTION (Python/jit/bytecode.cpp:8-14, builder.cpp:1235,
phx_frame_state.h cur_instr_offs semantics):
- jit_bc_instr_init expects INSTRUCTION INDEX (codeUnit[])
- jit_bc_instr_get_jump_target / next_offset / base_offset return whatever
  was stored at init (now INDEX after Class A fix)
- phx_block_map keys are BYTE OFFSETS
- OpcodeArrayEntry.base_offset is consumed downstream as BYTE OFFSET
  (cur_instr_offs assignment per builder_emit_c.c:3320)
- BCOffset.value() (caller-passed except_body_offset) is BYTE OFFSET
- BYTES = INDEX * sizeof(_Py_CODEUNIT) (= 2 in 3.12)

NAMED CONVERSIONS (Python/jit/bytecode_c.h, gated by _Py_OPCODE):
  static inline int phx_bc_offset_to_instr_index(int byte_off);
  static inline int phx_bc_instr_index_to_offset(int instr_idx);

Codifies the boundary-domain rule by example (per pythia python#137 python#2 +
supervisor 19:01:19Z + theologian 19:01:17Z).

THREE FIXES:

CLASS A (line 3241): jit_bc_instr_init was passed except_body_offset
(BCOffset.value() byte offset) where INSTRUCTION INDEX was expected.
codeUnit(code)[byte_offset] read PAST end of co_code → garbage opcode →
switch-default → Deopt with corrupt frame state. Found by Phase 0
HIR-diff (test_exc_raise_catch bb 12: correct Return -1 vs corrupt
LoadConst NoneType + Deopt at offset 58).

CLASS B (line 3273-3275, theologian class-of-bug audit 18:42:53Z):
target = jit_bc_instr_get_jump_target returns INDEX, but
phx_block_map_lookup_or_panic expects BYTE OFFSET. Without conversion,
JUMP_BACKWARD-in-except-body lookup fails: JIT_CHECK_C panic OR silent
wrong-block. Dormant pre-fix because no test had backward-jump-in-except-body.

CLASS C (line 3260, exposed by Phase 0' HIR-diff after Class A+B fix):
After Class A fix corrected the init to INDEX, jit_bc_instr_base_offset
returns INDEX. But entry->base_offset is consumed downstream (line 3320)
as BYTE OFFSET via match_tc.frame.cur_instr_offs assignment. Pre-fix
'correct by accident' — Class A's BYTES-as-INDEX init wrote BYTES into
bci->base_offset, so jit_bc_instr_base_offset returned BYTES, matching
downstream. Correct Class A exposed Class C: cur_instr_offs got INDEX
(half the correct BYTE value) → interpreter Deopt resumed at wrong
bytecode position → SIGSEGV in test_multiple_exceptions_in_loop
(deterministic 0/20 post Class A+B fix, vs 20/20 PASS pre-W27c).

DIAGNOSIS:
HIR-diff for test_multiple_exceptions_in_loop revealed Deopt CurInstrOffset
124 (correct, BYTES) → 62 (wrong, INDEX = 124/2). Direct evidence of the
domain mismatch.

LATENT in pushed W27c #2a (e4e7507 on SonicField/cpython): all three
classes present. Class A, B dormant (no test exercises emitInlineExceptionMatch
or JUMP_BACKWARD-in-except-body). Class C compensated by Class A — both
broken in opposite directions canceling out for downstream consumers of
entry->base_offset. ALL three must fix together.

INVESTIGATION CHAIN:
- testkeeper bisect 17:52:30Z localized #2b regression → W27c #2b sole
- pythia python#136 python#1 18:23:34Z flagged HEAP/RACE rode on absence-of-evidence
- generalist 18:24Z proposed HIR-diff Phase 0 falsifier
- generalist 18:32+18:34Z captured HIR_2a + HIR_2b dumps; found Class A
- theologian 18:42:53Z class-of-bug audit found Class B
- supervisor 18:43:17Z directed dual-fix
- pythia python#137 python#2 19:00:29Z flagged inline-arithmetic violates boundary-domain rule
- supervisor 19:01:19Z + theologian 19:01:17Z directed amend to named conversions
- testkeeper 19:06:55Z full Phoenix gate caught NEW regression
  (test_multiple_exceptions_in_loop deterministic 0/20)
- generalist 19:14Z HIR-diff Phase 0' on test_multiple_exceptions_in_loop
  revealed Class C (cur_instr_offs 124→62)
- supervisor 19:16:42Z authorized Class C fix + extended audit

OTHER OpcodeArrayEntry FIELDS AUDITED (per supervisor 19:16:42Z extended
class-of-bug discipline):
- entry->opcode: written from jit_bc_instr_opcode (no domain — opcode value);
  consumed in dispatch loop switch. CLEAN.
- entry->oparg: written from jit_bc_instr_oparg (no domain — oparg value);
  consumed in dispatch loop emit calls. CLEAN.
- entry->base_offset: Class C above; FIXED.
- entry->const_obj: written from PyTuple_GET_ITEM (PyObject*); consumed in
  dispatch loop hir_type_from_object. CLEAN (no domain conversion).
- entry->jump_target_block: Class B above; FIXED.

VERIFICATION pending (testkeeper 4-suite extended verify):
1. 30x test_exc_raise_catch (Class A regression)
2. 30x test_exc_binary_subscr_dict_in_try (Class A latent activation)
3. 30x test_exc_continue_in_loop (Class B latent activation)
4. 30x multi-except-in-loop sentinel (Class C latent activation)
5. Full Phoenix suite (which originally caught Class C)