TypeScript Bot (@typescript-bot) perf test this

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the perf test suite on this PR at a1f82c1. You can monitor the build here.

Update: The results are in!

Anders Hejlsberg (@ahejlsberg)
The results of the perf run you requested are in!

Performance looks good, even slightly improved, likely because we're no longer jamming an aliasTypeArgumentsContainsMarker property onto variance measurement type instantiations.

The failed fourslash test appears to be a preexisting problem in the main branch.

TypeScript Bot (@typescript-bot) test this
TypeScript Bot (@typescript-bot) user test this
TypeScript Bot (@typescript-bot) run dt

Heya Daniel Rosenwasser (@DanielRosenwasser), I've started to run the parallelized community code test suite on this PR at 813b04b. You can monitor the build here.

Heya Daniel Rosenwasser (@DanielRosenwasser), I've started to run the parallelized Definitely Typed test suite on this PR at 813b04b. You can monitor the build here.

Heya Daniel Rosenwasser (@DanielRosenwasser), I've started to run the extended test suite on this PR at 813b04b. You can monitor the build here.

The user suite test run you requested has finished and failed. I've opened a PR with the baseline diff from master.

Would it be possible to generate a playground for this PR?

I forget whether it checks out the head or merge commit—if the latter, it would need its merge conflicts fixed, but we can try.

TypeScript Bot (@typescript-bot) pack this

Heya Andrew Branch (@andrewbranch), I've started to run the tarball bundle task on this PR at 82c02a7. You can monitor the build here.

Anders Hejlsberg (@ahejlsberg) Daniel Rosenwasser (@DanielRosenwasser) did we have any big concerns over this or did we mostly forget about it? If we think we might want to merge it, let’s work on getting it in early for 4.7.

TypeScript Bot (@typescript-bot) test this
TypeScript Bot (@typescript-bot) user test this inline
TypeScript Bot (@typescript-bot) run dt
TypeScript Bot (@typescript-bot) perf test faster

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the extended test suite on this PR at 2e859a2. You can monitor the build here.

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the diff-based community code test suite on this PR at 2e859a2. You can monitor the build here.

Update: The results are in!

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the parallelized Definitely Typed test suite on this PR at 2e859a2. You can monitor the build here.

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the abridged perf test suite on this PR at 2e859a2. You can monitor the build here.

Update: The results are in!

Anders Hejlsberg (@ahejlsberg)
The results of the user tests run you requested are in!

Andrew Branch (@andrewbranch) could you request a new tarball? the spawned task has failed but that was executed before the recent merge, perhaps a fresh build after this merge would succeed

TypeScript Bot (@typescript-bot) pack this

Heya Anders Hejlsberg (@ahejlsberg), I've started to run the tarball bundle task on this PR at 2e859a2. You can monitor the build here.

Anders Hejlsberg (@ahejlsberg)
The results of the perf run you requested are in!

Hey Anders Hejlsberg (@ahejlsberg), I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/120377/artifacts?artifactName=tgz&fileId=30D5EC3BF9FD87861ED6C80354041E0086256D68AECB7803F51F754554A5659F02&fileName=/typescript-4.7.0-insiders.20220217.tgz"
    }
}

and then running npm install.

Yikes, the xstate performance test is heavily impacted by this (+200%). Definitely not ready to go yet.

Hah, I'm actually XState maintainer 😅 In a way - I think that our types need a solid overhaul anyway. I wonder if there is anything in particular that we could take into consideration when refactoring, are there any particular things that we could try to avoid. While... we are flattered to be included in your perf test suite 😉 it's also a good sign that maybe we are overcomplicating things for the typechecker.

Some recent changes to our types are in particular quite crazy (those designed to work with the typegen and have very unusual patterns used in them to "stitch" types together). My sincere hope is that this particular perf problem is not related to this part of the types as I took a loooot of time to get them working. Could you tell me against which version of XState are you running this benchmark? It would be useful for such information to be included somewhere in the result - as far as I can tell it's not in the comment of the bot nor in the attached benchmark file.

When it comes to the variance itself - I'm pretty sure our types "get it wrong" (from the user perspective). While I understand the concept of variance and can "compute" variance type in simple cases, I have no idea how I should think about it with such complicated types as ours, with a lot of layers, a lot of generics etc. Are there any resources/comments from the past that could help me understand this? Are there any tricks that could help TS to determine the variance type "quicker"?

On a more positive note - I've tested this build and it seems to fix my minimal repro case reported here and it also fixes the issue in our repo from which I've distilled this minimal repro. 🎉

Mateusz Burzyński (@Andarist) Thanks for taking such an interest in this!

Our current instructions for profiling compilation times are at https://github.com/microsoft/TypeScript/wiki/Performance-Tracing and there are some other resources at https://github.com/microsoft/TypeScript/wiki/Performance

The benchmarks run against whatever is currently checked into GitHub. We run against latest to avoid getting stale versions of projects which might not be taking advantage of newer language features.

How to make a project build faster in general is outside the scope of this comment box, but I'd be happy to talk about it in a separate issue after you've run tracing and identified some hotspots.

Ryan Cavanaugh (@RyanCavanaugh) would you recommend performing this profiling using the latest TS or using the build from this PR? I'm unsure what's better from your perspective - for us to attempt to recognize current issues~ or the stuff specifically related to this PR?

The benchmarks run against whatever is currently checked into GitHub. We run against latest to avoid getting stale versions of projects which might not be taking advantage of newer language features.

Thanks, that's good to know - I would recommend adding information somewhere about the commit against which those profiling sessions were executed. It would ensure that we can always investigate exactly the same builds etc when any issues are recognized by this automatic benchmarking.

How to make a project build faster in general is outside the scope of this comment box, but I'd be happy to talk about it in a separate issue after you've run tracing and identified some hotspots.

Thanks, I will certainly come back asking questions 🤣 I'll try to book some time in our next cycle to poke around this

I might also have found a problem with this PR unless the goal of this PR is not to handle all of the possible cases. Are perhaps still situations when you might bail out from the process and assume that some types are assignable when they shouldn't?

As far as I can tell the result of the spawn should not be accepted within the createMachine call at the bottom of this screenshot. I've performed various checks above this situation to confirm subtyping rules etc

I can try to prepare a reduced test case but since that's sometimes up to a day of work I'd like to confirm with you that it's worth it.

The issue can be observed on commit f256378e6 in xstate repo (this is a note for my own reference)

Closing in favor of #48080.