GeezerVison is showing github.com — runtime: green tea garbage collector · Issue #73581 · golang/go
runtime: green tea garbage collector #73581DescriptionGreen Tea 🍵 Garbage Collector
Authors: Michael Knyszek, Austin Clements
Updated: 15 August 2025
This issue tracks the design and implementation of the Green Tea garbage collector. As of the last update to this issue, development of Green Tea is still active. We'll produce more detailed design document once we're ready to commit to a design.
For now, Green Tea is available as an experiment in the Go 1.25 release. We don't expect any correctness issues with the experiment. It's running within Google, and we feel that it is production-ready. We encourage teams to try it out. Your feedback is instrumental to producing a good final design!
Introduction
Memory latency and bandwidth are becoming increasingly constrained as CPU clocks outpace DRAM clocks, increasing core counts offer more load on physically limited memory buses, and speed-of-light constraints necessitate increasingly non-uniform memory topologies. As a result, spatial locality, temporal locality, and topology-awareness are becoming ever more critical to high-performance systems.
Unfortunately, all of these trends are at odds with most of today’s garbage collection algorithms. Go’s garbage collector implements a classic tri-color parallel marking algorithm. This is, at its core, just a graph flood, where heap objects are nodes in the graph, and pointers are edges. However, this graph flood affords no consideration to the memory location of the objects that are being processed. As a result, it exhibits extremely poor spatial locality—jumping between completely different parts of memory—poor temporal locality—blithely spreading repeated accesses to the same memory across the GC cycle—and no concern for topology.
As a result, on average 85% of the garbage collector's time is spent in the core loop of this graph flood—the scan loop—and >35% of CPU cycles in the scan loop are spent solely stalled on memory accesses, excluding any knock-on effects. This problem is expected to only get worse as the industry trends toward many-core systems and non-uniform memory architectures.
In this document, we present Green Tea: a parallel marking algorithm that, if not memory-centric,1 is at least memory-aware, in that it endeavors to process objects close to one another together.
This new algorithm has an implementation that is ready for developers to trial on their workloads, and in this document we also present the results from evaluating this implementation against our benchmark suite. Overall, the algorithm shows a significant reduction in GC CPU costs on GC-heavy workloads.
Finally, this new marking algorithm unlocks new opportunities for future optimization, such as SIMD acceleration, which we discuss with other possible avenues of future work.
Design
The core idea behind the new parallel marking algorithm is simple. Instead of scanning individual objects, the garbage collector scans memory in much larger, contiguous blocks. The shared work queue tracks these coarse blocks instead of individual objects, and the individual objects waiting to be scanned in a block are tracked in that block itself. The core hypothesis is that while a block waits on the queue to be scanned, it will accumulate more objects to be scanned within that block, such that when a block does get dequeued, it’s likely that scanning will be able to scan more than one object in that block. This, in turn, improves locality of memory access, in addition to better amortizing per-scan costs.
Prototype implementation
In the prototype implementation of this new algorithm, the memory blocks we track are called spans. A span is always some multiple of 8 KiB, always aligned to 8 KiB, and consists entirely of objects of one size. Our prototype focuses exclusively on “small object spans”, which are exactly 8 KiB and contain objects up to 512 bytes.
A span is also the basic unit of storing heap metadata. In the prototype, each span stores two bits for each object: a gray bit and a black bit. These correspond to the tri-color abstraction: an object is black if it has been scanned, gray if it is in the queue to be scanned, and white if it has not been reached at all. In the prototype, white objects have neither bit set, gray objects have the gray bit set, and black objects have both bits set.
When scanning finds a pointer to a small object, it sets that object’s gray bit to indicate the object needs to be scanned. If the gray bit was not already set and the object’s span is not already enqueued for scanning, it enqueues the span. A per-span flag indicates whether the span is currently enqueued so it will only be enqueued once at a time. When the scan loop dequeues a span, it computes the difference between the gray bits and the black bits to identify objects to scan, copies the gray bits to the black bits, and scans any objects that had their gray bit set but not their black bit.
Limiting scope to small objects
The prototype focuses on small objects because we derive the most benefit from them. The per-scan overhead of small objects is much harder to amortize because the garbage collector spends so little time scanning each individual object. Larger objects continue to use the old algorithm.
The choice of which algorithm to use is made when scanning encounters a pointer. The span allocator maintains a bitmap with one bit for each 8 KiB page in the heap to indicate whether that page is backed by a small object span. The footprint of this fits easily into cache even for very large heaps, and contention is extremely low.
Since small object spans are always 8 KiB large and 8 KiB aligned, once the scanner knows the target of a pointer is in a small object span, it can use simple address arithmetic to find the object’s metadata within the span, thus avoiding indirections and dependent loads that seriously harm performance.
Work distribution
Go’s current garbage collector distributes work across scanners by having each scanner maintain a local fixed-sized stack of object pointers. However, in order to ensure parallelism, each scanner aggressively checks and populates global lists. This frequent mutation of the global lists is a significant source of contention in Go programs on many-core systems.
The prototype implementation has a separate queue dedicated to spans and based on the distributed work-stealing runqueues used by the goroutine scheduler. The opportunity for stealing work directly from other workers means less contention on global lists. Furthermore, by queuing spans instead of individual objects, there are far fewer items to queue and thus inherently lower contention on the queues.
Span work may be ordered several different ways. We explored several policies, including FIFO, LIFO, sparsest-first, and densest-first, random, and address-ordered. FIFO turned out to accumulate the highest average density of objects to scan on a span by the time it was dequeued for scanning.
Single-object scan optimization
If a span has only a single object to scan when it is dequeued, the new algorithm will have done more work to handle that single object than the current, object-centric algorithm does.
To bring the performance of the single-object-per-span case more in line with the current marking algorithm, we apply two tricks. First, we track the object that was marked when the span was enqueued. This object becomes the span's representative until the span is scanned. Next, we add a hit flag to the span that indicates an object was marked while the span was queued; that is, that at least two objects are marked. When scanning a span, if the hit flag is not set, then the garbage collector can directly scan the span’s representative, instead of processing the entire span.
Prototype evaluation
We evaluated the prototype implementation across a variety of benchmarks, on low-CPU-count, high-CPU-count, amd64, and arm64 Linux virtual machines. The rest of this section summarizes the results, focusing primarily on the differences in garbage collection CPU cost.
In select GC-heavy microbenchmarks (garbage, from golang.org/x/benchmarks, and binary-trees Go #2, from the Computer Language Benchmarks Game), depending on core count, we observed anywhere from a 10–50% reduction in GC CPU costs compared to the existing Go GC. The improvement generally rose with core count, indicating that the prototype scales better than the existing implementation. Furthermore, the number of L1 and L2 cache misses was reduced by half in these benchmarks.
On our bent and sweet benchmark suites, the results were more varied.
The results are positive overall, but include a mix of improvements and regressions.
Most benchmarks were either unaffected by the changes to the garbage collector or regressed or improved solely due to changes that had little to do with the garbage collector, such as code alignment changes. Some benchmarks regressed even though less CPU time is spent in the garbage collector. One reason for this is because the garbage collector's mark phase is active for less time, leading to less floating garbage which acts as a ballast in some benchmarks. Another reason for this is that less time spent in the garbage collector means more time spent in other scalability bottlenecks, either in the runtime or in user code, leading to a net apparent regression.
The Go compiler benchmarks appear to inconsistently show a very slight regression (0.5%). Given the magnitude and inconsistency of the regression, these benchmarks appear to be rather insensitive to this change. One hypothesis is that the occasional regression may be due to an out-of-date PGO profile, but remains to be investigated.
The tile38 benchmark shows substantial improvements across throughput, latency and memory use. In addition, it showed a 35% reduction in garbage collection overheads. This benchmark queries a local instance of a Tile38 in-memory geospatial database pre-seeded with data. Most of the heap consists of a high-fanout tree, so Green Tea is able to quickly generate large amounts of work and high densities.
The bleve-index benchmark has a heap topology that is quite difficult for Green Tea, though performance overall is a wash. Most of the heap consists of a low-fanout binary tree that is rapidly mutated by the benchmark. Green Tea struggles to generate locality in this benchmark, and half of all span scans only scan a single object. Our current hypothesis is that because of frequent tree rotations, the tree structure itself becomes shuffled across a large heap (100+ MiB). In contrast, the binary-trees benchmark does not perform rotations, so the tree layout retains the good locality of its initial allocations. This suggests that Green Tea has good locality when the application itself has good locality, unlike the current Go GC; but Green Tea, unsurprisingly, can’t create locality out of nothing. Both Linux perf and CPU profiles in the 16-core amd64 environment indicate a small ~2% regression in garbage collection overhead. Overall, the single object scan optimization was integral to making this benchmark perform well in the 16-core amd64 environment. In the 72- and 88-core environments we see a significant improvement, due to the design's improved many-core scalability. There remains a small overall regression on the 16-core arm64 environment that still needs to be investigated.
Future work
SIMD-accelerated scanning kernels
Scanning memory in larger blocks of memory unlocks the ability to apply SIMD to small objects in the garbage collector. The core idea is to generate a unique scanning kernel for each size class and use SIMD bit manipulation and permutation instructions to load, mask, swizzle, pack, and enqueue pointers. The regularity of the layout of objects in a single span and Go’s packed representation of pointer/scalar metadata for small objects both play a major role in making this feasible.
Austin Clements developed prototype AVX512-based scanning kernels that reduce garbage collection overheads by another 15–20% in the benchmarks where we already saw improvements. The prototype implementation does not currently use these kernels because they only apply to a small subset of objects at this point in time.
These SIMD kernels tend to require a higher density of objects in order to outperform sparsely scanning objects within a scan. These kernels are still being developed, so the prototype does not use them by default, and when they are enabled, it only uses SIMD scanning when a minimum density threshold is reached.
Concentrator network
Austin's original design for Green Tea used a sorting network called the concentrator network to achieve the even higher pointer density required by SIMD-based scanning, and to generate locality even for metadata operations like setting gray bits. The network was carefully designed to minimize queuing costs, so individual pointers could still be processed efficiently when sufficient scan density was unavailable.
There are two main reasons we did not pursue this direction in the short term. First, we found that even very low density can produce good results, especially with the single-object scan optimization. Second, the concentrator network is more complex to implement, as it is a greater departure from the existing algorithm. However, this design is still an avenue we plan to explore, since it is far more general and tunable.
Acknowledgements
Credit to Austin Clements for the inception of the key idea (that is, accumulating pointers on a span before scanning, which they wrote down in 2018!), initial prototyping of the concentrator network, which motivated further work, and in polishing up this text significantly.
Credit to Yves Vandriessche from Intel for providing many microarchitectural insights that were vital to making this design viable. Many of his suggestions were applied to the new core scan loop, including proper prefetching, batching subsequent pointer loads to hide their memory latency, and simpler iteration over object pointers.
Footnotes
-
"The Garbage Collection Handbook" by Richard Jones, Anthony Hosking, Eliot Moss, a canonical source for garbage collection techniques, divides parallel garbage collection algorithms into two categories, "processor-centric" and "memory-centric." Loosely, the former category consists of algorithms that aggressively balance work across processors to maximize parallelism, while the latter consists of algorithms that may not parallelize as well, but process contiguous blocks of the heap. The current marking algorithm in the Go garbage collector is firmly processor-centric. Unfortunately for us, the handbook's wisdom ends there, as it goes on to state that "[a]ll known parallel marking algorithms are processor-centric." (page 279) ↩
Reacted by Daniel Martí, Caleb Spare, thepudds, jimmyfrasche, Yuri, Florian Zenker, montag451, Justin Nuß, Tony Bai, JinglingB and 304 more Reacted by Daniel Martí, Caleb Spare, thepudds, jimmyfrasche, Yuri, AndrewHarrisSPU, Artyom Pervukhin, Florian Zenker, Tom Thorogood, montag451 and 219 more Reacted by Austin Clements, Dominik Honnef, Mateusz Poliwczak, Jo, thepudds, Daniel Martí, Caleb Spare, jimmyfrasche, Dan Peterson, Artyom Pervukhin and 171 more Reacted by jub0bs, niulai6, Septem, Yichao, Ramneek Singh, Jack Deal, Mohamed Ragab, Max Kotliar, Nikolay, Rajika Imal and 40 more Activitygopherbot commented on May 2, 2025 mknyszek commented on May 2, 2025 ContributorAuthor More actions How to try it out
First,
go install golang.org/dl/go1.25rc1@latest
Then build your program with GOEXPERIMENT=greenteagc go1.25rc1 build.
Sending feedback
Please focus your attention on whole programs. Microbenchmarks tend to be poor representatives of garbage collection behavior in real programs. Large changes in the foundations such as these can move the needle for all sorts of reasons, unrelated to (or even inversely related to) how efficient the garbage collector actually is.
If you encounter a situation where Green Tea isn't working out for you, we'd appreciate it if you could share some details with us that would be really helpful moving forward. (This is also helpful if it is working for you.)
- The platform you're running on.
- The CPU model (or if in the cloud, the VM type) you're running your code on.
- Excerpts from stderr of your program running with
GODEBUG=gctrace=2, both with and without Green Tea.
- A CPU profile of your program running with and without Green Tea.
- An execution trace capturing a few complete GC cycles, with and without Green Tea (a few seconds is usually enough).
Note: you can use GOEXPERIMENT=nogreenteagc to explicitly disable Green Tea at build time.
For problems you encounter, please file a new GitHub issue and reference this one, or you can email me directly at mknyszek(at)golang.org if you're not comfortable publishing details or diagnostics publicly.
For successes, post to this issue!
Thank you! Reacted by Rodrigo, Victor Ortega, Henry Oman, SwatiModi, MacJayLee, Woojong Koh, nathaponb, Anil Khatri, skartikey and Wojciech Kaczmarek Reacted by Yuri, PL Pery 🐢, Florian Zenker, Dominik Honnef, Dmitri Shuralyov, James Scott, Matheus Macabu, Tai Groot, Oleg, montag451 and 76 more Reacted by Serhii Ivanov, Gabriel, Nathapon B. (O), James and Woojong Koh gopherbot commented on May 2, 2025 dominikh commented on May 15, 2025 No discernible change Minor improvement in CPU time for Staticcheck.
CPU: AMD Ryzen 9 3950X 16-Core Processor
> ./staticcheck -debug.version | head -1
staticcheck (devel, v0.7.0-0.dev.0.20250511162810-63832850a84e)
Vanilla:
> GOEXPERIMENT=nogreenteagc go build
> go version ./staticcheck
./staticcheck: go1.25-devel_fac2ccbed3 Thu May 15 08:27:04 2025 -0700
> rm -rf ~/.cache/staticcheck/
> perf stat ./staticcheck -debug.cpuprofile=vanilla.cpu.pprof std
Performance counter stats for './staticcheck -debug.cpuprofile=vanilla.cpu.pprof std':
197,121.11 msec task-clock:u # 18.932 CPUs utilized
0 context-switches:u # 0.000 /sec
0 cpu-migrations:u # 0.000 /sec
741,678 page-faults:u # 3.763 K/sec
738,518,295,442 cycles:u # 3.747 GHz
124,817,958,966 stalled-cycles-frontend:u # 16.90% frontend cycles idle
785,988,849,431 instructions:u # 1.06 insn per cycle
# 0.16 stalled cycles per insn
163,480,505,449 branches:u # 829.340 M/sec
2,197,680,754 branch-misses:u # 1.34% of all branches
10.411895545 seconds time elapsed
182.959316000 seconds user
10.483651000 seconds sys
> rm -rf ~/.cache/staticcheck
> ./staticcheck -debug.trace=vanilla.trace std # trace collected separately
> rm -rf ~/.cache/staticcheck
> GODEBUG=gctrace=2 ./staticcheck std 2>vanilla.gctrace # GC trace collected separately
Green tea:
> GOEXPERIMENT=greenteagc go build
> go version ./staticcheck
./staticcheck: go1.25-devel_fac2ccbed3 Thu May 15 08:27:04 2025 -0700 X:greenteagc
> rm -rf ~/.cache/staticcheck/
> perf stat ./staticcheck -debug.cpuprofile=greentea.cpu.pprof std
Performance counter stats for './staticcheck -debug.cpuprofile=greentea.cpu.pprof std':
193,681.28 msec task-clock:u # 18.553 CPUs utilized
0 context-switches:u # 0.000 /sec
0 cpu-migrations:u # 0.000 /sec
782,480 page-faults:u # 4.040 K/sec
723,000,663,331 cycles:u # 3.733 GHz
130,424,001,536 stalled-cycles-frontend:u # 18.04% frontend cycles idle
788,115,346,312 instructions:u # 1.09 insn per cycle
# 0.17 stalled cycles per insn
158,001,376,147 branches:u # 815.780 M/sec
2,432,170,668 branch-misses:u # 1.54% of all branches
10.439577978 seconds time elapsed
178.367421000 seconds user
11.257257000 seconds sys
> rm -rf ~/.cache/staticcheck
> ./staticcheck -debug.trace=greentea.trace std # trace collected separately
> rm -rf ~/.cache/staticcheck
> GODEBUG=gctrace=2 ./staticcheck std 2>greentea.gctrace # GC trace collected separately
The attached files contain the CPU profiles and the gctrace output. I also have execution traces, but they span the whole execution and are ~50 MB each when compressed, which GitHub won't let me attach.
greentea.tar.gz
vanilla.tar.gz
perf stat before/after courtesy of normalize-benchfmt:
│ vanilla.bench │ greentea.bench │
│ sec │ sec vs base │
Task-clock:u 199.3 ± 0% 194.4 ± 1% -2.50% (p=0.000 n=10)
Wall-time 10.11 ± 0% 10.18 ± 1% +0.74% (p=0.005 n=10)
User-time 184.9 ± 0% 178.9 ± 1% -3.26% (p=0.000 n=10)
System-time 10.66 ± 2% 11.33 ± 3% +6.32% (p=0.000 n=10)
geomean 44.63 44.75 +0.25%
│ vanilla.bench │ greentea.bench │
│ val │ val vs base │
Context-switches:u 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Cpu-migrations:u 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
Page-faults:u 796.2k ± 4% 804.4k ± 4% ~ (p=0.579 n=10)
Cycles:u 739.5G ± 0% 719.6G ± 1% -2.69% (p=0.000 n=10)
Stalled-cycles-frontend:u 125.9G ± 0% 131.1G ± 0% +4.07% (p=0.000 n=10)
Instructions:u 786.6G ± 0% 789.0G ± 0% +0.30% (p=0.000 n=10)
Branches:u 163.6G ± 0% 158.1G ± 0% -3.36% (p=0.000 n=10)
Branch-misses:u 2.205G ± 0% 2.436G ± 0% +10.48% (p=0.000 n=10)
geomean ² +1.15% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
Reacted by Michael Knyszek, Michael Pratt, Anto, Elvis, lexno, ChienHo, Nicola Larosa, Riyad Parvez and Nanda Safiq mknyszek commented on May 15, 2025 ContributorAuthor More actions Thanks for sharing! I think there is a win here, though it's not a massive improvement. I think there are some effects that might be masking a bigger win.
At least according to the gctrace I see less overall CPU time spent in any individual GC cycle. On average around 10-15% .
Also, here are the distributions of wall time spent in the mark phase:
$ cat greentea.gctrace | grep ^gc | cut -d ' ' -f 5 | cut -d '+' -f 2 | dist
N 152 sum 3269.07 mean 21.507 gmean 12.3254 std dev 19.7445 variance 389.844
min 0.28
1%ile 0.365667
5%ile 0.9985
25%ile 4.94167
median 16.5
75%ile 29.5833
95%ile 59.1
99%ile 82.1533
max 101
⠀⠀⠀⠀⠀⠀⠀⠀⢀⡴⠒⠒⠢⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.036
⠀⠀⠀⠀⠀⠀⠀⡰⠋⠀⠀⠀⠀⠈⠓⠒⠒⠤⠒⠒⠒⠒⠒⠦⢤⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠤⠞⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠉⠙⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠋⠉⠉⠙⠒⠒⠦⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠁
0 50 100
$ cat vanilla.gctrace | grep ^gc | cut -d ' ' -f 5 | cut -d '+' -f 2 | dist
N 139 sum 3502.72 mean 25.1994 gmean 15.1408 std dev 20.7032 variance 428.621
min 0.28
1%ile 0.3018
5%ile 1.13
25%ile 6.28333
median 21
75%ile 36.6667
95%ile 66.4
99%ile 80.2733
max 81
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡤⠒⠒⠒⠲⢤⣀⠀⠀⠀⠀⠀⠀⠀⣀⣀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.028
⠀⠀⠀⠀⠀⠀⠀⣠⠔⠁⠀⠀⠀⠀⠀⠀⠈⠓⠒⠒⠊⠉⠉⠉⠁⠀⠈⠉⠙⠒⠦⢤⣀⡀⠀⠀⠀⠀⠀⠀⠀⢀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠤⠚⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠉⠉⠉⠉⠉⠉⠁⠈⠉⠉⠉⠉⠉⠉⠉⠉⠒⠒⠒⠒⠒⠒⠢⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
0
Simultaneously, because each GC cycle is shorter, there's less floating garbage, leading to a smaller live heap. This means for the same amount of allocation work, we're going to have more GC cycles, if GOGC is constant (which it is here). Proof of that is in the fact that with Green Tea there are 152 GC cycles, whereas the original GC only executes 139 cycles. Despite this increase in the number of cycles, the CPU profile suggests that around 10% less time is spent in the garbage collector.
Here are the heap size distributions:
$ cat greentea.gctrace | grep ^gc | cut -d ' ' -f 11 | sed 's/->/\./g' | cut -d '.' -f 2 | dist
N 152 sum 78714 mean 517.855 gmean 295.01 std dev 441.656 variance 195060
min 3
1%ile 4.71333
5%ile 23.85
25%ile 118.167
median 425
75%ile 746.583
95%ile 1422.05
99%ile 1576.32
max 1644
⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡤⠒⠒⠒⠲⢄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.001
⠀⠀⠀⠀⠀⠀⠀⣠⠔⠁⠀⠀⠀⠀⠀⠀⠙⠒⠦⠤⠤⠖⠒⠋⠉⠉⠉⠓⠒⠒⠒⠲⠤⠤⠤⣄⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠴⠊⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠙⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠚⠉⠉⠉⠉⠉⠉⠙⠒⠒⠢⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
0 1000
$ cat vanilla.gctrace | grep ^gc | cut -d ' ' -f 11 | sed 's/->/\./g' | cut -d '.' -f 2 | dist
N 139 sum 77043 mean 554.266 gmean 314.515 std dev 450.016 variance 202514
min 3
1%ile 4.45333
5%ile 22
25%ile 124.333
median 504
75%ile 820.833
95%ile 1492.1
99%ile 1602.14
max 1661
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠴⠒⠒⠒⠦⣄⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.001
⠀⠀⠀⠀⠀⠀⠀⣠⠞⠁⠀⠀⠀⠀⠀⠈⠑⠦⠤⠤⠤⠖⠋⠉⠀⠀⠀⠈⠉⠉⠙⠒⠦⠤⠤⢤⣀⣀⣀⡀⠀⣀⣀⣀⣀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠴⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠉⠁⠀⠀⠀⠈⠉⠉⠉⠒⠒⠒⠒⠒⠚⠉⠉⠉⠉⠙⠒⠒⠒⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
As for why this isn't improving overall end-to-end time much, I suspect that:
- The garbage collector is just not on the critical path. Does
staticcheck run in parallel or is it mostly a single goroutine?
- The increase in the number of GC cycles from a reduction in memory is eroding some of the CPU win.
(I could probably analyze this more efficiently if I just made a tool to parse gctraces... I've been doing quite a lot of this lately. 😅) Reacted by Anto, Jackie Li, devin, Mafa Rioch, Hamed Nemati, Vasily Leonenko, Nicola Larosa, Riyad Parvez and mengseeker corporatepiyush commented on Oct 29, 2025 Curious to know,
what importance does the 8 KiB page size holds ?
why it isn't configurable ?
what sort of pros and cons user might experience if it is set to 4KiB or 16KiB for example mknyszek commented on Oct 29, 2025 ContributorAuthor More actions
what importance does the 8 KiB page size holds ?
why it isn't configurable ?
what sort of pros and cons user might experience if it is set to 4KiB or 16KiB for example
It's the Go runtime internal page size. It was inherited from TCMalloc. There are trade-offs (bigger page size means more wasted memory, but smaller indexes, so faster metadata lookups), but Green Tea doesn't explore them, it just uses the existing page size.
It's not configurable because that's more configurations to test and maintain. With every configurable knob the state space of Go runtime configurations grows exponentially. This is not sustainable, so we avoid exposing configuration knobs and instead aim to provide good defaults.
The 8 KiB page size is baked deeply into the runtime at compile-time. It is constant to make doing math with the page size much cheaper, allowing the compiler to constant-fold many operations involving pages (extremely common in the memory manager) in the runtime.
It is not also easy to change. There are generated structures that depend heavily, and were tuned specifically for, an 8 KiB page size.
FWIW, the world has mostly settled on smaller page sizes anyway (4 KiB). Picking something close to that is generally a good idea. Very few new architectures are deciding to go with 64 KiB pages (popular with MIPS hardware, for example). Reacted by Piyush Katariya, pymq, David Somers and Leo Antunes zliu5-sc commented on Nov 4, 2025 hi team, @mknyszek thank you for the great work.
We're evaluating and planning to adopt Green Tea for our production workload, which is memory heavy (90~125GiB live heap at steady state; depending on region).
We've observed overall CPU utilization improvement, but with a caveat:
- for our workload, we observed that Green Tea builds runs GC less frequently , but consumes more CPU time each cycle
- net result is reduced CPU utilization overall; but when GC is forced to run more frequently (e.g. smaller
GOGC values; burst in traffic), we observe huge latency increase from the application, presumably from GC having to take more CPU time away from the application.
the follow graph (canarys are Green Tea; baseline is non-Green Tea) shows the contrast very well:
- happy case (steady heap = 85G; GOGC=80; GOMEMLIMIT = 170): Green Tea runs less frequently than non-Green Tea; CPU utilization drops 8~10% overall; small p99 latency regression
- bad case (steady heap = 125G; GOGC=75; GOMEMLIMIT = 180): Green Tea runs as frequent as non-Green Tea; no significant CPU utilization improvment (0~3%); big p99 latency regression
would appreciate your thoughts @mknyszek and also hear from any other early adopters of Green Tea in a production setting if you have experienced similar trade off.
( re: per-cycle CPU usage: we didn't capture them as metrics but from gc trace logs for Green Tea is about 12%; for non-Green Tea is about 7% )
at this point, our main blocker for adopting Green Tea is that it performs well when it is able to run less frequently (based on GOGC+GOMEMLIMIT combo), but it performs worse at higher memory stress (we are not sure if Green Tea will start to thrash on sudden traffic surge) mknyszek commented on Nov 4, 2025 ContributorAuthor More actions @zliu5-sc Thanks for reporting. It sounds like memory usage bumped slightly causing the speedup, but that it is slower overall.
Is this with Go 1.25 or Go at tip? There are lock contention issues with some workloads with the Go 1.25 implementation and the behavior you're seeing here reminds me a lot of those issues.
If you are indeed running Go 1.25, I would suggest trying at tip, where the issue is fixed. It's already enabled by default at tip, so no need to set any environment variables. (Try out https://golang.org/dl/gotip for doing so.)
Also, how are you measuring CPU utilization? OS metrics? Go runtime metrics? Depending on your answer, that will help identify if it's the issue I expect. (That is, if you're looking at OS-reported metrics measuring actual on-CPU time, then I suspect it is indeed lock contention at the root of things here for you. But we'll see, it's just one possible piece of evidence.)
If it persists at tip, any additional data you can share (from the list #73581 (comment)) would help a lot. Thanks. zliu5-sc commented on Nov 4, 2025 Thank you @mknyszek
- indeed the memory usage (as measured by container metric
container_memory_working_set_bytes; we run in GKE) is up, but still below GOMEMLIMIT (which we are happy about :P ); I agree with your assessment that current overall cpu utilization improvement likely came from this trade off
- same as above, we measured CPU utilization by
container_cpu_usage_seconds_total / <no. of cores requested>
- let me try tip
- will provide some gctrace after cleaning up
zliu5-sc commented on Nov 4, 2025 here's gctrace log excerpts from latest run. we run on GKE with c3d-standard-60 machines.
gc 442 @7361.095s 11%: 1.1+7668+0.56 ms clock, 68+37775/114987/232325+34 ms cpu, 145093->169196->103948 MB, 170214 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 441 @7342.726s 11%: 1.2+7893+0.38 ms clock, 72+41858/118381/220126+23 ms cpu, 143914->170908->108656 MB, 170288 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 440 @7321.955s 11%: 1.2+6643+0.090 ms clock, 74+98/99621/246796+5.4 ms cpu, 142774->158819->96103 MB, 166618 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 439 @7302.492s 11%: 1.3+7539+0.26 ms clock, 83+30627/112986/233373+15 ms cpu, 146710->170143->102982 MB, 170437 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 438 @7280.617s 11%: 1.0+6661+0.38 ms clock, 60+74/99877/225649+23 ms cpu, 143189->166610->103467 MB, 169601 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 437 @7261.655s 11%: 0.95+7339+0.13 ms clock, 57+30357/110073/215047+8.3 ms cpu, 146085->169856->103779 MB, 170055 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 436 @7239.909s 11%: 3.5+7410+0.095 ms clock, 212+46124/111141/210140+5.7 ms cpu, 144097->167858->104456 MB, 169971 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 435 @7218.885s 11%: 1.4+7293+0.23 ms clock, 88+1102/109378/254461+14 ms cpu, 146482->169308->103115 MB, 170282 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 434 @7199.218s 11%: 1.0+7913+0.41 ms clock, 63+1514/118631/273557+25 ms cpu, 143326->168189->105169 MB, 170157 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 433 @7180.303s 11%: 1.5+7623+0.17 ms clock, 91+13839/114336/260656+10 ms cpu, 144478->166861->103037 MB, 170040 MB goal, 12 MB stacks, 1 MB globals, 60 P
...
gc 513 @7360.437s 5%: 0.84+3680+0.43 ms clock, 50+192/55133/107229+25 ms cpu, 147456->160150->95770 MB, 171164 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 512 @7343.078s 5%: 0.94+3860+0.27 ms clock, 56+217/57896/103434+16 ms cpu, 139848->154968->97853 MB, 160238 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 511 @7327.924s 5%: 1.1+3350+0.24 ms clock, 67+156/50233/108839+14 ms cpu, 137945->146866->91558 MB, 158057 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 510 @7311.066s 5%: 1.1+3375+0.53 ms clock, 69+94/50602/111280+32 ms cpu, 136866->144846->90312 MB, 156820 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 509 @7294.836s 5%: 0.77+3471+1.0 ms clock, 46+193/52033/110712+64 ms cpu, 132700->141628->89606 MB, 152047 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 508 @7276.840s 5%: 1.0+3105+1.0 ms clock, 64+243/46548/111017+62 ms cpu, 138814->144897->86878 MB, 159052 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 507 @7260.842s 5%: 0.87+3554+0.41 ms clock, 52+75/53232/107965+24 ms cpu, 134219->144304->90881 MB, 153788 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 506 @7244.136s 5%: 0.87+3315+0.64 ms clock, 52+61/49692/112949+38 ms cpu, 135530->142844->87873 MB, 155289 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 505 @7225.843s 5%: 1.1+3243+0.35 ms clock, 67+128/48624/110580+21 ms cpu, 141333->149002->88731 MB, 161938 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 504 @7207.662s 5%: 1.0+3701+0.50 ms clock, 63+468/55507/109643+30 ms cpu, 139575->150993->92530 MB, 159925 MB goal, 12 MB stacks, 1 MB globals, 60 P
...
mknyszek commented on Nov 4, 2025 ContributorAuthor More actions The fact that OS-reported CPU utilization doesn't change much but the gctrace shows a huge difference really does suggest to me that this is the fixed-at-tip lock contention issue. The gctrace measures CPU time as monotonic time measurements at the GC worker level, which will include off-CPU time due to lock contention. The OS-reported CPU utilizations will NOT include this off-CPU time. Hence, that suggests to me that the discrepancy is off-CPU time.
Please do try tip-of-tree. If the problem persists or there's some new problem, we'll dig deeper. zliu5-sc commented on Nov 4, 2025 Thanks @mknyszek I'm in the process of importing tip into our bazel toolchain. Will report back result with tip as soon as I have them. Cautiously optimistic about it! zliu5-sc commented on Nov 5, 2025 hi @mknyszek it seems your intuition is spot on! here's a run with Green Tea on tip:
gc 459 @7398.393s 3%: 2.1+2695+0.29 ms clock, 129+375/40418/87320+17 ms cpu, 144858->151832->89434 MB, 170172 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 458 @7381.498s 3%: 1.1+2768+0.67 ms clock, 68+153/41481/87689+40 ms cpu, 145374->152900->89586 MB, 170041 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 457 @7365.164s 3%: 1.0+2911+0.23 ms clock, 62+397/43647/82004+14 ms cpu, 145119->155569->92120 MB, 170460 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 456 @7345.220s 3%: 1.0+2735+0.43 ms clock, 60+104/41008/86031+26 ms cpu, 145164->152917->90440 MB, 170065 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 455 @7330.313s 3%: 0.89+2906+0.47 ms clock, 53+70/43576/85982+28 ms cpu, 145648->155247->92120 MB, 170366 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 454 @7311.432s 3%: 1.0+2987+0.45 ms clock, 62+135/44786/86167+27 ms cpu, 144493->155049->93169 MB, 170457 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 453 @7293.173s 3%: 0.88+2725+0.26 ms clock, 53+56/40853/88622+15 ms cpu, 145176->152202->88999 MB, 170056 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 452 @7274.473s 3%: 1.1+2675+0.28 ms clock, 68+374/40115/85959+17 ms cpu, 144859->152721->89807 MB, 169886 MB goal, 12 MB stacks, 1 MB globals, 60 P
gc 451 @7257.877s 3%: 2.3+2858+0.66 ms clock, 139+290/42814/85931+39 ms cpu, 146119->155239->91702 MB, 170337 MB goal, 11 MB stacks, 1 MB globals, 60 P
gc 450 @7240.187s 3%: 1.3+2954+0.75 ms clock, 78+932/44266/83605+45 ms cpu, 145381->156200->93303 MB, 170706 MB goal, 12 MB stacks, 1 MB globals, 60 P
now the gc reported cpu time is more in line with container reported cpu time, but more importantly, it seems to have fixed our latency regression at p99 (possibly due to the same contention issue during the gc cycle)
may I seek your advice on how to adopt your fix before 1.26 release? will there be a minor release on 1.25 containing the fix? should we try cherry-pick the fix into 1.25 tree? Reacted by thepudds, Michael Knyszek, Woojong Koh and Martin Hotmann Reacted by thepudds and Woojong Koh mknyszek commented on Nov 5, 2025 ContributorAuthor More actions
may I seek your advice on how to adopt your fix before 1.26 release? will there be a minor release on 1.25 containing the fix? should we try cherry-pick the fix into 1.25 tree?
Unfortunately I don't think we'll be backporting the fix to Go 1.25 at this point, since it's just an opt-in GOEXPERIMENT in Go 1.25. The changes are too big and too invasive to cherry-pick.
My suggestion would be to either disable the GOEXPERIMENT and wait for Go 1.26, or run Go at tip for now. zliu5-sc commented on Nov 8, 2025 hi @mknyszek thanks again for your help. want to report back our finding:
as Go 1.26 is still a bit ways out, we experimented with cherry-picking the specific commit d7a38ad (and a few ancestors to resolve merge conflict).
for our workloads, this specific commit did indeed fix our p99 regression and still allow us to reap the benefits of Green Tea. mknyszek commented on Nov 19, 2025 ContributorAuthor More actions The freeze is upon us and I'm inclined to ship what we've got. It's on by default now. We'll still have the opt-out for one more release, and we can keep iterating if/when new things come up.
Closing out this issue! Reacted by Artyom Pervukhin, Simone Ragusa, thepudds, Woojong Koh, Lord-Y, serendipty01, Yonas Habteab, Martin Hotmann, Dmitri Shuralyov and mutezebra Reacted by Artyom Pervukhin, Simone Ragusa, thepudds, Woojong Koh, Yonas Habteab, ignoramous, Martin Hotmann, Dmitri Shuralyov and Elliott Fiedler Reacted by thepudds, Woojong Koh, Lord-Y, Mario Valderrama, Leo Antunes, Dmitri Shuralyov and nis057489
MetadataMetadataStatusShow more project fields
DevelopmentNo branches or pull requestsIssue actions
|
