Explore every run · Paper · Blog post · Leaderboard · Baselines · Training · Experiments · Runs · Data

ARC-AGI measures how efficiently a system picks up a new skill from a few examples, but it serves each task once. Continual-ARC serves a lifelong stream built from the three ARC-AGI editions instead, in which the same tasks recur after dormant gaps with fresh instances while new tasks keep arriving. On each instance the learner may pay for one more demonstration or submit an answer, and it may try again after a wrong one. On an instance it solves, a wrong answer and a demonstration each cost one; an instance it does not solve costs the whole budget of sixteen, however it failed, and we charge the learner the share of that budget it used.

Learning on the job can mean two things: remembering the rule of a task met before, and getting better at tasks never met. So after the stream we freeze the learner and give it two tests. Each is one round in which every instance starts from the learner's state at the end of the stream, so nothing learned on one test instance reaches another. The seen test serves fresh instances of the tasks the learner met, which measures memorization. The held-out test serves 100 ARC-AGI-1 evaluation tasks it never met, which measures generalization. Each learner is placed on the plane of the two by the share of its reset's spend it saves on each test, memorization across and generalization up, its reset being itself wiped before each instance, in its own harness and on the same instances, so a stronger model scores no higher for being stronger. Beside each share stand the budget the learner and its reset used and the share each solved, since a large share of a small spend is a small gain. A learner may keep anything at all between instances, which puts an agent harness keeping notes and a model updating its weights on the same scale.

This repository is the benchmark of the paper Continual-ARC: Measuring Skill Acquisition over a Working Lifetime: the environment, the schedules, the tracks and the measures. The learners the paper evaluates, adapters for three agent harnesses among them, are in baselines/, and the training of its trained rows is in training/. The records of every run it reports, with the trajectories behind them, are plain JSON in git in runs/, and every task, figure, run and trajectory of the paper can be explored and replayed at continual-arc.unify.ai. Every experiment the paper reports is listed in experiments/ with the code and the command that ran it and where its runs are, and the paper's own source is in paper/.

On GPT-6 Luna, a commercial model, the paper ran Prime Agent, Hermes Agent and OpenClaw through the stream with every learning mechanism they ship switched on, and again with everything they wrote deleted before every instance. Their persistence saved no measurable budget at any reasoning effort, and left to decide what to keep, they kept almost nothing that outlasted an instance: none saved a measurable share of the budget left to learn. A deliberately naive program library, which keeps one tested program per task and runs it when the task returns, saved about half of it at low effort and three fifths at medium and high. Run your own learner through it and submit the result.

• Any learner. The learner is a black box with four methods. Weight updates, external memory, program libraries and plain prompting are all welcome.
• Binary feedback. Every submission is answered correct or incorrect, as ARC Prize answers it.
• Demonstrations cost. One more input-output pair of the task is available on request, charged like a wrong attempt.
• Fresh instances. Every test input and every demonstration of the stream and the seen test is freshly drawn from the task's Re-ARC generator. A cached answer never fits the next grid, and caching a demonstration is a choice with a price.
• Composable interventions. Optional plug-ins can insert tagged auxiliary episodes without changing the schedule's exposure or gap metadata.
• Two frozen tests, one plane. The share of its reset's spend a learner saves on the seen test (memorization) and on the held-out test (generalization), with the budget used and share solved beside each, and the share learned over the stream as a secondary measure. Compute, tokens and memory are logged, not scored.

Everything behind the paper is in this repository, as plain files:

uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch     # clones Re-ARC and ARC-AGI at pinned commits
uv run pytest

Data lives in ~/.cache/continual-arc (override with CONTINUAL_ARC_DATA).

A schedule is a list of (task_id, instance_seed, phase, reset_before) slots. The environment serves one instance at a time and accepts three actions:

An instance that ends unsolved, at the attempt cap or given up, is charged the whole budget, max_cost, whatever it spent, so failing is never cheaper than asking; a solved instance is charged its wrong attempts and the demonstrations it bought. The budget used is 100 * mean_cost / max_cost, with max_cost = attempt_cap + demo_cost * demo_cap, 8 + 8 = 16, and lower is better; the score in summary.json and the report is its complement, 100 - budget used.

After the stream the learner plays two tests, each built by configs.test_schedule:

Each test is one round in which every instance is a user of its own, so every instance meets the learner as the stream left it, and nothing learnt on one test instance reaches another, since a learner changes what it carries only in end_round (regimes). A test takes about as long as one instance, and nothing tells the learner it has begun. A learner that keeps a demonstration the moment it buys one breaks that contract, and on the seen test it would pass the pair to another instance of the same task in the same round. A learner that cannot serve 100 users at once plays a test narrower, --width N users a round: then every instance is marked reset_before, and the learner's reset must restore its state at the end of the stream rather than wipe it. runner.run(..., max_concurrent=k) serves a concurrent learner's round k users at a time instead.

Both tests are read against the same learner reset before each instance, played on the same instances on its own, apart from the learner's own test, as the share of the reset's spend the learner saves:

share saved = 1 - U(learner) / U(reset) = T / U(reset)
T = U(reset) - U(learner)

with U the budget used over the test's instances, in points of the budget, and T the saving in points. The two shares place a learner on the plane of the tests, the seen test across and the held-out test up, and that plane is the headline. A share reads a learner whose reset already spends little as having gained no less, so the budget used and share solved of the learner and of its reset are read beside it: a large share of a small spend is a small gain. On the seen test every task was met in the stream, so perfect memory would spend nothing there, and its share is the share of what the test leaves to learn. The reset plays the learner's own model in its own harness, so a stronger model is not credited for being stronger, and the harness, the environment the learner learns in, is part of the baseline rather than measured against a model without it. The bare model is not a reference; a result also shows the bare model on the same instances, and a reset that plays below it is flagged beside the test. The reasons, and what remains open, are in docs/DESIGN.md.

continual-arc schedule make --config arc1 --seed 0 --no-epilogue --out stream.json
continual-arc schedule make --config arc1 --seed 0 --test held-out --out held-out.json
continual-arc schedule make --config arc1 --seed 0 --test seen --out seen.json

Playing a test from where the stream left the learner is the driver's job, as restoring any saved state is: keep what the learner carries at the end of the stream and give each test its own copy, in a run directory that holds the stream's alias-salt, so that the learner meets the stream's tasks under the names it learnt them by. For a learner whose state is plain Python:

import copy
import shutil
from pathlib import Path

from continual_arc.runner import run
from continual_arc.schedule import Schedule

from my_pkg.learners import MyLearner

learner = MyLearner()
run(learner, Schedule.load("stream.json"), out_dir="runs/my-stream")
for test in ("held-out", "seen"):
    out = Path(f"runs/my-stream-{test}")
    out.mkdir(parents=True)
    shutil.copy("runs/my-stream/alias-salt", out)        # the names the stream gave its tasks
    run(copy.deepcopy(learner), Schedule.load(f"{test}.json"), out_dir=out)

A learner that holds a model client or a process saves what it keeps instead, and loads it into a new instance for each test. The references play the same instances one at a time, --width 1, so that the runner resets the learner before each, each into a directory of its own, and continual-arc transfer reads the share saved and T from the two runs of a test, the learner's and its reset's, each with a 95% interval from a bootstrap over tasks, beside the budget used and share solved of each (--json prints them all; --also-run and --also-reset add further plays of either side on the same instances, read as their mean on each instance, as the paper reads its rows):

continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out held-out-1.json
continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out seen-1.json
continual-arc run --learner my_pkg.learners:MyLearner --schedule held-out-1.json --out runs/my-reset-held-out
continual-arc run --learner my_pkg.learners:MyLearner --schedule seen-1.json --out runs/my-reset-seen
continual-arc transfer runs/my-stream-held-out runs/my-reset-held-out
continual-arc transfer --test seen runs/my-stream-seen runs/my-reset-seen

The bare model's runs, shown beside T, come from continual_arc_baselines.systems.bare.BareModelLearner in baselines/, played on the same width-1 schedules (arc-baselines run --schedule held-out-1.json).

The budget used rewards whatever the learner brings to the stream, and most of that is its model and its harness. So a learner is read against itself reset, the same model in the same harness with its state wiped before every instance of the same stream, which carries nothing from one instance to the next. The reset is the scale's zero. Its one is perfect memory on the same system: the reset's own charge on each task's first exposure, and nothing on any return. Between them lies the learnable budget, which shrinks as the system gets stronger, and the share learned is the part of it a learner saves:

share learned = (U(reset) - U(learner)) / (U(reset) - U(perfect memory))

with U the budget used over the stream. A learner that keeps nothing scores 0, one that answers every returning task at no cost scores 1, and one worse than its reset scores below 0. Keeping the demonstrations a task was shown is a kind of memory, and a learner that does so scores above 0 for it. The learnable budget says how much a track leaves a system to learn: when it is small, the system has outgrown the track and the next edition measures it. The share learned reads memorization while it is being built, and the seen test what is left of it at the end, so it is read as a secondary measure beside the plane of the two tests.

continual-arc schedule make --config arc1 --seed 0 --no-epilogue --reset-each --out stream-reset.json
continual-arc run --learner my_pkg.learners:MyLearner --schedule stream-reset.json --out runs/my-stream-reset
continual-arc learned runs/my-stream runs/my-stream-reset

prints the share with a 95% interval from a bootstrap over tasks (continual_arc.metrics.share_learned). --reset-each serves the same stream with the learner wiped before every slot; a learner that keeps nothing between instances is its own reset. Each of the run's instances is read against the reset's play of the same instance, or, for one the reset never met (another user's, at more users than the reset served), against the reset's mean on the task.

The stream interleaves its tasks as work does: a few tasks are active at once, each is worked in a spell of a few instances and then rests, and new tasks keep arriving, so a task can rest long enough to be forgotten before it returns. One process generates it (stream_schedule): exponential spells, log-normal rests and Poisson arrivals, at the parameters each configuration sets, with every task served the same number of times. An order of instances never touches the learner's state.

The same process can serve the stream's visits one task at a time (--order sequential) or fully shuffled (--order random). The paper plays those two only to check that the order does not change its findings (its Appendix I). They are not settings of the benchmark, and a run in either is an ablation.

The stream can still end with the epilogue (--no-epilogue leaves it out): the held-out tasks once each, met while the learner still learns. No measure reads it. A learner that keeps learning through 100 held-out tasks mixes what it built on the stream with what it builds on them, and the held-out test reads the first alone. --order fresh builds no stream but the held-out tasks alone, met by a learner whose state is wiped before each, at the held-out test's own instances, so it can stand in for the held-out test's reset reference.

A configuration is served in one of two regimes, and they differ in one number, users: how many people the system serves at once, and so how much experience arrives between two updates. It is a setting of the run, not of the configuration (with_users(config, users), schedule make --users). Any learner can run at any number of users, whether it keeps notes or trains its weights, and learners are compared only at the same number. The canonical numbers are the powers of two from 1 to 16 (CANONICAL_USERS): a low-data regime by design, every update seeing at most 16 new instances, so that a learner that trains its weights between rounds finishes a whole stream on one machine in days, not weeks. A track serves as many users as its pool gives tasks of their own: 16 on ARC-AGI-1, 4 on ARC-AGI-2 and 1 on ARC-AGI-3.

In the collective regime every user brings tasks of their own (Dan, 28 Oct 2026: "one user has 25 puzzles, two users has 50, and four users have 100"): user k plays block k of the track's nested task order (configs.task_order, configs.user_tasks), which on ARC-AGI-1 is arc1's 25 tasks sorted, then 75 more drawn from the rest at the same seed, sorted, then the other 300 sorted, so that users 0 to 3 bring 100 tasks and users 0 to 15 all 400. Each user walks the individual regime's process over their own tasks with their own draw of it: user 0 at seed s walks exactly the individual stream at seed s, and user u's tasks and stream do not depend on how many users there are, so a run at fewer users is a run at more cut to its first users. Every user meets the same held-out tasks, in an order of their own. The held-out test is the same at every number of users; the seen test covers every user's tasks, 100 new instances spread evenly over them (two of each of 50 at two users, one of each of 100 at four) and one of every task at eight and sixteen, instance i a task of user i mod U, so it is made for the run's number of users (schedule make --test seen --users U). So more users bring more tasks as well as more instances at once. The users are deliberately concurrent: a round is one instance per user, all open at once, the round's records are committed together, and the learner's begin_round and end_round hooks bracket it. The contract is that whatever the learner carries between instances changes only in end_round, so every user of a round meets the state the round began with. That state is one for the whole team, whatever the learner keeps (Dan, 5 Nov 2026): a trained model's weights, a harness's store, a floor's cache, every user's instance of a round starting from it and every user's experience of the round changing it before the next. ARC holds nothing personal, and a store kept per user would set a harness that saw only its own user's tasks against weights trained on every user's. The baselines' harnesses keep one store this way, each user's writes merged into it in user order as the round ends (baselines/README.md). Every observation and record says whose work it is (user).

A learner that can serve the users of a round at the same time sets concurrent_users = True and reports its counters per user (user_resources(user)): the runner then serves every user's instance on a thread of its own, from its first action to its verdict, at its own pace. Any other learner is served one action at a time, one action on every open instance before a second on any. With one user every instance is a round of one and both are the serial loop.

Exposure index and gap are counted in rounds over the whole team (two users meeting a held-out task in the same round share both; a stream task is one user's alone), and each record also carries the user's own count (user_exposure_index, user_gap); with one user the two agree, the times a task was seen before and the steps since it was last seen. A round never spans a phase, so the epilogue starts on a round boundary and a test is a round of its own. records.jsonl only ever holds whole rounds, and a resumed run replays the interrupted round from its start. Add-ons are served one instance at a time and need one user.

Schedule.first_rounds(r) is a literal prefix of a schedule's own slots: rounds 0 to r - 1, the same tasks, instance seeds and users as the full schedule. It is a slice, not a rebuild, because the stream process gives every task a supply fixed by the configuration's full length, so regenerating a shorter stream from the same seed would not be this one's prefix. A resumed run checks its schedule's prefix slot by slot (runner.run(resume=True)), so a run stopped under first_rounds(20) resumes without replay under first_rounds(60), or under the uncut schedule.

# the first 20 rounds of the stream served to 16 users
continual-arc schedule make --config arc1 --users 16 --seed 0 --no-epilogue --rounds 20 --out stream-r20.json

The frozen tests need none of the stream, only the configuration and seed, so a learner can play them from its state at any round that was kept, as from the end of the stream. protocol_of lists a cut stream as a deviation from the canonical protocol ("stream cut at round 20"); meta.config is kept, so the label stays readable.

Continual-ARC has a track for each ARC-AGI edition, and each track has one configuration, named as the track is, beside a smoke configuration for tests:

Each track has one canonical protocol: its configuration's stream and then its two frozen tests, at the default environment settings, at each canonical number of users. A result is a point on the plane of the two tests' shares saved, with the budget used and share solved beside them and the share learned over the stream as a secondary measure. Seeds 0, 1 and 2 are all canonical; the paper reports seed 0, and the leaderboard compares runs at seed 0. Every run's summary.json and meta.json carry a protocol block that names the track and says whether the run is canonical; any other configuration, order, seed or environment setting is an ablation and the report labels it as such. continual-arc report prints the block a run recorded, as the release that played it judged it.

uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue --out stream.json
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --out held-out.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --out seen.json
uv run continual-arc schedule show stream.json --head 40
uv run continual-arc run --learner my_pkg.learners:MyLearner --schedule stream.json --out runs/my-stream
uv run continual-arc report runs/my-stream --plots
uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue --reset-each --out stream-reset.json
uv run continual-arc run --learner my_pkg.learners:MyLearner --schedule stream-reset.json --out runs/my-stream-reset
uv run continual-arc learned runs/my-stream runs/my-stream-reset   # share learned over the stream
# each test played from the end of the stream, and the learner reset on it (the two frozen tests, above)
# the headline: the share saved on each test, generalization (held-out) and memorization (seen)
uv run continual-arc transfer runs/my-stream-held-out runs/my-reset-held-out
uv run continual-arc transfer --test seen runs/my-stream-seen runs/my-reset-seen

from continual_arc.env import LearnerObservation, Submit, RequestDemos, GiveUp
from continual_arc.learner import BaseLearner

class MyLearner(BaseLearner):
    def reset(self):                          # wipe persistent state
        ...
    def act(self, obs: LearnerObservation):   # obs.task_id (an alias), obs.input, obs.attempts_used, obs.demo_cap, ...
        return RequestDemos()                 # or Submit(grid) or GiveUp()
    def observe(self, obs, action, feedback):
        ...                                   # SubmitFeedback / DemosFeedback / GiveUpFeedback
    def end_round(self):                      # what it carries between instances changes only here
        ...
    def resources(self):                      # cumulative counters, logged per instance
        return {"tokens": ..., "state_bytes": ...}

The runner shows a learner only the instance (LearnerObservation): the task under a name drawn for the run, the input, its own attempts and demonstration requests against their caps, and its place in its own work (step, round, user), which it could count itself. It is told nothing about the schedule: not the phase, how often the task has come before, whether its state was wiped, or how long the stream is. The name is an alias of the task's id (task-3f2a9c1e, or a codename with --alias-style names), salted per run in the run directory's alias-salt, because the real id is a lookup key: a learner with no model that runs the Re-ARC verifier named by the id, and finds each held-out input among the public evaluation pairs, would use none of the arc1 budget. The records keep the real ids. Every learner gets this view, the baselines' own adapters and trained rows included; runner.run takes the name style as alias_style.

The runner logs per-instance records (records.jsonl) and a summary (summary.json) with the budget used, acquisition cost, retention by gap, cost by exposure index, the learning curve, and the budget used on each test.

The stream is public and deterministic, so a learner could be built to know it. A result is verified only once we have rerun it from its code and read the code against these rules:

• Nothing about ARC brought in. No ARC, Re-ARC, ARC-GEN or other task data, answers or solvers inside the learner, its prompts or its starting memory, and no rebuilding of the schedule. What a model learned in pretraining is its prior, which the learner's own reset on the same model already carries.
• A published student and teacher, used unchanged. The student, and the teacher if there is one, are published checkpoints used as published and named exactly: an OpenRouter model id, or open weights as their Hugging Face repository at a commit (org/name@commit). A result is compared only with others on the same student and teacher, so a model changed before the stream is another student, standing alone: one made to start badly would save a large share of its reset's spend by recovering alone. Read every share beside the budgets it comes from, since a large share of a small spend is a small gain.
• Every model named. Every model the learner calls is named in the submission and reached through the tracking proxy of baselines/. Every measure reads the learner against itself reset, on the same models. A result names the model that plays (the student) and any model that only teaches it (the teacher), such as a stronger model writing its notes or labeling its games for training; the teacher's calls are part of its training and count in its training compute, shown beside the score.
• Learning only from what it was shown. A learner learns from its observations and feedback. The records, with their real ids and exposures, are the benchmark's.
• Anything else is allowed: memory, programs, tools, a cache, training the weights. Compute is logged and shown beside the score, never in it, and a learner that trains reports its training compute apart from what it costs to serve an instance.

The alias stops the free lookup and a model's recall of a famous task id; it makes a determined lookup cost a demonstration or two per task, since a learner can still match a bought pair against every solver. The rules and the rerun stop that, and a verified result is rerun on a held-out stream of new rules, which no lookup reaches (verified results).

A plug-in implements continual_arc.addons.Plugin. The runner calls hooks at run start and end, episode start and end, every observation, every action, and every feedback response. A hook can return a PluginResponse to transform the value seen by the next plug-in or by the learner, enqueue auxiliary episodes, attach diagnostic metadata, emit metrics and events, or stop after the current episode.

The environment still receives the final action and computes the result from its own task data. Plug-ins cannot change the score directly. Auxiliary episodes have episode_type: "addon" and appear in the separate addons summary block, while the original schedule and primary benchmark metrics remain separate.

Addon and BaseAddon remain available for older episode-completion add-ons. They are adapted automatically by PluginManager. New interventions should use Plugin or BasePlugin, and multiple plug-ins can be passed through the runner's plugins argument.

The command-line runner accepts repeatable --plugin and --plugin-kwargs arguments. The arguments are matched by position:

uv run continual-arc run \
  --learner my_pkg.learners:MyLearner \
  --schedule stream.json \
  --plugin my_pkg.plugins:ReviewPlugin \
  --plugin-kwargs '{"initial_interval":8}'

src/continual_arc/
  grids.py       grid validation and equality
  data.py        pinned fetch of each track's upstreams: Re-ARC and ARC-AGI-1; ARC-AGI-2, ARC-GEN and ARC-AGI-1; ARC-AGI-3's games
  rearc.py       seeded, verified access to the Re-ARC generators
  arcgen.py      seeded access to ARC-GEN's generators and the screened ARC-AGI-2 pool (arcgen_screen.json)
  tasks.py       task sources (generative Re-ARC and ARC-GEN, fixed ARC JSON) and registry
  arc3/          the ARC-AGI-3 track: the pinned games, a generator and solver per game, levels served fresh by Arc3Env
  schedule.py    Slot/Schedule, the one stream generator and its order presets, the phases and tests
  configs.py     named configurations, build_schedule() and test_schedule()
  env.py         the environment: observations (the slot's, and the learner's view of the instance), actions, feedback, cost accounting
  aliases.py     the per-run names a learner sees for tasks
  addons.py      plug-in protocols, composition, and intervention runtime
  learner.py     the learner protocol
  runner.py      run a learner over a schedule, write a run directory
  metrics.py     the budget used, the share saved and T on the frozen tests, the share learned and the diagnostics
  report.py      text reports and plots
  cli.py         continual-arc command line
  testing.py     data-free task source and trivial learners for tests
docs/DESIGN.md   design rationale and decisions
baselines/       the learners the paper evaluates, its harness adapters and the tracking proxy (baselines/README.md)
training/        the training of the paper's trained rows on Qwen3.5-9B (training/README.md)
experiments/     every experiment the paper reports: the code and command that ran it, and its runs (experiments/README.md)
runs/            the records and trajectories of every run the paper reports (runs/README.md)
data/            every instance the runs on ARC-AGI-1 and ARC-AGI-2 were served, with their schedules (data/README.md)
paper/           the paper's LaTeX source, as posted to arXiv, and the code that computes its tables and figures (paper/README.md)

The leaderboard opens with the paper's runs and presents any verified submission of a genuine learning algorithm, novel or not. A result is a learning method on a student model and, if it has one, a teacher model, placed on the plane of the two tests' shares saved. A result on the public stream is a claim until we have rerun its code on the held-out rules, which nobody outside unify has seen, so nothing goes on the leaderboard until your learner is verified (verified results).

1. Run your learner on the track's canonical protocol at seed 0 (arc1, arc2 or arc3): its stream with continual-arc run, then the two frozen tests from where the stream left it, and your learner reset: on the same stream, wiped before every instance (schedule make --reset-each, the share learned), and before each instance of the tests (the two frozen tests). To drive an agent harness, start from the adapters and the tracking proxy in baselines/, which count every model call whatever the harness does. Read what a learner may use first.

2. Unless the leaderboard already has it, run the bare model you play with on the two tests, one instance at a time, from the baselines (continual_arc_baselines.systems.bare.BareModelLearner): two cheap runs, shown beside each test, which is flagged if your reset plays below them. The bare model's tests of every model the paper ran are in the run records.

3. Copy every run directory the runner wrote to results/<track>/<name>/, and add a submission.json to your learner's stream, naming your learner, the model that plays (student, its OpenRouter id, or org/name@commit for open weights on Hugging Face) at its reasoning effort, the model that teaches it (teacher, named the same way, or null), your reset's stream and each test's three runs:

   {
     "name": "my-learner, as shipped",
     "learner": "my-learner 1.4.0, with its memory switched on",
     "student": "openai/gpt-6-luna",
     "effort": "medium",
     "teacher": null,
     "code": "https://github.com/you/my-learner",
     "contact": "[email protected]",
     "reset": "results/arc1/my-learner-stream-reset",
     "tests": {
       "held-out": {"run": "results/arc1/my-learner-held-out", "reset": "results/arc1/my-learner-reset-held-out",
                    "bare": "results/arc1/bare-luna6-medium-held-out"},
       "seen": {"run": "results/arc1/my-learner-seen", "reset": "results/arc1/my-learner-reset-seen",
                "bare": "results/arc1/bare-luna6-medium-seen"}
     },
     "notes": "anything a reader needs to know to read the number"
   }

4. Check it: uv run python scripts/check_result.py results/<track>/<name>. It rebuilds every run's schedule from its configuration, order or test, and seed, checks that its records cover every slot and that its summary is the one the records give, that your tests were played under your stream's task names, that your reset is a run of your learner and the bare runs are the bare model's, and prints the numbers the leaderboard will show: your place on the plane, the share saved on the seen and the held-out test with their intervals, T and the budget used and share solved by your learner and its reset on each, with the flag if your reset plays below the bare model, then the share learned over the stream.

5. Have your learner verified, then open a pull request with these runs to stand beside its verified result. Questions go to this repository's issues.

Verified runs take all three tracks, each on a private set of its own that nobody outside unify has seen. On arc1 and arc2 it is 50 new ARC-style rules, each a generator of fresh instances, checked against every public ARC task: 25 streamed over 200 instances, as the track streams its 25 tasks, and 25 held out. On arc3 it is new games on ARC Prize's engine, each a generator of fresh levels with a solver, streamed and held out as arc3 serves its own: 8 fresh levels of each stream game and 4 of each held-out game. If a track's private set is withdrawn, a learner is rerun on its public tasks at a seed of the run's own instead, and the result shows as reproduced, not verified, since their answers are public. To have a learner verified:

1. Open an issue with the Verify my learner form: the track, your public GitHub repository and the commit to run, the import path of your learner class (installable from the repository with pip, implementing continual_arc.learner, and built with no arguments, so its settings are its class's defaults), every OpenRouter model it calls with its reasoning effort (the one it plays with, the one that teaches it if one does, and any others), the budget for its model calls in whole US dollars, from 5 to 1,000 (on arc3, whose games cost far more than grids, at least 114 with one model and 21 more for each other), and the country you pay from. Each is a model as OpenRouter lists it, with no variant such as :online, and with an endpoint that keeps no data; a router or a model that searches the web as part of a call is not taken. A harness adapter on the baselines' SystemLearner depends on continual-arc-baselines @ git+https://github.com/unifyai/[email protected]#subdirectory=baselines alone, which brings the benchmark with it.
2. Pay the invoice we post on the issue, by card on Stripe's page: the budget, OpenRouter's 5.5% fee on buying that credit, and Stripe's fee on the payment. We make a key for your run on our own OpenRouter account, capped at the budget, which pays for your learner, its reset and each model you name played bare. What the run does not spend goes back to your card, less Stripe's fee. A person, and a card, may have 10 runs on each private set.

Once it is paid and a runner is free, the issue says the run has started: one run plays at a time on each runner. We install your repository and run your learner, whole, inside a sandbox on a machine of its own, deleted when the run ends, which has no public network and holds neither our rules nor the run's key. Its install reaches PyPI and GitHub alone, through a filter, and nothing else once the install ends. Your learner reaches nothing but the tracking proxy, on our side, which makes every model call on the run's key, as a plain chat completion to a model your form names at the effort it names, through a host on our list that keeps no data, the same one for every call to that model in the run (its maker's own where it keeps no data, else the cheapest that does, named in the result), so that your learner and what it is read against are served alike, each instance's calls opening with a mark of their own so that no provider's cache carries one instance's prompt to another, and stops the run once its budget is spent; your code never holds the key. The proxy drops any other field or header, and refuses any other model, endpoint or URL for a provider to fetch and any tool that runs at the provider, and the issue says how many calls it refused. In the sandbox new namespaces, io_uring, BPF and the like are refused, so a harness adapter on the baselines starts no sandbox of its own there, and any other nested sandbox would fail. It may hold 8,192 processes and 64 GiB in one file, and a run stops at 120 hours. At a seed of its own, your learner plays the private set's stream and then its two frozen tests from where the stream left it: the seen test, four fresh instances of each streamed rule or game, and the held-out test, one instance of each held-out rule or four fresh levels of each held-out game. Your learner plays them one at a time, whatever it is: before each one we restart the machine it runs on and put its working files back as the stream left them, so what it carries from the stream to the tests is what it keeps as files, and a learner whose memory lives only in its process plays the tests from nothing. Only once it has finished does anything else play. Your learner reset plays the private set's stream wiped before every instance, for the share learned, and the same test instances one at a time, wiped before each, which both tests are read against; each model you name plays the test instances bare too, and a test is flagged beside it when your reset plays below one of them that costs no more than the reset, or below the cheapest, when your learner plays no better than one that costs no more than your learner, or when your learner sends a larger share of its calls to a dearer model than its reset does. Spend outside any instance counts as the learner's. The issue then gets your place on the plane: the share saved on the seen and the held-out test, each with its 95% interval, T and the budget used and share solved by your learner and its reset on each, then the share learned on the stream with its interval, the learnable budget, the budget used and share solved on the stream and what each run's model calls cost, shares rounded to a hundredth, points to a whole point and dollars to a whole dollar, and the result joins the leaderboard as verified, or as reproduced on a track's public tasks, beside your public-stream result if you sent one. The key is deleted when the run ends. Only the numbers come back: the records, and anything your learner kept, stay with us, since they show the held-out tasks.

You pay ahead for your model calls; we pay for the machines. A run takes hours to days, like one on the public stream.

A learner that trains its weights plays on GPUs we rent on Modal, and you pay for those ahead too. The form asks for the open weights it starts from, as a Hugging Face model at a commit, the GPU and how many per sandbox, and a budget for GPU time from 5 to 10,000 US dollars, charged at Modal's price for that GPU with nothing added. Your learner, its reset and its reset's stream each play in a Modal sandbox of their own, with no network at all, built from your repository, the baselines and those weights, which your learner finds read-only at the paths in ARC_READ_PATHS. They reach our tracking proxy alone, as on any run. We keep a copy of the stream's sandbox at the end of a round once an hour has passed since the last, and if Modal ends the sandbox, the stream goes on from that copy in a new one, so your learner has to start again from its files at the end of any round, as a resumed run does. Before each test instance a new sandbox starts from the stream's last copy, so every instance begins from the files and weights the stream left and nothing passes from one to the next. GPU time on the stream is your learner's training compute, shown beside the score and never in it. GPU time during a test, your learner's start in each instance's sandbox included, counts in that test's cost, and the run stops if the GPU budget is spent. What it does not spend goes back to your card with the rest.

runs/ holds every run the paper reports, under CC BY 4.0: the records of the 228 runs of its GPT-6 Luna grid and of its open-model runs on all three tracks, each in the layout above, and the trajectories behind the GPT-6 Luna runs and the open-model runs the site replays: every message the benchmark sent and every reply, the harnesses' reasoning summaries and tool calls, and what each learner kept, file by file. Its README gives the registries and the layout. The v0.1.0 release carries the same records and the GPT-6 Luna trajectories as two archives, and the runs the site replays can be stepped through instance by instance at continual-arc.unify.ai/runs. Every GPT-6 Luna run but the fresh ones played the two frozen tests after its stream, most workstreams runs after a learning-on epilogue on the held-out tasks, which no measure reads now. Each run rebuilds slot for slot from its own configuration, and every one of their streams is the one arc1 builds today. They name it arc1-small, as every run played before v0.6.0 does, when the benchmark still named its configurations by size. Every instance the runs on ARC-AGI-1 and ARC-AGI-2 were served, grid by grid, is in data/, with the schedules that served them.

@article{lenton2026continualarc,
  title   = {Continual-ARC: Measuring Skill Acquisition over a Working Lifetime},
  author  = {Lenton, Daniel},
  journal = {arXiv preprint arXiv:2611.14505},
  year    = {2026}
}

Apache-2.0. Re-ARC is MIT and ARC-AGI is Apache-2.0; both are fetched, not vendored, and the instances in data/ drawn from them keep their sources' licenses.

Continual-ARC is made by unify. unify is building continual learning for the agents companies have already built, training the model to drive their own harness, so the agent gets faster and cheaper the more it runs, on a model they own. We publish the research behind it.