A pipeline and web app that tracks the U.S. government's declassified UFO / UAP file releases. It checks for new releases, downloads the files, extracts their text (OCR for scanned pages), classifies what each record is about, summarises every event each record describes and sorts those into a hierarchy, looks for connections across records (waves of reports, recurring details, clusters of similar cases), and shows it all in a visual site anyone can browse.

The main source is PURSUE (Presidential Unsealing and Reporting System for UAP Encounters) at war.gov/UFO. The Department of War has published six tranches there since May 8, 2026 (447 records as of the Sept 18, 2026 release), and more are expected.

 ┌───────────┐  ┌────────────┐  ┌────────────────┐  ┌──────────────┐  ┌────────────────┐  ┌─────────┐
 │ discover  │─▶│  download  │─▶│ extract text   │─▶│  classify    │─▶│  connections   │─▶│ web app │
 │ CSV index │  │ direct, or │  │ PyMuPDF / page │  │ rules always │  │ details, waves │  │ FastAPI │
 │ diff vs DB│  │ Wayback,   │  │ + Tesseract    │  │ + Claude opt.│  │ places, links, │  │ + JSON  │
 │           │  │ zip bundle │  │   OCR          │  │              │  │ clusters       │  │         │
 └───────────┘  └────────────┘  └────────────────┘  └──────────────┘  └────────────────┘  └─────────┘

Discovery (ufo/sources/pursue.py): the PURSUE page is built from one CSV index (/Portals/1/Interactive/2026/UFO/uap-data.csv) with a row per record: ID, title, type (PDF / video / image / audio), agency, incident date and location, description, and file link. Each run reads the current CSV URL off the landing page (it gets a new cache-busting query string with each release), parses it, and diffs it against the database:

• new records are queued and processed;
• records whose published metadata changed are re-classified (and re-downloaded if the file link changed);
• new release dates become new Release rows, numbered in order.

Fetching (ufo/fetch.py): government sites sit behind Akamai, whose bot check refuses requests from cloud and datacenter addresses unless they look like a browser's: every request carries the Sec-Fetch-* fetch-metadata headers and Accept-Encoding that browsers send, without which war.gov answers 403 "Access Denied". Every request tries the official URL first and falls back to the Internet Archive's byte-identical copy (web.archive.org/web/<ts>id_/<url>). Files that are in neither place are marked unavailable and retried on every run. As a last resort, the pipeline downloads that release's official zip bundle (linked from the PURSUE page) and extracts the missing files by name. It also asks the Archive to capture anything missing, so those files usually turn up on a later run.

Extraction (ufo/extract.py): PDFs are read page by page with PyMuPDF. A page with (almost) no embedded text, or a page that is mostly a scanned image under a thin text layer, is rendered at 300 DPI and OCR'd with Tesseract. Each page stores the method used (text / ocr / empty) and the OCR confidence, and the web app shows both. OCR_MODE=always OCRs every page and keeps whichever version reads better. Some released files carry a poor government OCR layer, and this setting fixes those.

Classification (ufo/classify/): every record gets labels in a fixed taxonomy (ufo/classify/taxonomy.py):

• The keyword rules classifier always runs and needs no API key. It uses the publisher's title and description (high weight) and the extracted text (the number of matches required grows with document length). Some phrases only count in the publisher's text: "Air Force Base" and "Squadron" are on the letterhead of every Blue Book file, and "military" alone or a branch name does not make a military encounter. A sensor the text mentions in passing is ignored when the incident predates it (no infrared / FLIR before 1965, no satellites before 1957). When the rules change (RULES_VERSION in ufo/classify/__init__.py), records they classified are re-labelled on the next run.
• When an LLM key is set, the model classifies each record from its metadata and extracted text using structured outputs. That adds a plain-language summary, key points, a 1–5 significance score, and the places, people and organizations named. If a call fails, the record keeps its rules-based result.
  • Provider: whichever key is present. ANTHROPIC_API_KEY selects Claude (claude-opus-5, with the API's server-side refusal fallback). OPENAI_API_KEY selects OpenAI (gpt-6-luna, reasoning effort none). With both set, Claude is used unless LLM_PROVIDER=openai.
  • When a key is added or the model changes, records classified by the rules or another model are re-classified on the next start.
• The same model tags sighting accounts. The keyword rules pick candidate paragraphs; the model reads each one and returns the details it actually reports, with the words each came from. It drops blank forms, instructions, names and explanations. Results are cached by text and model, so each paragraph is sent once. Paragraphs whose call fails keep their rule-based tags and are retried on the next run.

Connections (ufo/analyze/): after each run that brings in new or changed records, the pipeline re-analyses the whole collection (about 30 seconds for the current ~450 records and 10,000 pages):

• Sighting pages. Only pages that read like sighting reports are analysed (enough words such as observed, object, sighted, altitude), plus each record's own description. Contracts and research papers (including the AAWSAP DIRD series, whatever document type a classifier gives them) only count where a page reads strongly like a report, three times the usual bar, and their own description, dates and published location (a contractor's office) are never sighting evidence.

• Recurring details (features.py). Pages are searched for concrete observables: halo or glow, flashing, coloured lights, metallic surface, humming, silence, smells, hovering, extreme speed, right-angle turns, splitting objects, entering water, no wings or exhaust, trails, formations, electrical or engine interference, instrument anomalies, effects on witnesses, ground traces, radar, and infrared. Each hit stores the sentence it came from. A sensor is not credited on a page whose own year predates it (no infrared before 1965), and "no sound, except ... a rumbling roar" does not count as silent. Two kinds of false match are dropped: negations ("no sound", "no exhaust trail") and empty form fields ("Odor detected: None").

• Dates and places (dates.py, places.py). Dates are found in OCR'd forms ("July 7, 1947", "07JUL47", "7/7/47"). Places are U.S. states (with well-known places such as Roswell, Mount Rainier, Wright-Patterson) and countries. Bare "Washington" is treated as the FBI letterhead, not a sighting place.

• Waves. Sighting pages are counted per year from the dates they mention. A wave is a year with at least 2.5× the median of the surrounding two decades. For each wave the analysis lists the peak month, the details that were more common than usual, and the places. Waves found so far: summer 1947 (13× the usual level), July 1952, June 1954, March 1966 (Michigan; radar and humming stand out), 2019–23, and 2025.

• Reported together. Pairs of details that appear on the same page more often than chance ("lift"), e.g. no wings + silent.

• Sighting accounts (events.py). Each sighting page is cut into accounts: paragraphs, or one report form. Each account is tagged with what was observed, in eleven dimensions: shape, colour, light (glow, halo, flashing, beams), sound (silent, hum, whoosh, roar, whistle, bang), movement (hovering, sudden acceleration, abrupt turns, wobbling, spinning, vertical climb, landing, pacing a vehicle, vanishing), structure (no wings, trail, dome, windows, rim), effects (engine failure, instruments, witnesses, ground traces, odour, heat, animals), number of objects, duration, time of day, and how it was observed. The tagger skips negations ("no sound"), printed form labels and questions ("b. Roar, whistle, whoosh.", "Did the object hover?"), option lists ending in "etc.", blank questionnaires, and shapes named only in an explanation ("probably a meteor"). An account needs at least two details of the phenomenon to be kept.

• Sighting types, matches and links (signatures.py). This step uses only the event tags. Agency, archive series, place, decade and wording play no part. Each tag is weighted by its rarity and its dimension (movement, sound and effects count most; time of day and duration least).

  • Two accounts match when they share at least two details of the phenomenon itself, one of them about its light, sound, movement, structure or effects.
  • Quotes shown for a detail, a type or a connection are the most readable ones (a score for the share of real words in OCR text); a connection whose matching passage is garbled OCR links to the page instead.
  • A sighting type is a combination of two or three details that recurs in at least three records far more often than chance ("Flashing · Hovering · Red/orange"). Every account of a type has all of its details, and none that contradict them (an account tagged both silent and roaring is never in a "silent" type).
  • The map places every account by tag similarity (t-SNE).
  • A record's similar records, and the cross-links between records from different agencies or at least 15 years apart, come from their best-matching accounts. They need at least three shared details, or a copy of the same report. Files from the same published series don't count.

• Explained vs unresolved (outcomes.py, verdicts.py). Every classified record carries a stated verdict from the publisher's description (and the file text when an LLM classifies): unresolved, explained (balloon, aircraft, birds, astronomical, camera artifact, hoax) or no assessment given. A derived verdict is also read from every sighting page by keyword rules, the way the close-encounter kinds are: "evaluated as a weather balloon", "turned out to be Venus", "listed as unidentified", "could not be identified". Questions, negations, speculation ("may have been caused by") and witness impressions ("appeared to be a balloon", "they thought it was a plane") do not count. By default a record with no stated verdict takes its derived one and the two are counted together; /patterns?derived=0 shows stated verdicts only. Every derived verdict is quoted with its sentence, and one that contradicts the stated verdict is shown but not used. Sighting accounts that state their own verdict (or whose page settles on one) are compared too. The summary splits the groups by decade, agency, region, document type, shape, sensor, witness and domain, and lists the details the unresolved records report more often than the others (share divided by share, "lift"). Explained records get the same treatment once at least 8 of them have sighting details; until then they are listed one by one. Blue Book record cards state their conclusion as a ticked box, which OCR does not keep, so most historical verdicts are out of reach of the rules.

• Redaction (redaction.py). Two measures: the publisher's redaction flag (mostly the modern Department of War reports and videos), and markers left in the text of every page: FOIA exemption codes ("(b)(1)" national security, "(b)(3)" statute, "(b)(6)" personal privacy, "(b)(7)(C)" law-enforcement privacy; the short "b6" / "b7C" stamps FBI files use) and blacked-out blocks. Legal citations in contracts ("paragraph (b)(1) of this clause") are not counted. A record with at least 2 markers per page is heavily redacted. The summary shows the rate by agency, decade, kind, topic, shape, sensor, witness and region, which exemptions each agency cites, the common threads of the heavily redacted records (lift over the rest), and the most heavily redacted files.

• Close encounters (encounters.py), in Hynek's three kinds. Every sighting account is tested from its text and tags: first kind when the object came within about 150 m (a distance under 500 feet, landed, on the ground, directly overhead, treetop height), second kind when it left physical effects (any effect tag), third kind when occupants or beings were seen with it (not the occupants of a car). Each kind gets its decades, agencies, places, the details it reports more often than the other accounts, examples and a page listing every account (/patterns/encounters/{ce1,ce2,ce3}). A kind with fewer than 8 accounts or 3 records is listed rather than compared.

• Episodes (episodes.py, hierarchy.py). Every record is cut into episodes: one for each distinct event it describes, each summarised in a sentence or two and classed by what happened. A case-file section that lists a dozen sightings gets a dozen episodes; a report that runs over several pages is one episode listing all of its pages; a record that reports no event (a policy memo, a public letter, a study) gets one episode of class no specific event. Each episode carries the pages it was read from, an event class, the details it reports (the event tags above plus context: over water, from an aircraft, near a military or nuclear site, landed, occupants, crash, debris, police or military witnesses, policy / study / inquiry / claims), the year and place when the text states them, and the outcome the record gives (unexplained, identified with the explanation, hoax, or none stated).

  The hierarchy is derived from those fields in code, so it can be re-tuned without re-reading the files. The top level is the event class:

  Class	Subcategories (first match wins)
  Object or light seen in the sky	seen and tracked on radar · over water or from a ship · seen from an aircraft · several objects or a formation · fireball · structured object · hovering, speed or manoeuvres · lights at night · other
  Detected by instruments	infrared / targeting-pod video · radar track · sonar · photograph or film · telescope or theodolite · other
  Close encounter or physical effects	occupants seen · engine, electrical or instrument interference · effects on witnesses · animals reacted · landed · paced a vehicle · at close range · other
  Physical evidence found	crash or wreckage · fragments or material · markings on the ground · object found on the ground · other
  Observed from space or in orbit	seen by an astronaut · anomaly in a space photograph · tracked in orbit · other
  No specific event	policy and programs · analysis and studies · public letters · contact and conspiracy claims · other

  Where an episode comes from, in order of precedence:

  1. Hand-read (data/episodes/curated.json.gz): every record in the bundled snapshot was read page by page and classified by Claude in an editing session, not by keyword rules. The file is keyed by record id and loaded on every analysis; python -m ufo episodes check reports records without an entry or whose text has changed since it was read.
  2. LLM: with an API key set, records that have no curated entry (new releases) are read by the model with the same guide and vocabulary (llm_episodes.py). Results are cached by text and model, so a new release costs only its own records. LLM_EPISODES=false turns this off.
  3. Rules: otherwise each tagged sighting account becomes an episode with a summary written from its event tags ("A pilot saw a silvery disc-shaped object in daylight, tracked on radar. It hovered, accelerated away suddenly and made no sound."), and a record with no accounts gets one episode from the publisher's description.

  To extend the hand-read set (after a release, say), python -m ufo episodes export-input DIR --only-new writes the new records' sighting passages as JSON batches with a GUIDE.md; a reader (a person, or Claude) writes one output file per batch; python -m ufo episodes import DIR validates the files against the taxonomy and merges them into the curated file.

Run it by hand with python -m ufo analyze.

uvicorn ufo.web.app:app serves:

• Overview (/): totals, the latest release's most notable records, records per release, incident decades, and breakdowns by topic, type, assessment, shape, witness, sensor, region, location, agency and program. Every bar links to the matching filtered list.

• Releases (/releases): a strip plot of each release's incident years (one dot per record), each release's share by agency, and a release × topic heatmap. Each tranche is then listed with its main topics, agencies and highlights.

• Patterns (/patterns):

  • a timeline of reports with the waves shaded, and a card per wave;
  • recurring details with decade sparklines;
  • a decade × detail heatmap and the pairs of details reported together;
  • rare details with their quotes;
  • a U.S. tile map and a world breakdown;
  • sighting types on a map of sighting accounts, highlighted on hover, each with its own page listing every account (/patterns/types/{id});
  • connections across agencies and decades;
  • explained or unresolved: how the verdicts split by period, agency, region, type, shape, sensor and witness, what the unresolved cases report more often, and every verdict read from the text with its sentence (/patterns/verdicts/{explained,unresolved});
  • close encounters of the first, second and third kind, with what each kind has in common and every account (/patterns/encounters/{kind});
  • what the release withheld: redaction by agency, period and subject, the exemptions cited, and the common threads of heavily redacted files.

  A page-wide switch (?derived=0) leaves out everything read from the text (derived verdicts, redaction markers, close-encounter kinds) and shows the publisher's verdicts and redaction flags alone. Derived values are marked wherever they are counted.

  Every mark links to /patterns/evidence?feature=&year=&place=, which lists the matching sighting pages with quotes; without a query it lists every detail, wave and place to pick from.

• Events (/events): what happens in the files, one episode at a time. An org chart of the hierarchy (class, then subcategory, with counts and share bars), a map of every episode as a dot packed inside its subcategory and class circles (hover for the summary, click to open the record at that page), how each class ends (outcome stated by the record), the mix of classes by decade, and a card per subcategory with an example. /events/{class} lists each subcategory with a few examples; /events/{class}/{subcategory} lists every episode, grouped by record. Each record page has a "What happens in this record" section with its episodes and their place in the hierarchy. /api/events returns the hierarchy and every episode as JSON.

• Connections (/patterns/links): every cross-link, with its best-matching pair of accounts. /patterns/compare?a=&b= compares any two records account by account (with a record missing it asks for two, and offers the listed connections). Each matching pair shows:

  • the details both describe, highlighted in the text;
  • the details only one of them mentions;
  • whether the two are copies of the same report;
  • dates and places both pages mention.

  Archive metadata is kept in a separate, collapsed section and is not used for matching.

• Browse (/documents): full-text search across titles, summaries and OCR'd text, with filters by release, agency, media type and any label. With a search term the list is sorted by best match (a hit in the title or id outranks one deep in the text), each card shows where the term occurs, and common spellings are tried together (tictac, tic-tac and tic tac are one search). Records whose title is an archive filename get a readable one (18_6369445_General_1948_Vol_1 is shown as "General 1948 Vol 1"; the id keeps the published form), and place names are normalised to one spelling.

• Document (/documents/{id}): metadata, summary, labels, related records, the details found in the record (with quotes), any verdict its pages state (marked as derived, with the sentence), its tagged sighting accounts, similar records elsewhere and the sighting types it contains, a link to the original, and the extracted text page by page, with OCR pages and their confidence marked. When the rules-based summary is just the opening of the publisher's description, the rest of the description folds under it. Contracts, letters and studies show a "Document date" rather than an "Incident date". A record whose file the government site is still blocking says so in plain words and links to the original; the fetch errors are listed on the pipeline page.

• Pipeline (/pipeline): records by processing stage (waiting to download / extract / classify), recent runs with logs, and the files that could not be fetched.

• JSON API: /api/stats, /api/patterns, /api/events, /api/documents?q=&release=&agency=&media=&tag=facet:value, /api/documents/{id}. Timestamps are UTC and end in Z. POST /api/pipeline/run with Authorization: Bearer $ADMIN_TOKEN triggers a run. The generated API docs (/docs, /openapi.json) are off unless API_DOCS=true.

• Admin (/admin): the Run pipeline and Recompute patterns buttons, behind HTTP Basic auth (any user name, ADMIN_TOKEN as the password). It is not linked from the site and answers 404 when no token is set; ten wrong tokens in a minute from one address get a 429.

• Dates are shown as 2 Jul 1952 everywhere. Unknown pages and bad ids get an HTML error page with a way back (the API keeps JSON errors); a page number past the end redirects to the last page. /robots.txt and /sitemap.xml list every record page for search engines.

The web process runs the pipeline in a background thread on startup and then every PIPELINE_INTERVAL_HOURS (once a day by default), counted from the last finished run so redeploys don't trigger extra runs. Runs are guarded so they never overlap.

Every page carries a canonical link and Open Graph / Twitter card tags, so a shared link gets a preview (static/og.png, drawn by tools/og_image.py), and its title says what it shows: "FBI records · Browse", "Halo / glow evidence", the record's own title.

1. Create a project from this repo. Railway builds the Dockerfile, which installs Tesseract, and uses railway.json, which sets the /healthz healthcheck.
2. Add a Volume mounted at /data, which is where the SQLite database and downloaded files live. Alternatively, add a Postgres service and set DATABASE_URL=${{Postgres.DATABASE_URL}}. Downloaded files still go to /data (set KEEP_FILES=false if you don't want to keep them).
3. Optional variables:
   • ANTHROPIC_API_KEY or OPENAI_API_KEY: enables LLM summaries, classification and event tagging, using whichever key is set.
   • ADMIN_TOKEN (a long random string): enables /admin with the Run pipeline and Recompute patterns buttons (sign in with the token as the password), i.e. POST /api/pipeline/run and POST /api/analysis/run with Authorization: Bearer <token>.
   • PIPELINE_INTERVAL_HOURS (default 24).
4. Deploy. On first boot the app loads the bundled snapshot (data/seed/ufo-seed.json.gz) into an empty database so the site has data straight away, then the pipeline picks up anything newer.

Keep one replica. The scheduler runs inside the web process.

sudo apt-get install tesseract-ocr     # or: brew install tesseract
pip install -r requirements-dev.txt

python -m ufo run                # check sources, download, extract, classify
python -m ufo run --limit 10     # only process 10 pending records
python -m ufo status             # counts by processing status
python -m ufo reprocess classify # re-label everything (e.g. after adding an API key)
python -m ufo reprocess extract DOW-UAP-D102   # re-extract one record
python -m ufo extract-file some.pdf --force-ocr
python -m ufo analyze            # recompute waves, recurring details, clusters, links and episodes
python -m ufo episodes check     # which records have hand-read episodes, and the class counts
python -m ufo episodes export-input batches --only-new   # passages of new records, for reading
python -m ufo episodes import batches                    # merge a reader's output files
python -m ufo export-seed        # refresh data/seed/ufo-seed.json.gz

uvicorn ufo.web.app:app --reload
pytest

Subclass ufo.sources.base.Source, implement list_records() to return a RecordInfo for everything the source currently publishes, and register it in ufo/sources/__init__.py. Diffing, downloading, OCR, classification and the web app work unchanged. Candidates include AARO case-resolution reports (aaro.mil), the NARA UAP Records Collection (RG 615), and the FBI Vault.

• Summaries and labels are generated automatically. Keyword rules are coarse, and OCR of old or redacted scans is noisy. Every record links back to the original file.
• Episode summaries are short readings of that noisy text. The hand-read ones were written from the extracted pages, not the original scans, and where a table or clipping was too garbled to separate, its cases were grouped into one episode. Each summary links to the page it was read from, so it can be checked.
• "Official assessment" reports what the record itself concludes, not a judgment by this project.
• Patterns are statistical. A wave means more released pages mention a period, which depends on what the government chose to release, and not necessarily on more sightings. A link means two records are described in similar terms. The evidence pages exist so every pattern can be checked against the source text.