Upload portal for Free Law Project volunteer scanners to submit scanned legal documents (PDFs) for processing. A Django application that supports file uploads, staff review workflows, and S3-backed storage.

This project, including its code, tests, and this README, was vibe coded with Claude Code. It has not had extensive human review. Please read everything with skepticism!

# 1. Clone and enter the repo
git clone <repo-url> && cd scanning

# 2. Copy the dev environment file
cp .env.example .env.dev

# 3. Start everything
docker compose -f docker/scanning/docker-compose.yml up --build

# 4. Create a superuser
docker compose -f docker/scanning/docker-compose.yml exec scanning-django \
    python manage.py createsuperuser

The portal is now running at http://localhost:8002. Log in at /login/ with the superuser credentials you just created.

scanning/ serves as both the Django project package (settings, asgi, wsgi, urls) and the single app (models, views, forms). This is the simplest approach for a single-app project.

scanning/
  models.py           Scan model with Reporter/Status enums
  views.py            Upload, list, detail, review (function-based)
  forms.py            ScanUploadForm, ScanReviewForm
  urls.py             Root URL configuration
  admin.py            Scan admin registration
  storage.py          PrivateS3Storage + static storage
  context_processors.py
  workers.py          Custom UvicornWorker
  settings/
    django.py         Core Django settings
    project/
      logging.py, security.py, testing.py
    third_party/
      aws.py, sentry.py
  templates/scanning/ Login, upload, list, detail templates
  assets/
    templates/        base.html, cotton components
    tailwind/         Config + input CSS
    static-global/    Generated CSS output
  runpod-dotsmocr/    dots.mocr GPU worker image for RunPod Serverless

GPU work runs on RunPod Serverless workers. The dots.mocr OCR worker is built from scanning/runpod-dotsmocr/; see scanning/runpod-dotsmocr/README.md for the worker image, release workflow, endpoint configuration, and operational notes. (The legacy blackletter-gpu-worker -- YOLO detect + PaddleOCR analyze -- was removed with the legacy pipeline, issue #173.)

Settings follow the wiki project's split-file pattern. scanning/settings/__init__.py uses wildcard imports to compose the final config from:

settings/
  django.py              Core Django settings
  project/
    logging.py, security.py, testing.py
  third_party/
    aws.py, sentry.py

All settings use environ.FileAwareEnv() for environment-variable-based configuration.

• U.S. Reports
• Federal Cases
• Federal Reporter (1st, 2d, 3d)
• Federal Supplement (1st, 2d, 3d)

Staff users see a review form on the scan detail page. They can:

• Approve: Sets status=approved and records processed_at
• Reject: Resets status=uploaded with review notes

• Docker (or a Python 3.13 environment with PostgreSQL 16)
• An AWS account with S3 configured
• A domain with DNS and HTTPS configured (via a reverse proxy like Nginx or Caddy)

Create a .env file (or set environment variables directly). Every setting is read via django-environ's FileAwareEnv, so you can also use Docker secrets by pointing to files (e.g., SECRET_KEY_FILE=/run/secrets/key).

When DEVELOPMENT=False, Django uses S3 for both media uploads and static files. You need two S3 buckets:

Static files bucket (AWS_STORAGE_BUCKET_NAME): Stores collected static assets (CSS, JS). Files are served from the static/ prefix within the bucket.

Private uploads bucket (AWS_PRIVATE_STORAGE_BUCKET_NAME): Stores uploaded PDFs and cover images. All files are stored with private ACL and served via 5-minute signed URLs.

For the static files bucket:

• Enable public access (or serve via CloudFront)
• No special CORS or lifecycle rules needed

For the private uploads bucket:

• Block all public access (files are served via signed URLs)
• Suggested bucket policy: grant the IAM user s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:ListBucket
• A CORS rule is required. Uploads go straight from the browser to the bucket via a presigned POST, which is a cross-origin request. Without CORS the object still lands in S3, but the browser can't read the response, so the user sees an "Upload to storage failed" error and the scan is only linked later by recover_pending_uploads / the cleanup sweep. Allow the app origin(s) to POST (see the CORS example below).

[
  {
    "AllowedMethods": ["POST"],
    "AllowedOrigins": ["https://scanning.courtlistener.com"],
    "AllowedHeaders": ["*"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Use http://localhost:8002 as the origin for local development. List every origin that serves the app; the value must be the scheme + host only (no trailing slash or path).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::com-freelawproject-scanning-storage",
        "arn:aws:s3:::com-freelawproject-scanning-storage/*",
        "arn:aws:s3:::com-freelawproject-scanning-private-storage",
        "arn:aws:s3:::com-freelawproject-scanning-private-storage/*"
      ]
    }
  ]
}

docker build -t scanning-django -f docker/django/Dockerfile .

The Dockerfile:

• Installs Python dependencies via uv
• Installs Node dependencies and builds Tailwind CSS
• Copies the application code
• Runs as www-data user

Provision a PostgreSQL 16 instance (RDS, self-hosted, etc.) and create the database:

CREATE DATABASE scanning;
CREATE USER scanning_user WITH PASSWORD 'strong-password-here';
GRANT ALL PRIVILEGES ON DATABASE scanning TO scanning_user;

Run migrations:

docker run --env-file .env scanning-django migrate

The entrypoint's fallthrough case passes arguments to manage.py, so docker run scanning-django migrate is equivalent to python manage.py migrate.

Create the cache table (used for Django's database-backed cache):

docker run --env-file .env scanning-django createcachetable

When DEVELOPMENT=False, static files are stored in S3. Run collectstatic to upload them:

docker run --env-file .env scanning-django collectstatic --noinput

This uploads all static files to the static/ prefix of your AWS_STORAGE_BUCKET_NAME bucket.

docker run -it --env-file .env scanning-django createsuperuser

docker run -d \
    --name scanning-django \
    --env-file .env \
    -p 8000:8000 \
    scanning-django web-prod

This starts Gunicorn with Uvicorn workers (ASGI). Configuration:

• Workers: NUM_WORKERS env var (default: 4)
• Timeout: 180 seconds
• Bind: 0.0.0.0:8000

The application listens on port 8000. Put it behind a reverse proxy (Nginx, Caddy, etc.) for HTTPS termination.

Key production security settings are enabled automatically when DEVELOPMENT=False:

• SESSION_COOKIE_SECURE = True
• CSRF_COOKIE_SECURE = True
• SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
• HSTS: 2 years, with subdomains and preload

Nginx example:

server {
    listen 443 ssl;
    server_name scanning.free.law;

    ssl_certificate     /etc/letsencrypt/live/scanning.free.law/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/scanning.free.law/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 100M;
    }
}

# Django
SECRET_KEY=your-generated-secret-key-here
DEBUG=False
DEVELOPMENT=False
ALLOWED_HOSTS=scanning.free.law

# Database
DB_HOST=your-postgres-host.example.com
DB_NAME=scanning
DB_USER=scanning_user
DB_PASSWORD=your-strong-password
DB_SSL_MODE=require

# S3 (file storage + static files)
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_STORAGE_BUCKET_NAME=your-bucket-name
AWS_PRIVATE_STORAGE_BUCKET_NAME=your-private-bucket-name

# Sentry (optional)
SENTRY_DSN=https://[email protected]/0

# Workers
NUM_WORKERS=4

These shortcuts work in the scan process page (/scans/{pk}/process/) when focus is not on an input field.

The project uses a single Django app (scanning/) that also serves as the project package (settings, asgi, wsgi). This avoids unnecessary complexity for a focused, single-purpose application.

Files are organized by reporter and volume: uploads/{reporter}/{volume}/{uuid}.pdf. UUIDs prevent filename collisions while the directory structure keeps things browsable in S3.

Scans follow a simple status pipeline: uploaded -> processing -> pending_review -> approved -> extracted. Staff can approve (setting processed_at) or reject (resetting to uploaded) from the detail page.

All uploaded files use private ACL in S3 with 5-minute signed URLs. This ensures scanned documents are only accessible to authenticated users through the application.

Uses prefers-color-scheme (Tailwind's darkMode: 'media'). No manual toggle; the portal follows the user's OS/browser setting.

All CSS is built locally via Tailwind. No external network requests for assets.

Tests use Django's TestCase and run against a disposable test database:

# Run the full suite
docker compose -f docker/scanning/docker-compose.yml exec scanning-django \
    python manage.py test scanning.tests -v 2

# Run a specific test class
docker compose -f docker/scanning/docker-compose.yml exec scanning-django \
    python manage.py test scanning.tests.TestScanUpload -v 2

Or locally with uv:

uv run python manage.py test scanning.tests -v 2

docker compose -f docker/scanning/docker-compose.yml up starts:

pip install pre-commit
pre-commit install

Runs ruff (lint + format) and standard checks (large files, merge conflicts, trailing whitespace, etc.) on every commit.

Styles are in scanning/assets/tailwind/input.css using Tailwind's @layer directives. The config is at scanning/assets/tailwind/tailwind.config.js. The scanning-tailwind container watches for changes and rebuilds automatically.

Custom component classes: .btn-primary, .btn-outline, .btn-danger, .btn-ghost, .card, .input-text, .alert-*, .badge-* (status badges).

By default the daemon (run_daemon) runs every pass in one serial loop. With CELERY_ENABLED=True the daemon becomes the clock: on each pass's interval it sends a task instead of running the pass, the workers run the tasks on four queues (fast, io, heavy, opinions), and a button press sends its own task instead of waiting for a poll. The database stays the queue of record, so the broker never holds work that cannot be found again in the rows.

Locally the broker and one worker for every queue come up under the celery compose profile, beside the daemon:

CELERY_ENABLED=True docker compose -f docker/scanning/docker-compose.yml --profile celery up scanning-redis scanning-celery-worker scanning-daemon

In production each queue is its own deployment of the daemon image (command: worker with CELERY_QUEUES=<queue>), and the one daemon deployment stays as the clock. The heavy queue is the one that scales by replica count.

A worker run natively on macOS needs --pool threads: the default prefork pool spawns rather than forks there and fails every task. Linux, the compose path and production fork.

The switch is off after a merge: the daemon runs every pass itself, the views send nothing, and the deploy workflow skips a worker deployment that does not exist yet. Three things change with the switch off all the same: a running action stamps its claim every minute from a thread, and the stale sweep re-queues a claim after ten minutes without a stamp instead of an hour; a Mistral shard claimed and lost mid-render waits thirty minutes before its retry, where it was retried at once; and every process imports the Celery package at start. Turning it on is four steps, in this order, and the reverse of step 3 turns it off again.

1. Infra creates the four worker deployments from the daemon image (freelawproject/scanning:<sha>-daemon), each with command: ["worker"], its own CELERY_QUEUES (fast, io, heavy or opinions) set on the deployment and not in the shared secret, and the container named like the deployment, which is what the workflow's kubectl set image expects. One replica each; heavy is the one to scale. Give heavy a terminationGracePeriodSeconds that covers the longest apply, because a worker's shutdown waits for its running task and a pod killed past the grace period leaves its scan to the ten-minute stale sweep.
2. Infra creates a Redis for the scanning namespace, and its URL goes into the scanning-env secret as CELERY_BROKER_URL, with CELERY_ENABLED still off. Roll the workers out: they connect and sit idle.
3. Set CELERY_ENABLED=True in the secret and restart the daemon, the web deployments and the four workers. Every process reads the switch at start: the daemon's first log line then says "sending ticks to Celery", the web pods start sending a task for every button press, and the workers start sending the next pass when a step finishes, which they do not do while they hold the old value. Turning the switch off is the same restart of all three, or the workers keep sending to each other beside a daemon that runs the passes itself.
4. Check the worker logs show ticks received every few seconds, the queue depths read zero (LLEN fast and the other three on the broker), and a button press reaches a worker within a second.

Work in flight at the switch is not lost: every state is in the rows and in S3. A scan the daemon was running is re-queued by its shutdown and claimed by a heavy worker within seconds, which runs that action again from the start; queued scans are claimed the same way; external jobs already submitted are polled by the workers' sweep where the daemon left them. A Mistral shard the daemon was rendering at that moment waits the thirty minutes of the lost-claim clock, then retries.

The daemon can offload the two GPU-heavy blackletter steps (detect and analyze_pdf) to a RunPod Serverless endpoint. When RUNPOD_ENABLED=False (the default) the daemon runs blackletter in-process, so most local development needs no RunPod configuration. Flip the flag when you want to exercise the remote path end-to-end.

RunPod settings go in .env.dev:

RUNPOD_ENABLED=True
RUNPOD_ENDPOINT_ID=<your-endpoint-id>
RUNPOD_API_KEY=<your-runpod-api-key>

The daemon also needs AWS credentials so it can upload the local PDF to the dev private bucket and hand the worker a presigned GET URL. Two options:

Option A — paste the AWS console export block into your shell, then start the containers. This is the easiest path when you're using temporary SSO / STS credentials that rotate. The docker-compose file (docker/scanning/docker-compose.yml) forwards AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN from your shell into the containers and maps them onto AWS_DEV_* automatically:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
docker compose -f docker/scanning/docker-compose.yml up --build

Option B — hard-code dev credentials in .env.dev. Use this for long-lived IAM user keys:

AWS_DEV_ACCESS_KEY_ID=<your-aws-key>
AWS_DEV_SECRET_ACCESS_KEY=<your-aws-secret>

Either way, the dev S3 buckets (dev-com-freelawproject-scanning-storage and dev-com-freelawproject-scanning-private-storage) are selected automatically when DEVELOPMENT=True; you do not need to set AWS_PRIVATE_STORAGE_BUCKET_NAME.

Optional tuning knobs (all have sensible defaults, see scanning/settings/project/runpod.py):

python manage.py make_dev_data (run automatically by the web-dev entrypoint) normally seeds two users (staff / scanner) and a handful of Scan rows with placeholder PDF bytes (b"%PDF-1.4 test"). Those placeholder bytes are not valid PDFs and will fail the moment YOLO or PaddleOCR tries to read them on the worker, polluting the DB and burning endpoint quota on guaranteed errors.

When RUNPOD_ENABLED=True, the command short-circuits before doing anything. Create your own user with createsuperuser and upload a real PDF through the UI to drive the pipeline; the daemon will push it to the dev private bucket on demand and dispatch the GPU steps to RunPod.

Setting RUNPOD_ENABLED=True also activates the full S3 artifact sync (scanning/s3_sync.py), which is otherwise skipped in DEVELOPMENT=True. Intermediate processing files (detections.json, redacted/*.pdf, etc.) get pushed to dev-com-freelawproject-scanning-private-storage and pulled back on reprocess, so a local end-to-end run exercises the same recovery path as production.

1. Set the RunPod variables in .env.dev.
2. Export AWS credentials in your shell (Option A) or add AWS_DEV_* to .env.dev (Option B).
3. docker compose -f docker/scanning/docker-compose.yml up --build.
4. Create a user: docker compose -f docker/scanning/docker-compose.yml exec scanning-django python manage.py createsuperuser.
5. Log in at http://localhost:8002/login/ and upload a real PDF.
6. Watch the daemon container logs. You should see:
   • uploading <name>.pdf to s3://dev-com-freelawproject-scanning-private-storage/<key> before presign from runpod_client._ensure_presigned_url, immediately before the GPU step is dispatched.
   • RunPod poll ticks while the worker runs.
   • Uploaded N processing file(s) for scan <pk> to s3://... from s3_sync.upload_processing_files once the pipeline completes — the artifact sync that's now active in the dev RunPod path.
7. Set SCANNING_LOG_LEVEL=DEBUG for per-poll detail.

# Run migrations
docker exec scanning-django python manage.py migrate

# Create the cache table (needed once after initial DB setup)
docker exec scanning-django python manage.py createcachetable

# Create a superuser
docker exec -it scanning-django python manage.py createsuperuser

# Collect static files to S3 (production)
docker exec scanning-django python manage.py collectstatic --noinput

# Open a Django shell
docker exec -it scanning-django python manage.py shell

Quick reference for going to production:

• SECRET_KEY set to a strong random value
• DEBUG=False and DEVELOPMENT=False
• ALLOWED_HOSTS set to your domain(s)
• PostgreSQL configured with DB_SSL_MODE=require
• S3 buckets created (public for static, private for uploads)
• AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured
• collectstatic run to upload static files to S3
• migrate and createcachetable run against the production database
• Reverse proxy configured with HTTPS
• Superuser created
• Sentry DSN configured (optional)

AGPL-3.0-only