Install contribution dependencies¶

Install Python dependencies in a virtual environment.

$ pip install --editable . --group dev

Spell checking requires enchant. This can be installed on macOS, for example, with Homebrew:

$ brew install enchant

and on Ubuntu with apt:

$ apt-get install -y enchant

Install pre-commit hooks:

$ prek install

Linting¶

Run lint tools either by committing, or with:

$ prek run --all-files --hook-stage pre-commit --verbose
$ prek run --all-files --hook-stage pre-push --verbose
$ prek run --all-files --hook-stage manual --verbose

Running Tests¶

Create an environment variable file for secrets:

$ cp vuforia_secrets.env.example vuforia_secrets.env

Some tests require Vuforia credentials. To run these tests, add the Vuforia credentials to the file vuforia_secrets.env. See Connecting to Vuforia.

Then run pytest:

$ pytest

Connecting to Vuforia¶

To connect to Vuforia, Vuforia target databases must be created via the Vuforia Web UI. Then, secret keys must be set as environment variables.

The test infrastructure allows those keys to be set in the file vuforia_secrets.env. See vuforia_secrets.env.example for the environment variables to set.

Do not use a target database that you are using for other purposes. This is because the test suite adds and deletes targets.

To create a target database, first create a license key in the Vuforia License Manager. Then, add a database from the Vuforia Target Manager.

To find the environment variables to set in the vuforia_secrets.env file, visit the Target Database in the Vuforia Target Manager and view the “Database Access Keys”.

Two Cloud databases are necessary in order to run all the Cloud Target tests. One of those must be an inactive project. The script creates the inactive project automatically by deleting its license.

VuMark tests require one VuMark database.

Targets sometimes get stuck at the “Processing” stage meaning that they cannot be deleted. When this happens, create a new target database to use for testing.

To create databases without using the browser, use admin/create_secrets_files.py:

$ export VWS_EMAIL_ADDRESS=...
$ export VWS_PASSWORD=...
$ export NEW_SECRETS_DIR=...
# You may have to run this a few times, but it is idempotent.
$ python admin/create_secrets_files.py
# Each generated file gets its own active Cloud database credentials.

For the complete archive and GitHub Actions setup procedure, see Continuous Integration.

Skipping Some Tests¶

The tests run against several backends: the real Vuforia, the in-memory mock, and the Flask applications of the mock served in the test process. The last of these exercises the handlers which the Docker deployment runs, but not the split between its containers. The tests in tests/mock_vws/test_docker.py build and run the containers for that. The pytest-multi-backend plugin adds options which skip some tests:

--skip-backend=real   Skip tests against the real Vuforia
--skip-backend=mock   Skip tests against the in-memory mock Vuforia
--skip-backend=flask_in_process
                      Skip tests against the Flask applications served
                      in the test process
--skip-marker=requires_docker_build
                      Skip tests which build and run the Docker images

Give an option once per backend or marker to skip.

Shared database credentials¶

The shared API client fixtures, raw query helpers and recognition-report helpers accept CloudDatabaseCredentials from vws_mock_contract_tests.fixtures.credentials. An adapter supplies an object with read-only string attributes server_access_key, server_secret_key, client_access_key, client_secret_key and database_id. The helpers use those values for signing HTTP requests and constructing report URLs. They do not require a mock_vws.database.CloudDatabase object or access to its targets and storage state. The current adapters’ database objects satisfy the protocol, and another adapter can provide its own credential object. Backend setup and mock controls remain responsible for managing database state.

Tests which read a database name use NamedCloudDatabaseCredentials from the same module. It adds the read-only string attribute database_name for summary responses, recognition-count controls and report requests which distinguish a database’s name from its ID.

Shared backend setup¶

An adapter supplies the service URLs through base_vws_url and base_vwq_url. The shared client and prepared-request fixtures use those URLs for HTTP requests. The adapter also supplies vws_client_class and the active, inactive and VuMark database credentials required by each test. Mock adapters use synthetic credentials. Real Vuforia credentials and account restrictions belong to the real adapter.

The verify_mock_vuforia fixture sets up the selected VWS and query backend. The verify_model_target_mock_vuforia fixture does the same for the Model Target API. Their mock-only counterparts are mock_only_vuforia and model_target_mock_only_vuforia. Each setup fixture yields None once its backend is ready. The adapter handles cleanup after the test. Shared fixtures depend on these public setup fixtures with a None annotation. Backend selection values stay inside the adapter.

Adapters declare supported behavior through BackendCapabilities from vws_mock_contract_tests.utils.backend_capabilities. Shared tests use those values for ordering, rate-limit windows and transport checks. The seed_recognition_counts operation accepts a database name, a target ID and recognition counts through SeedRecognitionCounts from vws_mock_contract_tests.utils.recognition_counts. The unrouted_requests fixture supplies UnroutedRequests from vws_mock_contract_tests.utils.unrouted_requests, including whether unmatched requests produce an HTTP response. These controls keep backend operations and transport differences behind the adapter’s fixture contract.

Installed contract tests¶

The contract-tests project builds vws-mock-contract-tests with an explicit shared fixture plugin. The test-backends project builds the core repository’s adapter plugin separately. See contract-tests/README.rst for installation, invocation and revision compatibility.

Container deployment tests¶

The deployment tests build and run three separate services through the Docker Python SDK. They use the Docker engine selected by the current Docker configuration by default:

$ uv run --group=dev pytest tests/mock_vws/test_docker.py

To run the same tests with Apple’s container runtime, complete the runtime installation steps in Creating containers with Apple container, then install Socktainer 1.5.2 or later. It provides the Docker-compatible API used by the Python deployment tests:

$ brew install socktainer
$ socktainer --no-auto-start --no-docker-context

Keep Socktainer running in that terminal. In another terminal, select its socket for the test command:

$ DOCKER_HOST="unix://$HOME/.socktainer/container.sock" \
    uv run --group=dev pytest tests/mock_vws/test_docker.py

The deployment tests automatically negotiate the API version.

Both configurations exercise native health checks, published HTTP ports, application behavior and restarts. Running the mock directly with Apple’s CLI uses the same images and is documented in Creating containers with Apple container. Target-manager and VWQ use automatically published ports. VWS keeps an explicit loopback binding so its report download URLs are known before startup. Test containers, image tags and networks are removed after the run, including when setup fails. Hosted CI continues to use Docker. Apple deployment validation runs on a local Mac.

Verifying signed Model Target requests¶

Creating an advanced Model Target dataset with a state-based configuration or a standard dataset with inline CAD data is a “signed” request: the real Vuforia signs the trained dataset, and each signing consumes the account’s Model Target training allowance. The allowance is small (roughly 20 signings), it is shared by every CI job and every concurrent run, and it cannot be raised or reset. Verifying signed requests on every run exhausted the allowance within hours and then made every CI run fail with TRAINING_ALLOWANCE_EXCEEDED.

The signed test cases therefore run against the mock backends on every run, but are skipped against the real Vuforia by default. To verify them against the real Vuforia, for example after the allowance has recovered, opt in with:

--verify-model-target-signing
                      Run signed Model Target dataset tests against
                      the real Vuforia

The equivalent unsigned requests (a standard dataset, or an advanced dataset without a state-based configuration) are far cheaper and are verified against the real Vuforia on every run. With enough traffic even unsigned dataset creation can be rejected with TRAINING_ALLOWANCE_EXCEEDED; an unexpected allowance rejection is reported as an expected failure rather than a test failure, and the affected tests pass again automatically once the allowance recovers.

Retrying transient real Model Target failures¶

The load balancer in front of the real Model Target Web API sometimes answers a perfectly good request with a gateway error, or drops the connection. That is nothing to do with the contract under test, and a rerun of the same job passes, so a small number of retries is applied to requests which are safe to repeat.

The policy is in tests/mock_vws/utils/model_target_retries.py. It applies only while a test is running against the real Model Target backend, and only to GET requests: repeating a dataset creation can create a second dataset and can consume the account’s Model Target training allowance, so mutating requests are sent exactly once. Three attempts are made, with an exponential backoff with jitter of up to ten seconds, using tenacity. A request which still fails transiently on the last attempt is not hidden: the response is returned as it is, so the assertion reports the real status and body.

The mock backends are unaffected, so tests which configure a mock to answer with a 5xx still get that response immediately.

This is separate from the repository-wide pytest-retry configuration, which retries a test only when it fails with one of the exception types returned by pytest_set_filtered_exceptions in the top level conftest.py. The Model Target tests assert on requests.Response objects, so a gateway error reaches pytest-retry as an AssertionError and is never retried by it. Widening that allowlist would retry every failing assertion in the suite, which is why the Model Target policy sits in the request helper instead, before the assertion.

Documentation¶

Documentation is built on Read the Docs.

Run the following commands to build and view documentation locally:

$ uv run --group=dev sphinx-build -M html docs/source docs/build -W
$ python -c 'import os, webbrowser; webbrowser.open("file://" + os.path.abspath("docs/build/html/index.html"))'

Continuous Integration¶

See Continuous Integration.

Learnings about VWS¶

Vuforia Web Services, at the time of writing, does not behave exactly as documented.

The following list includes details of differences between VWS and expected or documented behavior.

When attempting to delete a target immediately after creating it, a FORBIDDEN response is returned. This is because the target goes into a processing state.

image is required for POST /targets, but it is documented as not mandatory.

The tracking_rating returned by GET /targets/<target_id> can be -1. This happens only while a newly uploaded target is processing, and it lasts for about a second. Updating a target returns it to the processing state but does not return the rating to -1; the new image’s rating is reported straight away.

The database summary from GET /summary has multiple undocumented return fields.

The database summary from GET /summary is not immediately accurate.

The documentation page Vuforia Query Web API states that the Content-Type header must be set to multipart/form-data. However, it must be set to multipart/form-data; boundary=<BOUNDARY> where <BOUNDARY> is the boundary used when encoding the form data.

The documentation page Vuforia Query Web API states that Content-Type will be the only response header. This is not the case.

The documentation page Vuforia Query Web API states that 10 is the maximum allowed value of max_num_results. However, the maximum allowed value is 50.

A response to an invalid query may have an application/json content type but include text (not JSON) data.

After deleting a target, for up to approximately 30 seconds, matching it with a query returns a 500 response.

A target with the name \uffff gets stuck in processing.

The documentation page Vuforia Query Web API states that “The API accepts requests with unknown data fields, and ignore the unknown fields.”. This is not the case.

The documentation page Vuforia Query Web API states “Maximum image size: 2.1 MPixel. 512 KiB for JPEG, 2MiB for PNG”. However, JPEG images up to 2MiB are accepted.

There is no documented limit on the number of pixels in an image, but POST /targets returns ImageTooLarge for an image with more than 37748736 pixels, whatever its file size, aspect ratio or color space. An image of a single color has a tiny file size whatever its dimensions, which is how this limit is reached. The Query API applies no such limit. It applies only its maximum width and height of 30000 pixels.

The request_count in a database summary is always 0.

The documentation for the target summary report says “Note: tracking_rating and reco_rating are provided only when status = success.”. However, reco_rating is never provided and tracking_rating is provided even when the status is “failed”.

Release Process¶

See Release Process.

Release notes¶

Write user-facing changes as Markdown in newsfragments/<issue>.change.md. Towncrier writes one Markdown file per version, used directly for GitHub release notes. Invalid fragment names fail release assembly.