Testing and verification levels

Apollo Download Manager replaces aria2 with its own transfer engine, so its correctness rests on its own tests. This page is the test strategy (#15):

• Levels L1–L5: what each proves, and how long it takes.

• The test pyramid, and which issue builds each layer.

• Fixtures: the rules, and the local servers Apollo Download Manager needs.

• Platform reporting: how results are reported per platform.

All check commands run every check even after one fails, print a PASS/FAIL summary with timings, and exit non-zero if anything failed. They run tools directly, never through a shell.

Levels and commands

In CI (CI.md): pull requests run pnpm check:pr for the selected areas, and every merge to main runs pnpm check:fast, pnpm check:links, pnpm check:product-name and pnpm check:guidance on Linux. Every run publishes one report with every failure (CI_DIAGNOSTICS.md).

For documentation, pnpm check:links checks that every relative link and heading anchor in the repository's Markdown files resolves. External links aren't fetched.

pnpm check:product-name (#421) fails when V2, the old working name, is used for the product in the documents, the README, the website's content, the extension's pages or the window's source. Identifiers in code, BitTorrent v2, the quoted 0.1.x name, quoted issue titles, and decision records' titles and quotes may keep it; the list and its reasons are in scripts/product-name-lib.mjs. Every pull request and every merge to main runs it.

Examples

pnpm test:focused -- velocity-core                 # cargo test -p velocity-core
pnpm test:focused -- velocity-core app_info        # only tests whose name contains app_info
pnpm test:focused -- scripts -t selectChecks       # Vitest, one describe block
pnpm test:watch                                    # Vitest in watch mode
cargo test -p velocity-core -- --nocapture         # cargo directly, with output
pnpm check:pr                                      # what this branch needs
pnpm check:pr --base HEAD                          # only the uncommitted changes

Areas for test:focused are the workspace crates (velocity-core, velocity-desktop-v2), frontend, scripts and extension (the browser extension, #64). An unknown area prints the valid ones.

How check:pr selects checks

The rules live in scripts/checks-lib.mjs and are unit tested in scripts/checks-lib.test.mjs. The CI change detection (#13) reuses them, so local and CI selection can't drift apart. The first matching rule wins:

Selecting too little would let a broken change through, so anything shared or unknown selects everything. A new top-level directory needs a rule in checks-lib.mjs (with a test) before it can select less.

Build settings for fast feedback

These settings were measured in #3. The timings are in DEVELOPMENT.md.

• Dependencies are built with opt-level = 1 and no debug info. The app runs at a usable speed, and dependencies are compiled once.

• Workspace crates are built with debug = "line-tables-only". Backtraces keep file and line numbers. The Tauri shell's library is also built as a staticlib and a cdylib for mobile targets. With full debug info, every test run re-linked several hundred megabytes. To debug with variables in a debugger, override it for that build: CARGO_PROFILE_DEV_DEBUG=true cargo build.

• The release profile (LTO, one codegen unit, opt-level = "s") applies only to --release builds and never to development.

Two other settings were measured and rejected:

• tsc --incremental was slower: 1.3–1.6 s against 0.9 s, because writing the build-info file costs more than it saves on a project this size. Revisit when the frontend grows.

• ESLint --cache was rejected for correctness. The configuration uses type-aware rules, and ESLint caches results per file. A type change in one file can change the lint result of another file whose cache entry is still valid, so a cached run could pass when a fresh one fails.

The test pyramid

From most numerous and fastest to fewest and slowest. Each layer proves what the one below can't.

Fixture policy

• Local only. Every test talks to servers it starts itself on 127.0.0.1 (or ::1), on a free port. A test's own libtorrent session listens on a port free for TCP and UDP (velocity_torrent::engine::free_listen_port, #438), never port 0: on Windows the system's free TCP port can be a reserved UDP one. No test contacts the public internet, a public tracker, the DHT or a CDN. A test that needs the network is a defect.

• Seeded data. Files are generated from a fixed seed with a known SHA-256. Every transfer test checks the SHA-256 of the result, not just "completed".

• Deterministic timing. Throttling and delays are controlled by the fixture, never by real network conditions. Timeouts are generous and bounded. A random seed is printed when a test fails.

• Misbehaviour on purpose. Fixtures expose switches for the server faults Apollo Download Manager must survive (below), so each test chooses its fault explicitly.

• No secrets. Test certificates, keys and passwords are generated for the test or committed as obviously fake fixtures, never real credentials.

• Real components where it matters. Fakes are fine at unit boundaries. Claims about the engine, processes, files or crash recovery use real libcurl, real processes and real files.

Fixture servers Apollo Download Manager needs

How the engine is tested

• Scheduler logic without a network. velocity-transfer runs against a simulated transport: random chunk sizes, delays, splits, cancellations, retries, duplicate and late callbacks, 200 substitutions and simulated crashes (runtime state dropped, durable coverage kept). After every simulated event, a property test asserts invariants I1–I7 and byte equality with the source.

• Protocol behaviour with real libcurl. velocity-curl and the engine run against the fixture servers above, so abort-at-boundary, HTTP/2 stream resets and libcurl's real callbacks are covered.

• Acceptance rows. Each row of the research report's section 5.4 becomes at least one named fixture test, checking the final SHA-256, the status shown and the wasted bytes.

• Crash recovery. kill -9 at random points over 100 trials: always resumes; re-downloaded bytes are at most one checkpoint interval plus in-flight data (#43). The app killed the same way comes back from its database and finishes the right file (#74).

• Disk faults. A full disk or a refused write mid-download fails with its own message, keeps what reached the disk, and resumes once fixed (#74).

The secret-leak scan (#71)

velocity_test_http::leaks::LeakScan looks for a test's known secrets everywhere Apollo Download Manager writes or shows something. A test that uses a login, a cookie, a key's passphrase or a tracker passkey runs it at the end:

• Where it looks: every file under the folders the test gives it (the database and its WAL, checkpoints, part and finished files), files in the system's temporary folder changed since the test began (three levels deep), texts such as the app's log, the diagnostics export and what the details panel shows, and a running child process's arguments and environment.

• In what forms: each secret as written, percent-encoded, and in base64 at each of its three possible alignments, so a Basic login or an encoded link is caught too. A secret must be at least 8 bytes, so it can't match by chance.

• A running child, on Linux only: it reads /proc/<pid>/cmdline and environ of the engine and the torrent helper. On Windows that part prints NOT RUN, and secret_leaks.rs checks the command a child is started with instead: one fixed argument and a cleared environment (PO decision on #71).

• Where it runs: crates/velocity-transfer/tests/secret_leaks.rs (HTTP login and cookie and FTP login through the child engine; SFTP key and password logins) and crates/velocity-torrent/tests/manager.rs (a private tracker's passkey, through the manager and the torrent helper). The scan's own tests plant each kind of leak and check it is found.

Timelines in ramp tests (#257)

velocity_test_http::timeline::Timeline samples a line every interval while a test runs, and the test puts the lines in its failure message.

• Why: the adaptive-connection tests time the ramp. A loaded CI runner bends that timing in ways a local run doesn't show, even with the test pinned to one or two cores.

• Where: adaptive_connections_stop_below_the_cap_when_more_stop_helping (crates/velocity-transfer/tests/connections.rs) and a_server_that_throttles_each_connection_gets_separate_connections (crates/velocity-transfer-h2c/tests/http2.rs) record, every 500 ms, the requests running and the ramp's count and reason. A failure on CI then shows what the ramp did.

The field test (#262)

The installed app, built from main, used the way a person uses it: real links from real servers, pause and resume, closing and killing the app mid-download, torrents. pnpm field drives the real window through Playwright and records each download every few seconds, with what is on disk and the app's connections. Every problem becomes an issue in epic #1. It runs by hand on Windows, before each release candidate, never in CI, and never against the public internet in automated tests (the fixture policy above still holds).

How to run a session, what it may download and how links are kept private: tests/field/README.md. The scenarios: tests/field/CHECKLIST.md. Session reports are comments on #262.

How the frontend is tested

• Only the native boundary is mocked: the UI is rendered in jsdom (the ui project in vite.config.ts) against a fake backend built on Tauri's own mockIPC (src/test/fakeBackend.ts), which answers with the contract fixtures and records every call. Events are sent with Tauri's emit. Components, hooks and state are real (UI_ARCHITECTURE.md).

• Clicks and typing go through Testing Library's user-event; a drag goes through pointer events. The assertions read roles, names, text and attributes directly, without extra matchers.

• Contract fixtures: the shapes the Rust side sends are generated from Rust into tests/fixtures/contract/ and validated by the UI's strict schemas, so neither side can drift (CONTRACTS.md).

• What the user sees: tests check text, roles and actions, not component internals.

• Keeping findByRole fast (#371): it styles every element on the page to see whether it's hidden, and that was most of an app test's time. Cost grows with the page's elements and CSS rules, so src/test/setup.ts removes the toast library's stylesheet and clears its toasts after each test. Without that, a file's later tests redrew every earlier toast.

Reporting results per platform

• Every check, per platform: PASS, FAIL, BLOCKED or NOT RUN, with the exact command. A pass on one platform says nothing about another (AGENTS.md, "Reporting").

• NOT RUN needs a reason, for example "no Mac available" (the PO accepted macOS as NOT RUN for #2) or "OpenSSH not available on Windows".

• CI results: reported only after seeing them. A job the selector didn't choose is NOT RUN, not PASS. CI's own statuses (FAILED, INFRA, BLOCKED, CANCELLED…) are explained in CI_DIAGNOSTICS.md.

• Platform runs use /platform-verify (Windows natively, Linux in the WSL clean clone).

Test-only dependencies

Test tools are dependencies too. Each needs PO approval before it's added (DEPENDENCIES.md), recorded in the issue that first needs it. These are expected:

Without approval, a hand-written seeded generator or a Node built-in module is the fallback.

Writing tests

• Test first: write the failing test, make it pass with the smallest change, then refactor. Bug fixes get a regression test. /tdd-change walks through it.

• Never weaken a test to get a pass: don't delete, skip or weaken it.

• Location: Rust unit tests live next to the code in #[cfg(test)] mod tests. Integration and fixture tests go in a crate's tests/ directory. Frontend tests are src/**/*.test.ts(x), and script tests are scripts/**/*.test.mjs. Names follow CONVENTIONS.md.

• Determinism: tests don't use the public network, uncontrolled timing or shared global state (see the fixture policy above).

• Wait for what you expect, not for a fixed time. A test that needs a message or a state waits until it arrives, bounded by a generous deadline, and only then checks that nothing more happens. A fixed window fails when the whole machine pauses: a 1 s freeze of the test process failed the manager's snapshot timing test 4 times in 16, and none of 380 runs under CPU load did (#188).