# Protocol: inference service, v1 For a service that runs a model for you somewhere else: a GPU instance a provider rents by the hour, or a managed endpoint the provider operates. The entry names the subject (the service, in a stated region and instance class) and holds the model and the runtime fixed to ones already measured on the bench, so the service's share of every number can be read off against a local entry. Version 1, written 2026-10-12. Steps 1 to 11 are the mandatory core; steps 12 to 16 run when the service allows. One that does not run is listed under "not measured" with the reason. Every step names the evidence file it produces. Times are UTC. Nothing is scored. How the capacity was obtained is the first line of the entry: paid at the list price, a free-trial credit claimed by me, or capacity supplied by the provider, and it is listed the same way on the public equipment page. Three things are never published: the account identifiers, the instance's own address, and any key. Remote addresses the instance talks to are findings and are published as found. ## Core (mandatory) ### 1. Provisioning, from zero Wall clock and every step from "no account" to "first token": what the provider asked for to open the account (identity, card, verification), the region and instance class chosen and why, the GPU and memory as the catalogue states them, the hourly price as shown on the day, the image, every command. Human decisions and failures counted, as in the local protocol. Evidence: `evidence/setup.log`, `evidence/provider.json` (class, GPU, memory, region, price, image, dates). ### 2. Speed, standard `llama-bench` (or the runtime's equivalent) on the instance, prompt processing at 512 tokens and generation at 128, three repetitions, JSON output. Tokens per second with the standard deviation, file size, parameter count, backend. Evidence: `evidence/llama-bench-pp512-tg128.json`. ### 3. Speed, agent-length context The same with 4,096 and 16,384 tokens already in context (`-d`). Evidence: `evidence/llama-bench-depth.json`. ### 4. Footprint Resident memory and GPU memory of the serving process once loaded, swap in use on the instance, the context size configured. Evidence: `evidence/egress-and-memory.txt`. ### 5. Egress, process and instance Two lists. First, every TCP and UDP socket the serving process holds during live runs (`ss -tunap` filtered to its PID, or `lsof` where available). Second, every outbound connection the instance as a whole opens over one hour of the test session, counted by destination address and port (`tcpdump` on the public interface, outbound only, summarized; the raw capture is kept, not published). The provider's own agents and metadata services appear here by name if the provider documents them, and as addresses if it does not. Any remote address the serving process holds is a finding. Evidence: `evidence/egress-and-memory.txt`, `evidence/egress-capture.txt`. ### 6. Holding a mandate, from outside The Major Labs Mandate Sandbox's seven bank-customer jobs, three runs each, run by the harness on a machine in the UK against the instance's endpoint over TLS, the model as the live agent under a signed MandateKit mandate, BudgetGuard on spend and call count, every step on a WitnessKit chain verified at the end, the sandbox's unrequested options placed inside ordinary tool results. Reported in the per-layer counts table of `bench/REPORTING.md`. Reasoning effort medium, temperature stated, held fixed. Kit versions and the per-layer kit protocols: `protocols/kits/`. This is pass 1 of `protocols/adversarial-suite-v1.md`, with the network in the loop. Evidence: `evidence/suite/pass1/summary.md`, `summary.json`, one `-run.json` per run with transcript and signed chain. ### 7. Holding a mandate, from inside Step 6 again with the harness running on the instance itself against localhost. The counts should match step 6 within the run-to-run variation already seen on the local entries; a difference is a finding about the network path, not the model. Evidence: `evidence/suite-local/pass1/`, the same layout as step 6. ### 8. Network path Time to first token and tokens per second for the same request (512 prompt tokens, 256 generated, three repetitions) measured by the harness from the UK machine and from the instance, side by side; the difference is the network's share. Alongside: round-trip time to the endpoint over 60 samples, and the hop count to the region (the provider's addresses only; nothing else on the path is published). Evidence: `evidence/network-path.json`. ### 9. What the service says it keeps Read on a stated date, quoted with the URL: the provider's published terms for the instance class or endpoint on logging of prompts and completions, retention, use for training, data location, sub-processors, and who can access the instance's disk. Then what the control panel and API expose about the instance (logs, metrics, snapshots) as observed. Reported as two columns, "stated" and "observed", one row per question. Evidence: `evidence/service-terms.md`. ### 10. Data residency, observed The region the provider labels the instance with, the registry record for the instance's address range (RIR lookup, the organization and country fields), the stated location of the block storage, and the addresses the runtime contacted in step 5 with their registry countries. Reported as stated versus observed. Evidence: `evidence/residency.txt`. ### 11. Cost and teardown The hourly price as shown on the day, the hours used, and the actual charge or credit consumed for the test window as the billing page reports it, with the cost of one suite pass derived from it. Then the deletion of the instance, its volumes and any snapshot, with the provider's confirmation and the time, and a check 24 hours later that nothing is still billing. Evidence: `evidence/cost.txt`, `evidence/teardown.log`, `evidence/photos/billing.png`. ## When the service allows ### 12. Concurrency `llama-server --parallel 4` (or the runtime's equivalent), context set so each slot holds at least 8,192 tokens. The harness fires one, two and four identical requests at once (512 prompt tokens, 256 generated), three repetitions per level, from the instance. Aggregate and per-request tokens per second, median time to first token, KV cache per slot. Evidence: `evidence/concurrency.json`. ### 13. Sustained 20 minutes Generation requests back to back for 20 minutes (one at a time, 512 prompt tokens, 512 generated, the same prompt), each request's tokens per second with a timestamp, from the instance. Reported: tokens per second in minute 1 and minute 20 with the percentage change. There is no wall meter or thermometer on a rented instance; what this step catches is a shared host. Evidence: `evidence/sustained-20min.jsonl`. ### 14. Larger model Where the instance's memory allows, steps 2, 3, 4 and 6 on the next model class up (on 48GB, gpt-oss-120b does not fit; on 80GB and above it does), the file named with size and hash. Evidence: `evidence/model-/`, mirroring the core file names. ### 15. Managed endpoint Where the provider also operates a managed endpoint for the same model, steps 6, 8 and 9 against it, with the price per million tokens as shown on the day and the cost of one suite pass derived from the tokens the harness counts. The instance and the endpoint are reported side by side. Evidence: `evidence/endpoint/`, mirroring the step 6, 8 and 9 file names. ### 16. Reasoning-effort sweep Step 7 at reasoning effort low and high (medium is the core), every other setting held. One counts table per level, with completion tokens and wall seconds. Evidence: `evidence/suite-local-low/`, `evidence/suite-local-high/`. ## What is reported | Step | Finding rows | Unit | |---|---|---| | 1 | account requirements, zero to first token, instance class, price, human decisions, failures | list, seconds, currency/hour, counts | | 2, 3 | prompt processing and generation at each depth | tokens/s, sd | | 4 | resident memory, GPU memory, swap, context | GB, tokens | | 5 | sockets held by the process; outbound destinations of the instance over one hour | list, counts | | 6, 7, 16 | the per-layer counts table from outside and from inside | counts | | 8 | time to first token and tokens/s from the UK and on the instance, round trip, hops | ms, tokens/s, count | | 9 | stated versus observed, one row per question | text | | 10 | region as labeled, registry country, storage location, egress countries | text | | 11 | hourly price, hours, charge, cost per suite pass, deletion time, 24-hour check | currency, hours, time | | 12 | aggregate and per-request speed, time to first token at 1, 2 and 4 | tokens/s, ms | | 13 | tokens/s in minute 1 and 20 | tokens/s, percent | | 14, 15 | the core rows for the larger model or the managed endpoint, side by side | as above | ## Not measured in v1 Power at the wall (not observable on a rented instance; the provider's published energy figures are quoted if any exist), private networking between instances, failover between regions, service-level agreements, fine-tuning, compliance attestations beyond what the provider publishes, price changes over time, managed Kubernetes, accuracy benchmarks (MMLU-class scores are the model publisher's business). Add in v2 when a service makes one of them matter.