Most FHIR benchmark write-ups land as a marketing PDF with a few bar charts and no path to verify. The newest one is different. On June 29, Health Samurai published a public test harness that runs HAPI FHIR, Medplum, the Microsoft FHIR Server, and Aidbox under the same conditions, with the methodology in the repo and the dashboard rerunning daily. Health Samurai builds Aidbox, so the vendor-run framing applies; the test methodology is what makes it worth reading anyway.
The numbers will move week over week. The harness is what stays the same. This piece walks through the methodology choices that shape the test, because those choices decide what any FHIR benchmark can actually tell you.
The Hardware Pinning
The host is one bare-metal machine with 64 CPU cores and 500 GB of RAM. Each server under test runs in its own Docker container with explicit CPU and memory limits: 8 vCPU and 24 GB of RAM. Medplum is the exception that proves the design rule. The platform scales horizontally by default, so the harness runs 8 replicas of 1 vCPU and 3 GB each, totalling the same allocation. That keeps the comparison apples to apples instead of penalizing a server for not matching a single-process model.
For the FHIR learning path and broader context on how server architecture shapes integration choices, the methodology piece matters more than any single throughput number.
The Load Generator
Grafana k6 drives the load. The k6 scripts live in the repo, so anyone can read them, fork them, or run them against a different server entirely. The CRUD scenario runs 300 concurrent threads across nine FHIR resource types, exercising create, read, update, and delete. The bundle import scenario runs 20 threads, which matches the way Synthea bundles actually arrive in a migration project. Search scenarios sweep six families: string, date, reference, token, quantity, and composite.
Using k6 instead of a custom Java rig matters because k6 has its own community and its own track record. Tuning a benchmark in a custom rig is harder to argue with than tuning one in a tool a hundred other teams already use.
The Synthetic Data
The dataset is Synthea, 1,000 generated patients, around 2 million FHIR resources after import. Synthea is the canonical choice for FHIR testing because the generator is open, the resource shapes are clinically reasonable, and the same seed produces the same data across runs. The 1,000-patient size is small enough to fit in memory comfortably and large enough that any server with a broken indexing strategy will show it. The benchmark team has flagged that a larger-scale follow-up is coming, which is the test that production loads actually care about.
For teams thinking about the gap between synthetic and production data, ADT vs ORM messages: which is harder to convert to FHIR covers the kind of real-world shape that synthetic Synthea does not produce.
The Containers and the Database Pairings
Each server runs in a Docker container with the database it was designed around. Aidbox, HAPI, and Medplum each pair with PostgreSQL 18. The Microsoft FHIR Server pairs with SQL Server 2022 Developer Edition. Forcing every server onto the same database would have produced a fairer-looking chart and a less useful one, because the Microsoft server is built against SQL Server and HAPI is built against Postgres. The harness tests each on its native pairing instead.
The Daily CI Rerun
The repo has a Drone CI configuration that runs the full benchmark daily. The dashboard updates with the latest snapshot, so any optimization a vendor pushes shows up the next day. That single design choice changes the trust model. A static benchmark is a snapshot; a daily rerun is a moving record that the open community can challenge by opening an issue against the public benchmark repo.
For teams that came up evaluating terminology servers under the same kind of methodology pressure, choosing a FHIR terminology server: buyers guide for US healthcare covers the same pattern in a different layer of the stack.
The numbers in this benchmark will move. The harness is what you can actually argue about, and arguing about the harness is how any honest performance conversation eventually has to work. The repo includes the Docker Compose files, the k6 scripts, the Synthea generation parameters, and the dashboard configuration, which is the minimum set of pieces a US EHR development team needs to rerun the same test against a workload of their own choosing.