Living benchmark

Database engines, measured through a real application

Most database benchmarks measure workloads nobody runs: generate uniform rows, pick flattering queries, call the driver directly, publish a number with an X after it. The number is usually true and almost always useless, because your application does not talk to a driver.

So this one swaps the storage engine underneath a working system and changes nothing else. The data, the queries, the indexes and the code path stay exactly as they are, and every measurement below includes the real cost of query translation, serialization, the access check, the soft-delete filter, the account filter and pagination.

First comparison: MongoDB against ScyllaDB. More engines get added here as they are measured, and the rules never bend for a new one.

The reasoning behind these numbers, the two runs we threw away and what we would tell a team facing this decision:

We swapped the database under a real application
Reference engine
Compared with
Factor from
Test kind
Scenario groups in focus

Stating MongoDB against PostgreSQL, median figures. Narrowing the groups dims rows it does not cover — nothing is removed from the page.

Run D — seven engines, strict index parity

Drop-in comparison
MongoDBDocument store
2Won
7Lost
0Parity

Against PostgreSQL, on 9 scenarios, median figures. No overall factor is stated: the spread is the result.

PostgreSQLRelational
6Fastest
2Beaten
1Parity

Reference engine — every factor on this page is stated against it. Its own record above is against the fastest other engine on each of 9 scenarios, median figures.

CockroachDBDistributed relational
0Won
8Lost
1Parity

Against PostgreSQL, on 9 scenarios, median figures. No overall factor is stated: the spread is the result.

ScyllaDBWide-column store
1Won
8Lost
0Parity

Against PostgreSQL, on 9 scenarios, median figures. No overall factor is stated: the spread is the result.

Apache CassandraWide-column store
1Won
8Lost
0Parity

Against PostgreSQL, on 9 scenarios, median figures. No overall factor is stated: the spread is the result.

Apache CouchDBDocument store
0Won
9Lost
0Parity

Against PostgreSQL, on 9 scenarios, median figures. No overall factor is stated: the spread is the result.

FerretDBDocument layer over PostgreSQL
0Won
8Lost
0Parity

Against PostgreSQL, on 8 scenarios, median figures. No overall factor is stated: the spread is the result.

Run D — seven engines, strict index parity — 250 000 measurement records from a seeded generator, so every engine holds byte-identical rows. Factors are computed from the median figures printed in this table, stating MongoDB against Postgres. All 9 scenarios are listed; the group filter dims rows, it does not remove them.
ScenarioRowsMongoDBPostgresCockroachScyllaDBCassandraCouchDBFerretDBFactorRelative
Medianp95Medianp95Medianp95Medianp95Medianp95Medianp95Medianp95
Bulk insert, 250 000 recordsWrite throughput with six secondary indexes already in place.
Writes
250 00044 238 rows/s—11 155 rows/s—536 rows/s—13 305 rows/s—31 328 rows/s—10 001 rows/s—10 399 rows/s—MongoDB 3.97×
Point read by identifier, 200 requests200 sequential primary-key lookups, one round trip each.
Key access
20062 ms75 ms98 ms106 ms222 ms270 ms103 ms112 ms150 ms171 ms3.06 s3.08 s——MongoDB 1.59×
All measurements for one inspectionEvery child record of one parent, on an indexed foreign key.
Key access
4943.8 ms4.2 ms1.9 ms2.5 ms11 ms12 ms4.6 ms5.3 ms8.9 ms24 ms261 ms266 ms16.7 s18.8 sPostgres 2.01×
First sorted page, 50 rowsPage one of the collection, sorted by creation date.
Pagination
501.6 ms1.7 ms1.1 ms1.3 ms9.5 ms11 ms9.7 ms11 ms16 ms17 ms251 ms263 ms20.5 s20.7 sPostgres 1.45×
Deep page, page 200Skip 9 950 rows, return the next 50. What a numbered pager asks for.
Pagination
505.5 ms6.1 ms4.6 ms5.1 ms11 ms15 ms5.15 s5.20 s1.57 s1.66 s30.0 s30.7 s2.50 s2.56 sPostgres 1.18×
Filtered scan on an unindexed booleanEvery record where an approval flag is false. Indexed on neither engine.
Scans and filters
10 00072 ms104 ms35 ms41 ms39 ms55 ms5.09 s5.13 s1.49 s1.54 s27.7 s28.5 s2.29 s2.32 sPostgres 2.05×
Range scan on an unindexed numberOne indexed equality plus a greater-than on an unindexed measurement value.
Scans and filters
1652.5 ms2.9 ms1.4 ms1.4 ms10 ms13 ms3.9 ms4.4 ms7.5 ms8.2 ms252 ms261 ms16.8 s17.3 sPostgres 1.76×
Exact count of the whole collectionHow many records are there. Not an estimate.
Whole-collection work
250 00672 ms73 ms26 ms28 ms169 ms214 ms5.11 s5.15 s1.49 s1.53 s29.8 s34.0 s11.6 s11.9 srow counts differ
Load the whole collectionEvery record, deserialized into the application.
Whole-collection work
250 0061.55 s1.64 s522 ms653 ms531 ms688 ms5.12 s5.21 s1.50 s1.54 s29.7 s29.9 s25.8 s26.2 srow counts differ

Method

  • Median of 10 iterations, 2 warmups discarded, p95 reported alongside
  • 200 sequential point reads per iteration
  • Every query issued through the application's data service, query handler, adapter loader and data layer — never through the driver
  • Row counts compared across all seven engines before any timing was reported
  • Reads measured at 250 000 rows; writes measured afterwards by appending a second 250 000 on top, so every engine writes into a populated, indexed collection
  • Engine versions queried from each server's own interface rather than written down by hand
  • A 120-second per-iteration ceiling — an engine that exceeds it is recorded as a timeout rather than left blank

Index parity

  • All seven engines: secondary indexes on _type, _besiktning_id, _system_id, _ventil_id, _room_id and _anmarkning_id
  • The index SET comes from one declaration in the codebase, read by every adapter, so no engine can drift from another
  • Present on every engine while the rows were written, not added afterwards
  • godkand and varde left unindexed on purpose — but see the caveat: PostgreSQL and CockroachDB carry additional adapter-created indexes, including a GIN index on the whole document

What this run does not establish

  • PostgreSQL and CockroachDB are NOT running unindexed on the scan scenarios. The dp adapter creates idx_fb_matning_doc_gin — a GIN index over the entire JSONB document — alongside the six parity indexes, so their filtered-scan and range-scan numbers are index-assisted where the other five engines are scanning.
  • The write column is an APPEND into a collection already holding 250 000 rows. Run C measured a fill into an empty one. The two write comparisons measure different work and must not be read against each other.
  • MongoDB's row count is 250 007, not 250 000 — it also holds seven real Franska Bukten measurement rows. The seven-row difference is left in rather than deleted from a live collection.
  • Every request was sequential. Nothing here says anything about behaviour under concurrency, which is where ScyllaDB, Cassandra and CockroachDB are designed to pay off.
  • One node each, developer settings, all seven containers resident on one machine. Idle CPU was under 1.5% per container during the run, but no engine was given an isolated host.
  • FerretDB's indexed lookups came out slower than its full scans, which suggests the parity indexes were accepted and then not used. That is unexplained and is reported as measured rather than interpreted.

Run C — strict index parity, indexes present during the insert

Drop-in comparison
MongoDBDocument store
0Won
0Lost
0Parity

Against PostgreSQL, on 0 scenarios, median figures. No overall factor is stated: the spread is the result.

ScyllaDBWide-column store
0Won
0Lost
0Parity

Against PostgreSQL, on 0 scenarios, median figures. No overall factor is stated: the spread is the result.

Run C — strict index parity, indexes present during the insert — 250 006 measurement records, generated from a seeded generator so both engines hold identical rows. Factors are computed from the median figures printed in this table, stating MongoDB against Postgres. All 9 scenarios are listed; the group filter dims rows, it does not remove them.
ScenarioRowsMongoDBScyllaDBFactorRelative
Medianp95Medianp95
Bulk insert, 250 000 recordsWrite throughput with six secondary indexes already in place.
Writes
250 00011 028 rows/s—17 147 rows/s——
Point read by identifier, 200 requests200 sequential primary-key lookups, one round trip each.
Key access
20068 ms83 ms115 ms122 ms—
All measurements for one inspectionEvery child record of one parent, on an indexed foreign key.
Key access
4944.5 ms6.3 ms6.1 ms6.9 ms—
First sorted page, 50 rowsPage one of the collection, sorted by creation date.
Pagination
501.9 ms2.5 ms12 ms13 ms—
Deep page, page 200Skip 9 950 rows, return the next 50. What a numbered pager asks for.
Pagination
506.0 ms6.5 ms6.74 s9.63 s—
Filtered scan on an unindexed booleanEvery record where an approval flag is false. Indexed on neither engine.
Scans and filters
10 00080 ms147 ms6.23 s6.56 s—
Range scan on an unindexed numberOne indexed equality plus a greater-than on an unindexed measurement value.
Scans and filters
1652.5 ms3.7 ms4.9 ms5.7 ms—
Exact count of the whole collectionHow many records are there. Not an estimate.
Whole-collection work
250 00676 ms78 ms6.34 s7.93 s—
Load the whole collectionEvery record, deserialized into the application.
Whole-collection work
250 0061.95 s2.38 s6.43 s7.15 s—

Method

  • Median of 10 iterations, 2 warmups discarded, p95 reported alongside
  • 200 point reads per iteration
  • Every query issued through the application's data service, query handler, adapter loader and data layer — never through the driver
  • Row counts compared across engines before any timing was reported, and they matched on every scenario
  • The whole production database migrated first: 41 collections, zero row-count mismatches

Index parity

  • Both engines: secondary indexes on _type, _besiktning_id, _system_id, _ventil_id, _room_id and _anmarkning_id
  • Verified by reading each engine's own index catalog, not by trusting the setup script
  • Present on both engines while the 250 000 rows were written
  • godkand and varde left unindexed on both sides on purpose, so the scan scenarios are genuinely unindexed work for everyone

What this run does not establish

  • That MongoDB does key lookups faster in general — the point read was sequential, and ScyllaDB crossed a network boundary MongoDB did not.
  • Anything about concurrency. Every request in this run was sequential; ScyllaDB's architecture is built to pay off under load, and load was not applied.
  • Anything about multi-node behaviour, replication or failover. One node, developer mode.
  • How well either engine could perform after being redesigned around it — that is a different test, and it is not this one.

Run B — parity indexes created after the seed

Drop-in comparison
Invalid run, published anyway.The six MongoDB indexes were created after the rows were written, so no write comparison exists for this run and the reads were served by indexes built on settled data rather than maintained through the insert. The reads are close to the valid run; the method still fails.
Run B — parity indexes created after the seed — 250 006 measurement records. Factors are computed from the median figures printed in this table, stating MongoDB against Postgres. All 8 scenarios are listed; the group filter dims rows, it does not remove them.
ScenarioRowsMongoDBScyllaDBFactorRelative
Medianp95Medianp95
Point read by identifier, 200 requests200 sequential primary-key lookups, one round trip each.
Key access
20056 ms66 ms116 ms119 ms—
All measurements for one inspectionEvery child record of one parent, on an indexed foreign key.
Key access
4944.5 ms5.2 ms4.9 ms6.8 ms—
First sorted page, 50 rowsPage one of the collection, sorted by creation date.
Pagination
501.6 ms2.2 ms9.7 ms21 ms—
Deep page, page 200Skip 9 950 rows, return the next 50. What a numbered pager asks for.
Pagination
505.1 ms6.0 ms5.51 s5.62 s—
Filtered scan on an unindexed booleanEvery record where an approval flag is false. Indexed on neither engine.
Scans and filters
10 00081 ms89 ms5.44 s5.54 s—
Range scan on an unindexed numberOne indexed equality plus a greater-than on an unindexed measurement value.
Scans and filters
1652.8 ms8.4 ms4.6 ms6.3 ms—
Exact count of the whole collectionHow many records are there. Not an estimate.
Whole-collection work
250 00666 ms68 ms5.45 s5.47 s—
Load the whole collectionEvery record, deserialized into the application.
Whole-collection work
250 0061.99 s2.50 s5.48 s5.56 s—

Method

  • Median of 10 iterations, 2 warmups discarded
  • 200 point reads per iteration
  • MongoDB seeded first, indexed second — the defect
  • No write measurement taken, because a valid one was not possible

Index parity

  • MongoDB: six secondary indexes, created after the seed
  • ScyllaDB: the same six, created with the table and present during the insert
  • Field parity, but not lifecycle parity

What this run does not establish

  • Any statement about write throughput. There is none in this run.
  • That an index created after a bulk load behaves identically to one maintained during it.

Run A — no index parity

Drop-in comparison
Invalid run, published anyway.MongoDB carried only its default identifier index while ScyllaDB carried six secondary indexes. The three ScyllaDB wins below are artefacts of that gap, not properties of either engine. The timings are real; they answer the wrong question.
Run A — no index parity — 250 006 measurement records. Factors are computed from the median figures printed in this table, stating MongoDB against Postgres. All 9 scenarios are listed; the group filter dims rows, it does not remove them.
ScenarioRowsMongoDBScyllaDBFactorRelative
Medianp95Medianp95
Bulk insert, 250 000 recordsWrite throughput with six secondary indexes already in place.
Writes
250 00027 105 rows/s—21 077 rows/s——
Point read by identifier, 200 requests200 sequential primary-key lookups, one round trip each.
Key access
20051 ms61 ms114 ms124 ms—
All measurements for one inspectionEvery child record of one parent, on an indexed foreign key.
Key access
49488 ms93 ms5.2 ms6.5 ms—
First sorted page, 50 rowsPage one of the collection, sorted by creation date.
Pagination
5086 ms88 ms9.4 ms18 ms—
Deep page, page 200Skip 9 950 rows, return the next 50. What a numbered pager asks for.
Pagination
505.9 ms6.6 ms6.12 s20.8 s—
Filtered scan on an unindexed booleanEvery record where an approval flag is false. Indexed on neither engine.
Scans and filters
10 00085 ms93 ms5.76 s7.36 s—
Range scan on an unindexed numberOne indexed equality plus a greater-than on an unindexed measurement value.
Scans and filters
16594 ms103 ms4.8 ms8.1 ms—
Exact count of the whole collectionHow many records are there. Not an estimate.
Whole-collection work
250 00675 ms78 ms5.61 s6.01 s—
Load the whole collectionEvery record, deserialized into the application.
Whole-collection work
250 0062.30 s2.85 s5.63 s5.94 s—

Method

  • Median of 10 iterations, 2 warmups discarded
  • 200 point reads per iteration
  • Every query issued through the application's data service, not the driver

Index parity

  • MongoDB: the default identifier index only
  • ScyllaDB: six secondary indexes, created automatically by the adapter
  • Not parity — this is the defect the run exists to demonstrate

What this run does not establish

  • Nothing. No comparison in this table is valid.
  • Adding the six matching MongoDB indexes moved the inspection lookup from 88 ms to 4.5 ms and the sorted first page from 86 ms to 1.6 ms.
  • All three ScyllaDB wins disappeared once the index sets matched.

The engines

MongoDB

Document store
Version
8.2.3
Driver
mongodb 7.2.0

Built for

  • Flexible document structures
  • General secondary indexes
  • Ad hoc filtering and sorting
  • Range queries and aggregation
  • Query patterns that change after launch

How it was deployed

  • Native process on 127.0.0.1:27017, reached over loopback
  • Whole machine available, no container and no virtualisation layer
  • The friendliest deployment in the comparison — stated because it matters

ScyllaDB

Wide-column store
Version
6.2.3-0.20250119.bff9ddde1283, CQL 3.3.1 (Cassandra-compat 3.0.8)
Driver
cassandra-driver 4.9.0, consistency LOCAL_ONE, fetch size 5000

Built for

  • High write throughput
  • Predictable low latency under concurrency
  • Horizontal scaling across nodes
  • Partition-key reads
  • Data models designed around known queries

How it was deployed

  • Docker container from image scylladb/scylla:6.2
  • Running under WSL2 Ubuntu 26.04, reached through WSL port forwarding
  • Two CPU shards and 4 GB of memory, with --overprovisioned 1 --developer-mode 1
  • Crosses a network boundary MongoDB did not have to cross

PostgreSQL

Relational
Version
17.10 (Debian 17.10-1.pgdg13+1)
Driver
pg 8.22.0

Built for

  • Relational integrity and transactions
  • Complex joins and aggregation
  • Mixed read and write workloads on one node

How it was deployed

  • Docker container from image postgres:17, 2 GB memory
  • Published on port 5433 — this machine runs a native PostgreSQL 17 on the default port that the benchmark must not touch
  • Documents stored as JSONB, indexed fields as expression indexes over doc ->> 'field'
  • Carries four additional adapter-created indexes on top of the six parity ones, including a GIN index over the whole document — so its scan scenarios are index-assisted

Apache Cassandra

Wide-column store
Version
5.0.8, CQL 3.4.7
Driver
cassandra-driver 4.9.0, consistency LOCAL_ONE, fetch size 5000

Built for

  • High write throughput
  • Multi-datacenter replication
  • Query-driven table design

How it was deployed

  • Docker container from image cassandra:5.0, 4 GB memory, 3 GB JVM heap
  • Sat at 3.686 GB of its 4 GB cap during the run — under real garbage-collection pressure
  • Shares the entire CQL adapter with ScyllaDB, so anything that differs between the two is the runtime and nothing else
  • Its COUNT(*) exceeded the server's own 5-second read timeout, so the adapter counts by paged scan instead

Couchbase

Document store
Not yet measured

Built for

  • Low-latency key-value reads from cache
  • SQL-style querying over documents
  • Scaling read and index services independently

How it was deployed

  • Not deployed yet.

CockroachDB

Distributed relational
Version
CCL v24.3.5 (built 2025-02-03, go1.22.8)
Driver
pg 8.22.0 — the PostgreSQL driver, unchanged

Built for

  • Horizontal scale without giving up SQL
  • Serializable distributed transactions
  • Surviving the loss of a node or a region

How it was deployed

  • Docker container from image cockroachdb/cockroach:v24.3.5, single node, --insecure, 3 GB memory
  • Shares the entire PostgreSQL adapter and the same expression indexes
  • Carries the same four extra adapter-created indexes as PostgreSQL, GIN included
  • Returns its row counts as strings where every other engine returns numbers

Apache CouchDB

Document store
Version
3.4.3 (e12b967d7)
Driver
none — CouchDB's whole interface is HTTP and JSON, so fetch is the client

Built for

  • Multi-master replication, including to offline clients
  • Crash-only durability
  • Reading through precomputed views

How it was deployed

  • Docker container from image couchdb:3.4, 2 GB memory
  • One CouchDB database per dp collection, which makes an exact count a metadata read rather than a scan
  • dp's underscore-prefixed fields are ILLEGAL here — CouchDB reserves that prefix and answers doc_validation: Bad special document member: _type. Every field is renamed on write and back on read.
  • A Mango index is required to sort: this is the one engine that refuses a query rather than degrading, so the adapter sorts in the application when no index covers the field

FerretDB

Document layer over PostgreSQL
Version
v1.24.0, presenting MongoDB wire 7.0.42
Driver
mongodb 7.2.0 — the MongoDB driver, unchanged

Built for

  • Keeping MongoDB drivers and queries on an open-source stack
  • Reusing PostgreSQL operations, backups and tooling
  • Migrating away from a document store without rewriting queries

How it was deployed

  • Docker container from image ghcr.io/ferretdb/ferretdb:1.24.0, 2 GB memory
  • Backed by the same PostgreSQL 17.10 container, reached over a private docker network
  • Needed no adapter code at all — dp's MongoDB queries arrive unchanged
  • No change streams, so dp subscriptions cannot be served: the one capability gap here that is not a performance question
  • Its indexed lookups measured SLOWER than its full scans, which suggests the parity indexes were accepted and then not used. Unexplained.
The rules

How the next engine gets tested

PostgreSQL, Cassandra, Couchbase, CockroachDB, CouchDB and FerretDB can all go through the same harness. These eight rules stay fixed, because the moment one of them bends for a single engine the whole comparison is worthless.

  1. Same application

    The real data-access path, not a driver script written for the occasion. A migration happens inside an application, and that is where its cost shows up.

  2. Same data

    The production database migrated in full, plus a generated set built from a seeded generator so every engine holds byte-identical rows.

  3. Same scenarios

    The same functional queries, expressed through the application rather than hand-tuned per engine. Each engine answers the question its own way, and the cost of its way is the result.

  4. Verify results first

    Row counts must match across engines before a single timing is reported. A fast query returning the wrong rows is not an optimization, it is an efficient defect.

  5. Equivalent indexes

    Verified from each engine's own catalog, and present while the data is written. Comparing an indexed store against an unindexed one measures your setup script.

  6. Document the environment

    Including the parts that disadvantage one side. This run's WSL deployment disadvantaged ScyllaDB, and that is stated on its card rather than left out.

  7. Separate the modes

    Drop-in results and engine-optimized results never share a table. Mixed together, the winner is whichever engine got the most engineering attention.

  8. Publish the bad runs

    Both invalid runs are on this page, labelled. They taught more about benchmarking than the valid one did, and a method nobody can see fail is a method nobody can check.