79% more throughput with 45% smaller footprint: How Self-Managed Databases on Amazon EC2 benefit from Datafy Auto-Scaler

Ryan Sayre, Field CTO

I spent years at AWS helping enterprise customers architect high-availability, mission-critical storage on Amazon Elastic Block Store (EBS), and am now working as Field CTO at Datafy. I tend to look at storage through a very specific lens: the physics of the datapath and the operational realities of cloud infrastructure.

When users choose to self-manage PostgreSQL or MySQL on Amazon EC2, they do so for one primary reason: total control and flexibility at the best cost. They want full ownership over the kernel, the filesystem parameters, and the I/O scheduler. Amazon EBS is an extraordinary foundation for that control, offering rock-solid durability, seamless snapshotting, and dependable performance.

However, when you pair the underlying durability of EBS with host-level intelligent data management, you can unlock a tier of operational elasticity and throughput that standard block storage cannot achieve alone.

We set out to measure the exact performance and cost profile of this combination using industry-standard database benchmarks. The results demonstrate that Datafy and Amazon EBS are distinctly “better together”: delivering 79% more transaction throughput while running on 45% less provisioned physical storage capacity, with identical write latency and verified data integrity.

Here is how the storage physics work, what the numbers revealed, and why it matters for production database workloads.

Redefining Elasticity on Amazon EBS

Amazon EBS revolutionized cloud infrastructure with the introduction of Elastic Volumes. In modern environments, teams routinely take advantage of this feature-often automating it through CloudWatch alarms and Lambda functions-to expand volume capacity or modify provisioned IOPS on the fly without unmounting or restarting instances.

Elastic Volumes solved the immediate risk of running out of disk space, but they operate as a strictly one-way street: volumes can easily grow, but they cannot shrink.

In production database environments, workloads are rarely monotonic. Log files rotate, tables are dropped, temporary tablespaces churn, and data gets archived. Yet because EBS volumes cannot be shrunk in place, your storage capacity ratchets permanently to its highest historical watermark. The only native way to reclaim that unallocated space is through manual, out-of-band data migrations-creating new, smaller volumes, synchronizing data, and executing disruptive cutovers during maintenance windows. Consequently, organizations routinely pay for provisioned capacity they no longer need simply to avoid the operational pain of shrinking.

On the performance side, general-purpose gp3 volumes provide an included baseline of 3,000 IOPS and 125 MiB/s at no additional cost. To push beyond that baseline, database teams typically have to pay for provisioned IOPS tiers on gp3 or transition the database to premium volume types like io2 Block Express.

Datafy complements the AWS storage model by introducing host-level intelligent data management:

  1. True Bidirectional Elasticity (Live Growth and Shrinkage): Datafy presents a single, fully elastic logical volume to PostgreSQL. Beneath the filesystem, Datafy dynamically provisions underlying EBS blocks as data lands and automatically releases them back when data is removed-delivering bidirectional elasticity live with zero application downtime.
  2. Aggregated Baseline Performance: Rather than binding a database instance to a single monolithic volume, Datafy distributes storage across an array of underlying EBS volumes. This pools the included baselines of multiple volumes beneath the database, providing a broader performance pipe without requiring high provisioned IOPS tiers.

The Benchmark: HammerDB and the Test Configurations

To measure whether dynamic volume management imposes any measurable penalty at the storage layer, we conducted an empirical evaluation comparing Datafy against standard Amazon EBS gp3 volumes.

Why HammerDB?

We used HammerDB running the TPROC-C workload against PostgreSQL 16. TPROC-C is an open-source, industry-standard transactional benchmark derived from the TPC-C specification. It generates heavy, concurrent, small-block (8 KiB to 11 KiB) random read and write traffic across 400 warehouses with high queue depths. By configuring the workload to access all warehouses uniformly, it forces the database to actively stress the underlying storage subsystem rather than masking disk I/O behind memory caches.

Understanding "Test Arms"

In rigorous scientific and performance benchmarking, separate comparison tracks are formally called test arms-identical, controlled environments where only a single architectural variable is altered. For everyday engineering discussions, you can think of them simply as our two parallel test configurations:

  • The Baseline Configuration (Raw EBS): An Amazon EC2 c7g.2xlarge instance (8 vCPU, 16 GiB RAM, Graviton3) running PostgreSQL 16 on Amazon Linux 2023. The database ran on a standard 200 GiB gp3 data volume (3,000 IOPS baseline, 125 MiB/s) with a separate 100 GiB gp3 Write-Ahead Log (WAL) volume.
  • The Datafy Configuration: An identical c7g.2xlarge instance running the exact same OS and PostgreSQL configuration, but with storage managed by Datafy across a multi-volume array (two 55 GiB volumes striped together for data, and two 27 GiB volumes striped together for WAL).

Both test configurations were initialized from the exact same database snapshot, guaranteeing byte-identical datasets of 400 warehouses (~40 GiB working data) at the start of testing.

The benchmark was executed simultaneously across both environments down to the exact absolute epoch second-eliminating any cross-day network or shared infrastructure variance. Across five concurrency levels (8, 16, 32, 64, and 128 virtual users) repeated three times, every iteration was separated by verified database resets that truncated and cleaned data back to the exact initial build watermark.

The Storage Physics: Throughput and the Management Trade-Off

Every storage abstraction layer requires bookkeeping. Datafy manages thin provisioning, chunk allocations, and logical-to-physical block maps in real time. Handling that metadata introduces a minor operational overhead-in our tests, roughly 10% more device operations per transaction (1.10x) compared to writing directly to an unmanaged EBS volume.

Crucially, because Datafy’s intelligent data management stripes the workload across multiple underlying EBS volumes, it aggregates the available device I/O pipelines.

The underlying array supplied 1.96x the device IOPS of the single volume it replaced. Because that added I/O capacity vastly outstripped the 10% metadata overhead, the database engine unlocked substantial net throughput gains across every concurrency level tested:

Concurrency (Virtual Users)

Raw EBS Throughput (NOPM)

Datafy Throughput (NOPM)

Measured Throughput Lift

 

8 VU

6,750

11,807

+75%

16 VU

6,957

11,668

+68%

32 VU

6,682

12,225

+83%

64 VU

6,596

11,556

+75%

128 VU

6,297

11,711

+86%

Across the entire steady-state run, Datafy delivered an average +79% higher New Orders Per Minute (NOPM).

Equally important: physical device write latency was identical between both environments at 1.01 ms. The metadata management did not slow down individual I/O requests; it simply handled them across a wider datapath.

Capacity Efficiency: 45% Smaller Provisioned Footprint

High throughput is compelling, but cloud storage efficiency must also be cost-effective.

In both configurations, PostgreSQL operated under the impression that it had 300 GiB of available logical storage space (200 GiB for data tables, 100 GiB for transaction logs). The allocated EBS footprint backing that space, however, was fundamentally different:

  • Raw EBS: Provisioned a static 300.0 GiB of allocated EBS capacity.
  • Datafy: Dynamically provisioned 164.0 GiB across its striped array  (110 GiB for data, 54 GiB for WAL) to comfortably hold the ~48 GiB of active database data.

By right-sizing physical block allocation to actual utilization-and retaining the capability to shrink physical storage as data is removed-Datafy eliminated 136 GiB of unneeded storage, representing a 45.3% reduction in provisioned EBS capacity.

Instead of leaving capital tied up in empty blocks to prepare for hypothetical peaks, storage spend directly tracks actual usage.

First-Touch Writes: Bulk Data Loading

A fair question often raised by database architects is: What happens during initial bulk ingest?

When a database writes into virgin storage blocks that have never been allocated, the management layer must allocate and map metadata for the very first time. To test whether this first-touch penalty impacts performance, we measured the time required to build and populate the entire 400-warehouse database (~40 GiB) from scratch on empty volumes across three separate machine pairs.

Across all three test iterations, Datafy completed the initial bulk load 7.6% faster than raw EBS (21 min 43 s versus 23 min 31 s):

I/O Operation Type

Raw EBS Latency

Datafy Latency

Architectural Role

 

Data Page Reads

3.46 ms

1.49 ms

57% faster (Client-blocking query reads)

WAL Writes (Commits)

9.97 ms

3.92 ms

61% faster (Client-blocking commit flushes)

Data-File Writes

7.67 ms

10.21 ms

33% slower (Asynchronous background checkpoints)

The latency distribution illustrates why the system behaves this way:

  • The operations that synchronous database clients wait on-reading random pages from disk and syncing the WAL on commit-were significantly faster under Datafy. Splitting sequential write streams across multiple volume stripes cut tail commit latency substantially (p90 dropped from 35.9 ms to 6.7 ms).
  • Data-file writes showed higher latency in host statistics (10.21 ms vs 7.67 ms), but these writes are handled asynchronously by background writers and checkpointers. The higher host wait times are simply the result of deeper request pipelining: Datafy concurrently processed three times the operations, clearing dirty buffers more aggressively and driving the bulk load to finish sooner.

Direct Data Integrity Verification

In database infrastructure, speed and cost optimization mean nothing if durability is jeopardized. Following our test suites, we verified data consistency across both configurations using native PostgreSQL integrity checks:

  • pg_checksums –check: Read every data page across the entire cluster back through Datafy’s storage layer, returning 0 bad checksums across the 45 GiB installation.
  • pg_amcheck –heapallindexed: Verified that every heap tuple matched its indexed entries across all tables, returning 0 structural corruption errors.
  • Runtime Monitoring: Sampled every two minutes under peak sustained load, PostgreSQL logged 0 runtime checksum failures across more than 27 million pages read.

Self-managing relational databases on AWS EC2 does not require choosing between operational flexibility and raw I/O performance. By complementing the foundational reliability and snapshotting capabilities of Amazon EBS with intelligent data management, you eliminate the one-way growth constraint of traditional block volumes and achieve multi-volume throughput scaling.

The CloudFormation templates, benchmark harness scripts, and raw CloudWatch metrics are easily reproduced. These are available on request so you can reproduce the results in your own environment.

If you want to review the telemetry or test this architecture against your own schemas, please get in touch and we can walk through how your workloads perform on the curve.

In This Article