Building Ferro: rewriting the JVM and Python data stack in Rust as a solo developer

2026-04-26T00:00:00+00:00

I’ve spent the last several months writing Rust. The output is what I’m going to call “Ferro”: a set of Rust-native rewrites of the JVM and Python data infrastructure most teams take for granted — search, streaming, orchestration, identity, package registries, log shipping, monitoring. There are seven product repositories now, on the order of 800,000 lines of own-code Rust, around 18,000 tests. I am a single developer.</p>

This post is about why</em> and how</em>. It is also the public-launch post for ferro-protocols</code></a> — the first public-facing piece of that work. ferro-protocols</code> is a workspace of small, narrowly scoped crates carved out of the larger ecosystem and published under Apache-2.0, so the parts of the work that have value independent of the products themselves can ship to crates.io and be used by anyone.</p>

Why rewrite all of it</h2> Most of the JVM data stack — Elasticsearch, Kafka, Logstash, Metricbeat, Filebeat — was designed in an era where a JVM-resident service with a 4 GB heap and a 60-second GC pause window was the acceptable cost of doing business. The Python data stack — Airflow, the SDK ecosystem around it — was designed when “spawn a CPython worker per task” was a reasonable scheduling primitive.</p> It is no longer the right answer for most installations. Cold-start matters. Memory matters. P99 latency matters. Container density matters. The Datadog-shaped vendors are pricing on agent footprint because that’s where the squeeze is. A 30 MB resident-set, single-binary Rust replacement of Filebeat ships on every container in a fleet without making the bill irrational; a 200 MB JVM does not.</p> Rust has been ready for this rewrite for a couple of years now. The async story is solid (Tokio is mature, rustls is mature, openraft exists, sqlx works). The ecosystem has stabilised on permissive licensing. cargo</code> makes producing a static binary trivial. What’s been missing is people doing the work</em>.</p> I started doing the work.</p> What’s in the ecosystem today</h2> Concretely, by individual product:</p> Product</th> Compat target</th> Status as of 2026-04-26</th></tr></thead> FerroSearch</strong></td> Elasticsearch 9.3 / OpenSearch 2.19 + 3.6 / OpenSearch Dashboards 3.6</td> YAML REST 100% on ES, 100% on OS 2.19, 100% on OS 3.6, 100% on OSD 3.6 api_integration, plus 92% on Kibana 9.3 in Docker mode. Single Rust engine.</td></tr> FerroStream</strong></td> Kafka 4.2 wire</td> Brokers, KRaft, Tiered Storage, Share Groups (KIP-932), Streams API surface, OMB benchmarking harness. Performance has caught up with Redpanda on consume in the synthetic suites; the more-complex multi-topic OMB workloads are still being root-caused.</td></tr> FerroFlow</strong></td> TPC-H + Kafka SQL connectors</td> TPC-H 22/22, ~275 compat tests, six real connectors (sink/source/TLS/SASL/S3/GCS/partition/LSM).</td></tr> FerroAuth</strong></td> Keycloak 26 IdP wire</td> Runnable IdP with WebAuthn L3, mTLS, LDAP sync, HA stores. Passkey-issuance live smoke pass.</td></tr> FerroRepo</strong></td> OCI Distribution v1.1 + Cargo + Maven + PyPI + Helm + Go modules + APT/YUM, with Sigstore + SLSA + TUF</td> v0.1.0 GA tag local, 1391 tests pass, OCI conformance suite green.</td></tr> FerroAir</strong></td> Apache Airflow 3</td> Phase 0 done — PyO3 + airflow.sdk integration proven, AIP-72 stub, 100% parser parity on a 64-DAG sample.</td></tr> FerroBeat / FerroHeartbeat / FerroMetric / FerroStash</strong></td> Filebeat / Heartbeat / Metricbeat / Logstash</td> All running. FerroMetric reaches a Linux 9.5 MB RSS at 50K events/s (compared with ~31 MB for FerroBeat and a multi-hundred-MB JVM baseline for the originals).</td></tr> </tbody></table> This is “the largest single-developer rewrite of a major data infrastructure category I can find,” but I want to be careful not to over-claim. Plenty of those numbers come from synthetic test suites; benchmarks against production-shaped workloads are an ongoing, partially-finished exercise (the OMB hang I mentioned above is a real, current bug — a non-daemon ScheduledThreadPoolExecutor</code> in the OMB Java client side that prevents JVM exit, fixed in the harness but not yet re-run on EC2).</p> The shape of the work</h2> Each Ferro product is a Rust workspace of dozens of crates. Most of them are private monorepos. They share a few common rules:</p> forbid(unsafe_code)</code> workspace-wide. Exceptions need a SAFETY:</code> comment and live under feature flags (py-embed</code> in FerroAir is the one current case, for the PyO3 boundary).</li> cargo clippy --all-targets -- -D warnings</code> with pedantic elevated to warn</code>. No exceptions.</li> cargo deny check</code> clean. License allowlist (Apache-2.0, MIT, BSD family, ISC, MPL-2.0, etc.). openssl</code>, openssl-sys</code>, native-tls</code> blocked — we use rustls everywhere.</li> 80%+ line coverage target. Some products are well above (FerroBeat 87.6%, FerroMetric 86.4%); the cap is set by what’s measurable in a realistic CI tier.</li> Zero unwrap</code> on user-controlled paths. Zero TODO</code> / FIXME</code> / HACK</code> in source.</li> </ul> These rules are non-negotiable in any of the seven repos. The result is that when I extract a module into a public crate (more on that in a moment), I do not have to spend time tightening it up to OSS quality — it already meets that bar.</p> Diligence: I run myself through synthetic DD rounds</h2> The work is not just code. Every Ferro product has a due-diligence/</code> directory with round-by-round findings logs from synthetic investor-style audits. I run these through Codex CLI (with GPT-5 class models) and Anthropic Claude in parallel — same brief, two adversaries — and merge the findings. They go through identification (“R1 surfaced 54 actionable findings”), then remediation rounds (“R2 closed 38 of those, surfaced 16 new ones, …”) until the remediation backlog stabilises.</p> For perspective: FerroMetric reached R14 Clean Pass</strong> with 0 actionable findings remaining. FerroAuth is at R14 with 4 medium / 3 low / 1 info open. The findings logs are commit-pinned, so “this round happened” is auditable, and the fixes are commit-pinned, so “this finding was closed by this commit</em>” is auditable.</p> This is the part I want potential acquirers and partners to look at — not a list of features. The features are easy to claim. What’s hard is showing the work behind</em> them.</p> The first public release: ferro-protocols</h2> The publicly-published part of all of this starts with ferro-protocols</code></a> — a mono-repo workspace whose individual crates are extracted from the products above. The first two crates in the repo are:</p> ferro-lumberjack</code> (v0.1.0)</h3> The Logstash Lumberjack v2 wire protocol. Filebeat and Heartbeat agents speak it. Logstash receives it. Until now, no Rust crate implemented either side.</p> ferro-lumberjack</code> is the frame codec, async client, and async server</strong> for Lumberjack v2, with TLS in both directions via rustls. The frame codec is pure-data (no I/O, usable from any runtime); the client + server are Tokio-only. There are 66 tests including 6 real-socket end-to-end client↔server tests with self-signed TLS.</p> It is at v0.1.0</code> because the implementation is extracted from production use in FerroBeat and FerroHeartbeat — not new code. The server side is fresh, but exercised by real client↔server e2e tests.</p> ferro-lumberjack on crates.io</a> · docs.rs</a></p> ferro-airflow-dag-parser</code> (v0.0.1)</h3> A static AST-based extractor for Apache Airflow™ Python DAG files. Recovers dag_id</code>, task_ids</code>, dependencies, schedule, and seven categories of dynamic-fallback markers — without running the source.</p> This is a primitive that, as far as I can tell, does not exist elsewhere in any language</strong>. Apache Airflow’s reference scheduler imports every dags/*.py</code> through CPython on every poll cycle, which is fine for small fleets and a heavy tax on big ones. FerroAir uses this crate as a fast-path that handles the static fraction of DAGs in microseconds, and only routes to CPython the ones whose structure depends on runtime state.</p> The crate is in alpha (v0.0.x</code>) because the public API surface is still shaping; the implementation has 75 tests and ships with a panic_safe</code> shim that catches upstream parser panics found by fuzz testing — the kind of pre-publish hardening that turns a discovered bug into a regression test instead of a production crash.</p> ferro-airflow-dag-parser on crates.io</a> · docs.rs</a></p> The roadmap (yes, it includes “all of the above”)</h2> ferro-protocols</code> is a mono-repo on purpose. As each product matures past the “hardening behind closed doors” stage, the parts that have value as standalone crates will surface here. Already planned for the next several weeks, in rough order:</p> ferro-cargo-registry-server</code> — Cargo Alternative Registry server-side (sparse + git index; first public Rust crate to do this).</li> ferro-maven-layout</code> — Maven Repository Layout 2.0 (no Rust implementation exists yet).</li> ferro-oci-server</code> — OCI Distribution Specification v1.1 server-side primitives (clients exist; servers do not).</li> ferro-painless</code> — Elasticsearch Painless lexer/parser/JIT (extracted from FerroStash).</li> ferro-esql-parser</code> — ES|QL parser (the Elasticsearch query language; absent from the Rust ecosystem).</li> ferro-aql-parser</code> — Artifactory AQL.</li> ferro-pep503-pep691</code> — PyPI primitives.</li> ferro-go-module-proxy</code> — Go module proxy server-side.</li> ferro-helm-chart-repo</code> — Helm 3 chart repository primitives.</li> ferro-keycloak-realm-import</code> — Keycloak realm JSON 26.x.</li> ferro-logstash-dsl-parser</code> — Logstash .conf</code> DSL.</li> </ul> All Apache-2.0. All extracted from production use (or to be production-tested in their parent product first). All published under a Developer Certificate of Origin contributor flow — no CLA — so contributing is friction-free.</p> What I am not claiming</h2> A tactical list of things that are still open:</p> An acquisition track for FerroStream and adjacent products is open but not closed.</strong> FerroStream is the part of this work intended for sale to a strategic buyer; conversations exist but nothing has signed.</li> OMB benchmarks against real cloud are not yet public.</strong> Synthetic benches show parity-or-better; the real EC2 13-hour workload run with the harness fixes is queued, not done.</li> FerroAir Phase 1 is not done.</strong> Phase 0 is a 4-pillar PoC with 100% parser parity over a 64-DAG sample. The full Airflow 3.x compat surface is Phase 1 work.</li> Marketplace listings (FerroAuth Pro, FerroRepo Pro, FerroAir Pro) are pending company formation.</strong> They are intentionally gated on the Track A close — premature commercial launch under personal-name billing is the wrong move.</li> I am not going to claim this is all “production-ready” for your environment.</strong> It is production-tested in mine. Yours may be different.</li> </ol> The diligence rounds, the test counts, and the per-product README files are the load-bearing artefacts. Look at those, not at marketing copy.</p> How to actually use it</h2> [</span>dependencies</span>]</span></span> ferro-lumberjack</span> =</span> "</span>0.1</span>"</span></span> ferro-airflow-dag-parser</span> =</span> "</span>0.0</span>"</span></span></code></pre> The ferro-protocols</code> repo has the canonical READMEs, contribution flow (DCO), security policy, and the workflow files used by CI. Issues + PRs welcome on the repo. Discussions are a better channel for design questions.</p> Why I’m publishing now</h2> A single-developer ecosystem rewrite of this scale is unusual enough that the natural skeptical question is “how is this real?”</strong> The answer is to put the parts that can</em> be published into the open, on terms (Apache-2.0, DCO, on crates.io with semver) that make them easy to inspect, depend on, and contribute to. Acquirers will look at the repo. Hiring managers will look at the repo. Other Rust-data-infrastructure people will look at the repo. The way to make the work falsifiable is to ship.</p> If you are working on Airflow internals and need a static DAG parser, or you operate Logstash and need a Beats sender or receiver in Rust, you can install both crates today.</p> The rest of the ecosystem will surface here as it matures.</p> — Y.U.</p>

Product</th>	Compat target</th>	Status as of 2026-04-26</th></tr></thead>
FerroSearch</strong></td>	Elasticsearch 9.3 / OpenSearch 2.19 + 3.6 / OpenSearch Dashboards 3.6</td>	YAML REST 100% on ES, 100% on OS 2.19, 100% on OS 3.6, 100% on OSD 3.6 api_integration, plus 92% on Kibana 9.3 in Docker mode. Single Rust engine.</td></tr>
FerroStream</strong></td>	Kafka 4.2 wire</td>	Brokers, KRaft, Tiered Storage, Share Groups (KIP-932), Streams API surface, OMB benchmarking harness. Performance has caught up with Redpanda on consume in the synthetic suites; the more-complex multi-topic OMB workloads are still being root-caused.</td></tr>
FerroFlow</strong></td>	TPC-H + Kafka SQL connectors</td>	TPC-H 22/22, ~275 compat tests, six real connectors (sink/source/TLS/SASL/S3/GCS/partition/LSM).</td></tr>
FerroAuth</strong></td>	Keycloak 26 IdP wire</td>	Runnable IdP with WebAuthn L3, mTLS, LDAP sync, HA stores. Passkey-issuance live smoke pass.</td></tr>
FerroRepo</strong></td>	OCI Distribution v1.1 + Cargo + Maven + PyPI + Helm + Go modules + APT/YUM, with Sigstore + SLSA + TUF</td>	v0.1.0 GA tag local, 1391 tests pass, OCI conformance suite green.</td></tr>
FerroAir</strong></td>	Apache Airflow 3</td>	Phase 0 done — PyO3 + airflow.sdk integration proven, AIP-72 stub, 100% parser parity on a 64-DAG sample.</td></tr>
FerroBeat / FerroHeartbeat / FerroMetric / FerroStash</strong></td>	Filebeat / Heartbeat / Metricbeat / Logstash</td>	All running. FerroMetric reaches a Linux 9.5 MB RSS at 50K events/s (compared with ~31 MB for FerroBeat and a multi-hundred-MB JVM baseline for the originals).</td></tr> </tbody></table> This is “the largest single-developer rewrite of a major data infrastructure category I can find,” but I want to be careful not to over-claim. Plenty of those numbers come from synthetic test suites; benchmarks against production-shaped workloads are an ongoing, partially-finished exercise (the OMB hang I mentioned above is a real, current bug — a non-daemon `ScheduledThreadPoolExecutor</code> in the OMB Java client side that prevents JVM exit, fixed in the harness but not yet re-run on EC2).</p>` The shape of the work</h2> Each Ferro product is a Rust workspace of dozens of crates. Most of them are private monorepos. They share a few common rules:</p> forbid(unsafe_code)</code> workspace-wide. Exceptions need a SAFETY:</code> comment and live under feature flags (py-embed</code> in FerroAir is the one current case, for the PyO3 boundary).</li> cargo clippy --all-targets -- -D warnings</code> with pedantic elevated to warn</code>. No exceptions.</li> cargo deny check</code> clean. License allowlist (Apache-2.0, MIT, BSD family, ISC, MPL-2.0, etc.). openssl</code>, openssl-sys</code>, native-tls</code> blocked — we use rustls everywhere.</li> 80%+ line coverage target. Some products are well above (FerroBeat 87.6%, FerroMetric 86.4%); the cap is set by what’s measurable in a realistic CI tier.</li> Zero unwrap</code> on user-controlled paths. Zero TODO</code> / FIXME</code> / HACK</code> in source.</li> </ul> These rules are non-negotiable in any of the seven repos. The result is that when I extract a module into a public crate (more on that in a moment), I do not have to spend time tightening it up to OSS quality — it already meets that bar.</p> Diligence: I run myself through synthetic DD rounds</h2> The work is not just code. Every Ferro product has a due-diligence/</code> directory with round-by-round findings logs from synthetic investor-style audits. I run these through Codex CLI (with GPT-5 class models) and Anthropic Claude in parallel — same brief, two adversaries — and merge the findings. They go through identification (“R1 surfaced 54 actionable findings”), then remediation rounds (“R2 closed 38 of those, surfaced 16 new ones, …”) until the remediation backlog stabilises.</p> For perspective: FerroMetric reached R14 Clean Pass</strong> with 0 actionable findings remaining. FerroAuth is at R14 with 4 medium / 3 low / 1 info open. The findings logs are commit-pinned, so “this round happened” is auditable, and the fixes are commit-pinned, so “this finding was closed by this commit</em>” is auditable.</p> This is the part I want potential acquirers and partners to look at — not a list of features. The features are easy to claim. What’s hard is showing the work behind</em> them.</p> The first public release: ferro-protocols</h2> The publicly-published part of all of this starts with ferro-protocols</code></a> — a mono-repo workspace whose individual crates are extracted from the products above. The first two crates in the repo are:</p> ferro-lumberjack</code> (v0.1.0)</h3> The Logstash Lumberjack v2 wire protocol. Filebeat and Heartbeat agents speak it. Logstash receives it. Until now, no Rust crate implemented either side.</p> ferro-lumberjack</code> is the frame codec, async client, and async server</strong> for Lumberjack v2, with TLS in both directions via rustls. The frame codec is pure-data (no I/O, usable from any runtime); the client + server are Tokio-only. There are 66 tests including 6 real-socket end-to-end client↔server tests with self-signed TLS.</p> It is at v0.1.0</code> because the implementation is extracted from production use in FerroBeat and FerroHeartbeat — not new code. The server side is fresh, but exercised by real client↔server e2e tests.</p> ferro-lumberjack on crates.io</a> · docs.rs</a></p> ferro-airflow-dag-parser</code> (v0.0.1)</h3> A static AST-based extractor for Apache Airflow™ Python DAG files. Recovers dag_id</code>, task_ids</code>, dependencies, schedule, and seven categories of dynamic-fallback markers — without running the source.</p> This is a primitive that, as far as I can tell, does not exist elsewhere in any language</strong>. Apache Airflow’s reference scheduler imports every dags/*.py</code> through CPython on every poll cycle, which is fine for small fleets and a heavy tax on big ones. FerroAir uses this crate as a fast-path that handles the static fraction of DAGs in microseconds, and only routes to CPython the ones whose structure depends on runtime state.</p> The crate is in alpha (v0.0.x</code>) because the public API surface is still shaping; the implementation has 75 tests and ships with a panic_safe</code> shim that catches upstream parser panics found by fuzz testing — the kind of pre-publish hardening that turns a discovered bug into a regression test instead of a production crash.</p> ferro-airflow-dag-parser on crates.io</a> · docs.rs</a></p> The roadmap (yes, it includes “all of the above”)</h2> ferro-protocols</code> is a mono-repo on purpose. As each product matures past the “hardening behind closed doors” stage, the parts that have value as standalone crates will surface here. Already planned for the next several weeks, in rough order:</p> ferro-cargo-registry-server</code> — Cargo Alternative Registry server-side (sparse + git index; first public Rust crate to do this).</li> ferro-maven-layout</code> — Maven Repository Layout 2.0 (no Rust implementation exists yet).</li> ferro-oci-server</code> — OCI Distribution Specification v1.1 server-side primitives (clients exist; servers do not).</li> ferro-painless</code> — Elasticsearch Painless lexer/parser/JIT (extracted from FerroStash).</li> ferro-esql-parser</code> — ES\|QL parser (the Elasticsearch query language; absent from the Rust ecosystem).</li> ferro-aql-parser</code> — Artifactory AQL.</li> ferro-pep503-pep691</code> — PyPI primitives.</li> ferro-go-module-proxy</code> — Go module proxy server-side.</li> ferro-helm-chart-repo</code> — Helm 3 chart repository primitives.</li> ferro-keycloak-realm-import</code> — Keycloak realm JSON 26.x.</li> ferro-logstash-dsl-parser</code> — Logstash .conf</code> DSL.</li> </ul> All Apache-2.0. All extracted from production use (or to be production-tested in their parent product first). All published under a Developer Certificate of Origin contributor flow — no CLA — so contributing is friction-free.</p> What I am not claiming</h2> A tactical list of things that are still open:</p> An acquisition track for FerroStream and adjacent products is open but not closed.</strong> FerroStream is the part of this work intended for sale to a strategic buyer; conversations exist but nothing has signed.</li> OMB benchmarks against real cloud are not yet public.</strong> Synthetic benches show parity-or-better; the real EC2 13-hour workload run with the harness fixes is queued, not done.</li> FerroAir Phase 1 is not done.</strong> Phase 0 is a 4-pillar PoC with 100% parser parity over a 64-DAG sample. The full Airflow 3.x compat surface is Phase 1 work.</li> Marketplace listings (FerroAuth Pro, FerroRepo Pro, FerroAir Pro) are pending company formation.</strong> They are intentionally gated on the Track A close — premature commercial launch under personal-name billing is the wrong move.</li> I am not going to claim this is all “production-ready” for your environment.</strong> It is production-tested in mine. Yours may be different.</li> </ol> The diligence rounds, the test counts, and the per-product README files are the load-bearing artefacts. Look at those, not at marketing copy.</p> How to actually use it</h2> [</span>dependencies</span>]</span></span> ferro-lumberjack</span> =</span> "</span>0.1</span>"</span></span> ferro-airflow-dag-parser</span> =</span> "</span>0.0</span>"</span></span></code></pre> The ferro-protocols</code> repo has the canonical READMEs, contribution flow (DCO), security policy, and the workflow files used by CI. Issues + PRs welcome on the repo. Discussions are a better channel for design questions.</p> Why I’m publishing now</h2> A single-developer ecosystem rewrite of this scale is unusual enough that the natural skeptical question is “how is this real?”</strong> The answer is to put the parts that can</em> be published into the open, on terms (Apache-2.0, DCO, on crates.io with semver) that make them easy to inspect, depend on, and contribute to. Acquirers will look at the repo. Hiring managers will look at the repo. Other Rust-data-infrastructure people will look at the repo. The way to make the work falsifiable is to ship.</p> If you are working on Airflow internals and need a static DAG parser, or you operate Logstash and need a Beats sender or receiver in Rust, you can install both crates today.</p> The rest of the ecosystem will surface here as it matures.</p> — Y.U.</p>