Data engineering
The pipelines that get your data where AI and reporting can use it, fresh and correct.
What we build
- Change data capture. Stream every change out of your databases as it happens (Kafka Connect, Flink, Qlik), from one source or many, with schema changes and ordering handled.
- Lakehouse builds. Apache Iceberg tables with a catalog, partitioning and compaction, on storage you own.
- Warehouse integration. Batch or streaming sync into Snowflake or the warehouse you already use.
- Streaming jobs. Flink jobs that transform data in flight and deliver it exactly once.
Experience
- Moved billions of rows between SQL Server, PostgreSQL and Snowflake for a lending platform, across private networks and client VPNs.
- Designed and ran Kafka and Kafka Connect for that platform, including a custom converter for awkward database keys.
- shopify-lakehouse: an open-source reference build, Kafka to Flink to Iceberg tables, with Spark compaction and SQL on top.
Projects are quoted as a fixed bid after an assessment, or as time and materials when the scope is still moving.
Not sure where to start? A Data Foundation assessment is the usual first step.