PeerDB launched on Y Combinator's "Launch YC" this week!
"Data-store first ETL/ELT, enabling 10x faster syncs and transforms"
PeerDB is a postgres-compatible SQL interface to seamlessly integrate your data stores. Data-store nativity at it’s core, enabling scalable ETL.
Founded by Sai Krishna Srirampur & Kaushik Iska. Prior to PeerDB, Sai worked at Microsoft leading Solutions Eng. for all PostgreSQL services on Azure. He was an early engineer at Citus Data (YC S11) and saw it through the Microsoft acquisition. Kaushik built operating systems and led data teams at Google, Safegraph and Palantir. He also represented India in ACM ICPC World Finals. They're are on a mission to build the next generation ETL/ELT!
HOW IT WORKS
PeerDB is a Postgres-compatible SQL interface to seamlessly integrate multiple data-stores. It enables you to sync, transform and query data across your stores using simple SQL commands. PeerDB takes a datastore native approach in engineering — enabling 10x faster and a highly reliable ETL experience for you.
🔴 Problem
Existing ETL/ELT tools primarily focus on supporting a wide range of connectors at the expense of delivering high quality ones. This becomes evident when your workloads need scale or have demanding feature requirements. While Sai was working at Microsoft, it was common to see large enterprise customers try out the existing ETL tools and fail – as the tools were not meeting performance and reliability SLAs or were lacking the required features. Customers spent significant time and resources to build and manage in-house data-pipelines.
🟢 PeerDB's Solution
Data-store nativity at its core, enabling 10x faster ETL/ELT
PeerDB takes a data-store first approach to data movement. It supports highly adopted stores, and implements multiple infrastructural and data-store native optimizations – providing a blazing fast and a feature-rich ETL experience.
They are starting their journey by providing the world's best data-integration/ETL experience for PostgreSQL, one of most used and popular databases. For example on Postgres they parallelize initial load for larger tables using CTID scans. This yields orders of magnitude speedups - syncing 100s of GB is reduced from days to minutes.
PeerDB's architecture is designed for real-time syncs, which enables 10x faster Change Data Capture with data-freshness of a few seconds even at large throughputs (10k+ tps). For example, in a sync from Postgres to BigQuery or Snowflake, PeerDB is 10 times faster than other tools and provides <30s data-freshness guarantees.
SQL as the ETL/ELT standard
The Postgres compatible SQL layer for ETL makes life of data-engineers very easy. They can develop pipelines using a framework they are familiar with, without needing to deal with custom UIs and REST APIs. They can use Postgres' 1000s of integrations to build and manage ETL.
PeerDB extends Postgres' SQL grammar with a few new intuitive SQL commands to enable real-time data streaming across stores. As PeerDB evolves, it will make this grammar richer and make it the standard for ETL/ELT.
