seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
- 9.8k
- Java
- Apache-2.0
- 7 h ago
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
Concurrent and multi-stage data ingestion and data processing with Elixir
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL
The Data Integration Library project provides a library of generic components based on a multi-stage architecture for data ingress and egress.
Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows
K2I - Kafka to Iceberg streaming ingestion engine. A Rust CLI tool inspired by Moonlink architecture that consumes from Kafka, buffers with Apache Arrow for sub-second query freshness, and writes to Apache Iceberg tables.
RAG engine that loads external data with Render Workflows into pgvector and serves cited search over MCP
This project encompasses end-to-end ETL and ML pipeline development. Data ingestion from TMDB API covered top-rated, current, upcoming, and popular movies with genres. Performed EDA to derive several valuable insights and observations. Developed a regression model with 97% r2 score to predict average movie ratings accurately.