data-ingestion

Open repos and models their makers tagged data-ingestion, most starred first. The same for everyone.

AI on my own machineRepo

seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

  • 9.8k
  • Java
  • Apache-2.0
  • 7 h ago
apacheTake it
GitHubRepo

paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

  • 3.5k
  • Java
  • Apache-2.0
  • 8 h ago
apacheTake it
DatabasesRepo

olake-go

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

  • 2.1k
  • Go
  • Apache-2.0
  • 10 h ago
datazip-incTake it
AI on my own machineRepo

opencrawling

Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows

  • 29
  • Java
  • Apache-2.0
  • 17 h ago
opencrawlingTake it
Developer toolsRepo

k2i

K2I - Kafka to Iceberg streaming ingestion engine. A Rust CLI tool inspired by Moonlink architecture that consumes from Kafka, buffers with Apache Arrow for sub-second query freshness, and writes to Apache Iceberg tables.

  • 23
  • Rust
  • 4 mo ago
osodevopsTake it
Data and analyticsRepo

TMDB-Movies-End-to-End-ETL-and-ML-Pipeline

This project encompasses end-to-end ETL and ML pipeline development. Data ingestion from TMDB API covered top-rated, current, upcoming, and popular movies with genres. Performed EDA to derive several valuable insights and observations. Developed a regression model with 97% r2 score to predict average movie ratings accurately.

  • 0
  • Python
  • Apache-2.0
  • 15 mo ago
SayamAltTake it
Search GitHub and Hugging Face for data-ingestion

Get , open it once and sign in. Then come back here and press the button again.

data-ingestion: open source repos and models on Take My Repo