pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
- 62k
- Python
- 10 h ago

Reliable message delivery using the transactional outbox pattern for Kotlin
Repo softwaremill/okapi, read from GitHub 1 min ago. softwaremill has not joined Take My Repo.
softwaremill/okapi is a public repo on GitHub by softwaremill. Reliable message delivery using the transactional outbox pattern for Kotlin. It declares the Apache-2.0 license on GitHub. It is written mainly in Kotlin, has 14 stars and last changed on 2026-11-18. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-20.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Reliable message delivery for Kotlin and Java services, using the transactional outbox pattern.
Okapi is a Kotlin/JVM library implementing the transactional outbox pattern. Messages are stored in the database within the same transaction as your business operation, then delivered asynchronously over HTTP or Kafka. This prevents messages from being lost between the database commit and the delivery attempt, without requiring distributed transactions.
This example assumes an existing Spring Boot application connected to PostgreSQL, with a configured DataSource and PlatformTransactionManager.
1. Add the dependencies. The BOM keeps module versions aligned:
dependencies {
implementation(platform("com.softwaremill.okapi:okapi-bom:1.0.0"))
implementation("com.softwaremill.okapi:okapi-core")
implementation("com.softwaremill.okapi:okapi-postgres")
implementation("com.softwaremill.okapi:okapi-http")
implementation("com.softwaremill.okapi:okapi-spring-boot")
runtimeOnly("org.liquibase:liquibase-core")
}Liquibase creates the okapi_outbox table on startup — no changelog edits on your side. See Database schema.
2. Provide a deliverer bean. This example uses HTTP. ServiceUrlResolver maps a logical service name to a base URL, so deployment topology stays out of your publishing code:
@Bean fun httpDeliverer(): HttpMessageDeliverer = HttpMessageDeliverer(ServiceUrlResolver { serviceName -> when (serviceName) { "notification-service" -> "https://notifications.example.com" else -> error("Unknown service: $serviceName") } })
3. Publish inside your transaction. Inject SpringOutboxPublisher and call it right after your business write:
@Service class OrderService( private val orderRepository: OrderRepository, private val outboxPublisher: SpringOutboxPublisher, ) { @Transactional fun placeOrder(order: Order) { orderRepository.save(order) outboxPublisher.publish( OutboxMessage("order.created", order.toJson()), httpDeliveryInfo { serviceName = "notification-service" endpointPath = "/webhooks/orders" }, ) } }
The order row and the outbox row now commit together or not at all. Autoconfiguration takes care of scheduling, retries, delivery and cleanup.
SpringOutboxPublisherthrowsIllegalStateExceptionif you callpublish()outside an active read-write transaction. That is deliberate — an outbox write that can't commit atomically with your business data defeats the purpose of the pattern.
Runnable, self-contained applications live in okapi-examples. Each one is an independent Gradle project that consumes okapi as a published dependency, with its own docker-compose.yml for the databases and brokers it needs.
publish() writes a PENDING row to okapi_outbox in your transaction.FOR UPDATE SKIP LOCKED so workers do not process the same row concurrently.DELIVERED; a retriable failure leaves it PENDING while retry attempts remain; an exhausted retry budget or permanent failure marks it FAILED.x-outbox-id header to deduplicate on — see Deduplicating on x-outbox-id.next_attempt_at, then id) within each delivery type: a new row is due when it is created, a retried one once its backoff has passed, so a failing row does not hold up newer ones. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.OutboxStore used for processing must implement RouteAwareOutboxStore; otherwise processing fails before an unsafe claim. The older claimPending(limit) method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself.FAILED by older workers.READ COMMITTED. The autoconfigured Spring TransactionRunner (with MysqlOutboxStore) and ExposedTransactionRunner (on a MySQL/MariaDB dialect) both use it. At MySQL's default REPEATABLE READ, the claim's gap locks would block every concurrent publish() until delivery finishes. If you supply your own TransactionRunner, an OutboxStore of your own, or a TransactionTemplate with an explicit isolation level, keep it at READ COMMITTED on MySQL. With a JtaTransactionManager, or a JpaTransactionManager without the Hibernate/EclipseLink JpaDialect, okapi cannot set it (startup logs a WARN): configure transaction_isolation = 'READ-COMMITTED' on the server or in the JDBC URL instead. READ COMMITTED requires binlog_format=ROW (the default since MySQL 5.7.7) or MIXED; with STATEMENT, InnoDB rejects the scheduler's writes with ERROR 1665.okapi.processor.max-retries (default 5) counts retries after the first attempt — six attempts in total before a row becomes FAILED. FAILED is terminal. A retriable failure is retried after an exponential backoff with jitter: okapi.processor.initial-backoff (default 10s) before the first retry, backoff-multiplier (default 2) times longer for each further one, capped at max-backoff (default 10m) and shortened at random by up to backoff-jitter (default 0.2). With the defaults, a message survives roughly five minutes of downstream outage before it becomes FAILED; raise max-retries to cover longer ones (e.g. 10 retries span about 50 minutes). The backoff only applies to rows claimed by 2.0 workers.x-outbox-idEvery delivery carries an x-outbox-id header holding the outbox entry's UUID — the same value publish() returns. It is set by all transports (the name is the constant OutboxHeaders.OUTBOX_ID in okapi-core, so consumers can reference it without depending on a transport module), and the value is stable across retries: if okapi delivers the same entry twice, both copies carry the same id. A consumer that records ids it has already processed can therefore drop the repeat.
// Kafka consumer val outboxId = record.headers().lastHeader(OutboxHeaders.OUTBOX_ID)?.let { String(it.value()) } if (outboxId != null && !seenIds.add(outboxId)) return // already processed, skip
// HTTP receiver (Spring MVC) @PostMapping("/webhook") fun receive(@RequestHeader("x-outbox-id") outboxId: String, @RequestBody payload: String) { if (!seenIds.add(outboxId)) return // already processed, skip // ... }
Notice that:
x-outbox-id you set yourself in DeliveryInfo. Over HTTP your value is replaced outright. Kafka headers are multi-valued, so your value is still present in the record, but okapi's is the one appended last — which is why consumers must read it with lastHeader(...) (or take the last of headers(...)) rather than the first match. Pick a different header name if you need to pass an identifier of your own.publish() level. The id identifies an outbox entry, not a business event. Calling publish() twice for the same event creates two entries with two different ids, and a consumer deduplicating on x-outbox-id will process both. Guarding against that is the publisher's job — deduplicate on a business key in the payload, or make the publish itself idempotent.In a typical single-DataSource application, all properties are optional. Multi-DataSource setups may require explicit qualifiers, as described below.
| Property | Default | Description |
|---|---|---|
okapi.processor.enabled |
true |
Set false to disable delivery entirely (e.g. on instances that only publish). |
okapi.processor.interval |
1s |
Delay between scheduler ticks, i.e. how often an idle outbox is polled. A tick with a backlog keeps going (see max-batches-per-tick), so this does not cap throughput. |
okapi.processor.batch-size |
10 |
Maximum entries claimed per batch; each batch runs in its own transaction. |
okapi.processor.max-batches-per-tick |
1000 |
Batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. Batches of a tick claim only entries due when the tick started, so a retry waits for the next tick. 1 restores the 1.x one batch per tick, which caps throughput at about concurrency × batch-size / interval. |
okapi.processor.max-retries |
5 |
Retries after the initial attempt before an entry becomes FAILED. |
okapi.processor.initial-backoff |
10s |
Delay before the first retry of a retriable failure. |
okapi.processor.backoff-multiplier |
2.0 |
Factor applied to the delay for each further retry (>= 1). |
okapi.processor.max-backoff |
10m |
Upper bound on the delay between retries. |
okapi.processor.backoff-jitter |
0.2 |
Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. |
okapi.processor.concurrency |
1 |
Parallel workers per tick, each draining its own batches. Tune based on database capacity and delivery latency; see Performance. |
okapi.processor.transport-dispatch |
parallel |
How a batch spanning several MessageDeliverer beans is dispatched: parallel (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~max(Tᵢ)) or sequential (one group after another, all on the calling thread). Use sequential only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via DataSourceUtils, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. |
Delivered entries are deleted on a schedule so they do not accumulate. FAILED entries are never purged and must be managed separately.
| Property | Default | Description |
|---|---|---|
okapi.purger.enabled |
true |
Set false to manage retention yourself (partitioning, external cron). |
okapi.purger.retention |
7d |
How long delivered entries are kept. |
okapi.purger.interval |
1h |
How often the purger runs. |
okapi.purger.batch-size |
100 |
Rows deleted per batch; each batch is its own transaction. |
| Property | Default | Description |
|---|---|---|
okapi.liquibase.enabled |
true |
Set false if your application manages the outbox schema itself. |
okapi.liquibase.changelog-table |
okapi_databasechangelog |
Liquibase tracking table for okapi's migrations. |
okapi.liquibase.changelog-lock-table |
okapi_databasechangeloglock |
Liquibase lock table for okapi's migrations. |
| Property | Default | Description |
|---|---|---|
okapi.datasource-qualifier |
unset | Bean name of the outbox DataSource. When unset, the single or @Primary DataSource is used. |
okapi.transaction-manager-qualifier |
unset | Bean name of the outbox PlatformTransactionManager. When unset, it is resolved automatically. Set explicitly in multi-PTM setups — see Transactions. |
| Property | Default | Description |
|---|---|---|
okapi.metrics.enabled |
true |
Set false to disable okapi's Micrometer autoconfiguration. Logs a startup warning when disabled. |
okapi.metrics.refresh-interval |
15s |
How often gauges poll the store. Each refresh runs two queries, wrapped in a single read-only transaction when a transaction manager is available. |
Replace okapi-postgres with okapi-mysql; the okapi publishing API stays the same. Add rewriteBatchedStatements=true to your JDBC URL (jdbc:mysql://host:3306/db?rewriteBatchedStatements=true) so Connector/J can send a JDBC batch as a single multi-statement request. okapi cannot set this for you, since it doesn't own your DataSource.
Provide a KafkaMessageDeliverer bean with your own producer, and publish with the matching builder:
@Bean fun kafkaDeliverer(producer: KafkaProducer<String, String>): KafkaMessageDeliverer = KafkaMessageDeliverer(producer)
outboxPublisher.publish(
OutboxMessage("order.created", order.toJson()),
kafkaDeliveryInfo { topic = "order-events" },
)You can register HTTP and Kafka deliverers together; okapi routes each entry by delivery type. You provide and configure the Kafka producer.
okapi-core has no framework dependencies, so any JVM service can use it, from a Ktor app to a background worker. You assemble the pieces yourself — create the store, the deliverer, and the processor, start an OutboxScheduler, and hand it a TransactionRunner (a one-method interface wrapping a block in whatever transaction mechanism you already use). Copy the database-specific SQL linked in Database schema into your application's migrations.
okapi-exposed bridges okapi's transaction and connection abstractions to Exposed: ExposedTransactionRunner, ExposedTransactionContextValidator, and ExposedConnectionProvider. Useful for Ktor and standalone Kotlin services.
okapi ships Liquibase changelogs that create its table and indexes:
classpath:com/softwaremill/okapi/db/postgres/changelog.xml (from okapi-postgres)classpath:com/softwaremill/okapi/db/mysql/changelog.xml (from okapi-mysql)With okapi-spring-boot and Liquibase on the classpath, these run automatically against the configured DataSource at startup, tracked in dedicated Liquibase tables by default to avoid conflicts with the application's migration history.
If you use another migration tool, apply every SQL migration for your database, in order:
TIMESTAMPTZ timestamp columns, 004 retry backoff column and claim index (its second changeset must run outside a transaction)DATETIME(6) timestamp columns, 004 retry backoff column and claim indexFor an existing installation, apply every new migration (up to 004) before starting updated workers. MySQL claims explicitly select the claim index from 004 and fail if it is missing. PostgreSQL's 002 and the index changeset of 004 use CREATE INDEX CONCURRENTLY, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. CREATE INDEX CONCURRENTLY waits for every transaction in the database that started before it, not only those on okapi_outbox, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.
PostgreSQL's 003 converts the timestamp columns from TIMESTAMP (the okapi JVM's local wall-clock time) to TIMESTAMPTZ. It reads existing values in the session TimeZone; pgjdbc sets that to the JVM default zone, so the bundled Liquibase changelog converts them correctly when it runs inside the application. When applying it by hand, run it inside a transaction (psql -1) after SET TIME ZONE '<zone your okapi JVMs run in>' in the same session. It takes an ACCESS EXCLUSIVE lock: it first waits (up to 30 s, then fails and retries on the next start) for open transactions on okapi_outbox, queuing every other statement on the table meanwhile, and then, unless the session zone is UTC (PostgreSQL 12+), rewrites the table while reads and writes wait. 1.x instances keep working after it, apart from one failed poll per pooled connection. See UPGRADING.md for when the in-app conversion uses the wrong zone. See UPGRADING.md.
MySQL's 003 converts the timestamp columns from TIMESTAMP(6), which ends in January 2038, to DATETIME(6) holding UTC. Stop every 1.x instance, publishers included, before it runs: 1.x writes these columns in the JVM zone. It copies the table; it first waits (up to 30 s, then fails and retries on the next start) for open transactions on okapi_outbox to finish, queuing every other statement on the table meanwhile, and then blocks writes, including publish(), until the copy finishes. See UPGRADING.md, including when rows written by 1.x can read back shifted.
okapi stores messages in the fixed okapi_outbox table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:
| Table | Purpose |
|---|---|
okapi_outbox |
Outbox entries. Name is fixed. |
okapi_databasechangelog |
Liquibase history for okapi's migrations (configurable; Liquibase integration only). |
okapi_databasechangeloglock |
Liquibase lock for okapi's migrations (configurable; Liquibase integration only). |
Each processor worker runs its claim, delivery and state update inside one transaction per batch, which keeps FOR UPDATE SKIP LOCKED active until delivery state is saved. While a backlog remains, a worker runs batch after batch within one tick, each in a transaction of its own. Each purge batch also runs in its own transaction. With a single DataSource and PlatformTransactionManager, no additional transaction configuration is needed.
Multiple data sources or transaction managers? Set both okapi.datasource-qualifier and okapi.transaction-manager-qualifier to the beans used for the outbox. okapi fails fast when it detects a mismatch. If the selected transaction manager does not expose its DataSource, okapi cannot verify the pairing and logs a warning instead.
Wiring schedulers by hand? TransactionRunner is a required constructor parameter, with no default:
OutboxScheduler( outboxProcessor = processor, transactionRunner = transactionRunner, config = OutboxSchedulerConfig(...), )
For Spring Boot metrics with Prometheus, add:
implementation("com.softwaremill.okapi:okapi-micrometer") implementation("org.springframework.boot:spring-boot-starter-actuator") runtimeOnly("io.micrometer:micrometer-registry-prometheus")
Expose the Prometheus endpoint:
management: endpoints: web: exposure: include: health,prometheus
Metrics are then available at /actuator/prometheus.
| Metric | Type | Description |
|---|---|---|
okapi.entries.delivered |
Counter | Successfully delivered entries |
okapi.entries.retry.scheduled |
Counter | Failed attempts rescheduled for retry |
okapi.entries.failed |
Counter | Permanently failed entries |
okapi.batch.duration |
Timer | Processing time per batch; under the scheduler from the claim until the transaction has committed (not including listener callbacks) |
okapi.entries.count |
Gauge | Current entry count (tag: status) |
okapi.entries.lag.seconds |
Gauge | Age of the oldest entry (tag: status) |
Aggregating across instances. Counters and timers are per-instance, so sum them:
sum by (job) (rate(okapi_entries_delivered_total[5m]))
Gauges represent shared database state and are emitted by every instance. Do not sum them across instances; aggregate by Prometheus job and status:
max by (job, status) (okapi_entries_count)
Without Spring Boot, construct MicrometerOutboxListener and MicrometerOutboxMetrics with your MeterRegistry. Refresh gauges with OutboxMetricsRefresher or your own scheduler.
For custom reactions to delivery events, implement OutboxProcessorListener. OutboxProcessor takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. Counters therefore count committed status changes, not delivery attempts: a send whose transaction then rolled back (a failed commit, or a block Exposed retried) is not counted, though the message may have reached the consumer. If you drive OutboxProcessor yourself, call processBatch inside your transaction and publishEvents after it commits; processNext does both at once, so inside a transaction its events precede the commit.
The runtime modules build on okapi-core; okapi-bom only aligns their versions. Pick a storage module, one or more transports, and a framework adapter if you want one.
From Reliable Message Delivery: the Transactional Outbox Pattern With Okapi.
| Module | Purpose |
|---|---|
okapi-core |
Abstractions, processing loop, scheduling, retry policy. No framework dependencies. |
okapi-postgres |
PostgreSQL storage via plain JDBC (FOR UPDATE SKIP LOCKED) |
okapi-mysql |
MySQL 8+ storage via plain JDBC |
okapi-http |
HTTP webhook delivery (JDK HttpClient) |
okapi-kafka |
Kafka topic publishing |
okapi-spring-boot |
Spring Boot autoconfiguration — selects a store module and wires registered deliverers and metrics |
okapi-exposed |
Exposed ORM integration for transactions and connections |
okapi-micrometer |
Micrometer counters, timers and gauges |
okapi-bom |
Version alignment for all of the above |
| Dependency | Supported | Notes |
|---|---|---|
| Java | 21+ | Required |
| Spring Boot | 3.5.x, 4.0.x | okapi-spring-boot |
| Kafka Clients | 3.9.x, 4.x | Included transitively by okapi-kafka; you can override the version in your build. |
| Exposed | 1.x | okapi-exposed |
okapi-spring-boot does not bring Spring Boot transitively; your application controls the Spring Boot version.
The storage modules use plain JDBC. With Spring Boot, they participate in the transaction selected through a PlatformTransactionManager; okapi-exposed provides adapters for Exposed-managed transactions.
Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See benchmarks/ for methodology and measured results.
Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and interval only sets how quickly a new message is picked up when the outbox was idle. A failing or misconfigured downstream gets one batch per worker per tick, as before, while other delivery types keep draining. To limit the rate at which okapi sends to a healthy downstream system, set max-batches-per-tick (with batch-size and interval) rather than relying on the poll interval alone.
./gradlew build # Build and test all modules (Docker required — Testcontainers) ./gradlew ktlintFormat # Format code — mandatory before committing ./gradlew :okapi-benchmarks:jmh # Run JMH benchmarks (~30 min, see benchmarks/README.md)
All suggestions are welcome. Take a look at the open issues and pick one, or report your own.
If you are unsure why or how something works, ask on Discourse or open an issue. That usually means the documentation or the code is unclear, and fixing it helps everyone.
When your PR is ready, see our guide to preparing a good PR.
okapi is built and maintained by SoftwareMill. We offer commercial development services — get in touch to learn more.
Copyright (C) 2026 SoftwareMill. Licensed under the Apache License 2.0.
softwaremill/okapi is a public repo on GitHub by softwaremill. Reliable message delivery using the transactional outbox pattern for Kotlin.
It declares the Apache-2.0 license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-11-18. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Java 面试 & 后端通用面试指南,覆盖计算机基础、数据库、分布式、高并发、系统设计与 AI 应用开发
Distributed transaction solution in micro-service base on eventually consistency, also an eventbus with Outbox pattern
The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.



Zero-cost, compile-time, type-safe dependency injection library.

Easy, fast, transparent generic derivation of typeclass instances
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/softwaremill/okapiTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.