NoSQL embedded document store for Java
Repo nitrite/nitrite-java, read from GitHub 1 min ago. nitrite has not joined Take My Repo.
NitriteJsonImporter Base64-decoded the keys, documents and index descriptors of an exchange JSON file and read them through a plain ObjectInputStream. An attacker-supplied file could make Importer.importFrom(...) run the readObject() of any Serializable gadget class on the classpath. The filters added for GHSA-9297-g93h-86gg and GHSA-7w7v-2j76-mqwp covered only the MVStore adapter.ObjectInputStream), now read through ObjectUtils.createFilteredInputStream, which resolves only Nitrite's own, JDK and kotlin.collections types. It checks classes in resolveClass, so it works on Android too.Serializable classes now fail to import unless those classes are allowed with the new ImportOptions.setAllowedClasses(...), e.g. setAllowedClasses("com.example.model.**").KryoObjectFormatter ran Kryo with setRegistrationRequired(false), so a crafted database file could name any class on the classpath and Kryo would load it, instantiate it and set its fields, a known route to gadget chains.registerSerializer(...) are read as before. A class stored by name must now be a Nitrite, JDK or kotlin.collections type, or match the patterns given to the new KryoObjectFormatter(String allowedClasses), e.g. new KryoObjectFormatter("com.example.model.**"), passed to RocksDBModuleBuilder.objectFormatter(...). Anything else fails with a NitriteSecurityException naming the class, before the class is loaded.KryoObjectFormatter in the JVM; serializers registered on one formatter no longer leak into others. The on-disk format is unchanged.Spatial implementation, so queries, updates and removals on a reopened spatial index failed with a ClassCastException from DefaultSpatial to MVSpatialKey. Cursors and bounds comparisons now accept any Spatial. The on-disk format and public method signatures are unchanged.Document.createDocument(Map) keeps nested objects as plain Maps (as deserialized JSON does), but embedded-field lookup only descended into Document values. So get("address.city") returned null and where("address.city").eq(...) never matched. Lookup now descends into any Map value.org.dizitart.no2.**;java.**, so a document written by a Kotlin app holding emptyList(), emptySet() or emptyMap() - stored as kotlin.collections.EmptyList, EmptySet and EmptyMap - could no longer be read. kotlin.collections.** is now allowed by default, in the store filter and in the v1 migration filter.NitriteMVStore.openMVMap used to roll the store back one version per failed attempt, discarding committed data and ending in Chunk N not found, which hid the real cause. It now fails at once with a NitriteSecurityException that names the rejected class and points to MVStoreModuleBuilder.allowedClasses(...), and leaves the file untouched.AssertionError: 0 != 1 (Windows CI, TransactionTest.testRollbackDropCollection)
deregisterVersionUsage() gets the store lock with tryLock(). When the background writer held the lock, the drop was skipped, oldestVersionToKeep stayed behind, and H2's close-time assertion oldestVersionToKeep == currentVersion failed. The 5.4.0 fix released every outstanding version but could still lose that race.NitriteMVStore.close() now stops the background writer before it releases the outstanding versions, then forces one uncontended drop.Eleven changes from @brettwooldridge, most of them found on a production system, and one H2 workaround found while testing them. Four are data-integrity fixes, three of which can end with a store that will not reopen or a query that quietly returns the wrong rows.
ObjectDataType, which reads stored documents, index entries and metadata back through a plain ObjectInputStream. A database file from an untrusted source - an uploaded backup, an imported or synced .db - could make an ordinary find() instantiate any Serializable class on the classpath and run its readObject(), so a gadget class there meant remote code execution. The allowlist added for GHSA-9297-g93h-86gg covered only the legacy v1 migration path.ObjectDataType that deserializes through a JEP 290 allowlist: Nitrite's own types and JDK types (org.dizitart.no2.**;java.**). A process-wide jdk.serialFilter still applies on top. Top-level object arrays, whose elements H2 reads with its own unfiltered type, are refused; Nitrite never stores one. The on-disk format is unchanged.Serializable classes now fail to read unless those classes are allowed with the new MVStoreModuleBuilder.allowedClasses(...), which takes JEP 290 patterns, e.g. allowedClasses("com.example.model.**").MVStore's chunk retention and versions-to-keep are left at H2's defaults (#1301, #1303)
MVStoreUtils.openOrCreate forced setRetentionTime(0) and setVersionsToKeep(0) on every store since 2020. With both at 0, H2 may reuse a chunk's blocks while the chunk map it writes at close still lists that chunk, and the file then refuses to open at all - read-only or not - with MVStoreException: Double mark: 394/5 ... at FreeSpaceBitSet.markUsed. Only H2's recovery mode gets past it. (h2database/h2database#2752, #4083, both open.)MVStoreModuleBuilder.retentionTime(ms) and versionsToKeep(n). Both are null by default, which leaves H2's values in place. Passing 0 restores the old behaviour, at the cost described above.close() still compacts synchronously.A read no longer hands out anything the store holds (#1294)
tryCommit. Whatever a read hands back must therefore share no mutable state with it, or a caller's in-place edit is written straight into the store, bypasses the indexes, and can race the serialization into a ConcurrentModificationException and a store panic.Document.clone() copied the top-level map and embedded documents only, so a List, Set, Map, array, byte[], Date or Calendar inside the copy was still the instance in the store and found.get("tags").add(x) reached the page. And NitriteCollection.getById() returned the stored instance itself.clone() is now a deep copy: containers and arrays are copied recursively, preserving the concrete collection class where it has a public no-arg constructor and the comparator of sorted sets and maps; Dates and Calendars are cloned; immutable values are shared, and so are values of a type the copy does not know, which the javadoc now states. getById() hands out a clone as find() does, and returns null for a missing id instead of passing null through the processor chain.The store catalog copies its stored name set instead of mutating it in place (#1296)
StoreCatalog.writeCollectionEntry wrapped the catalog document in MapMetaData, which took the Set straight out of it and added the new name to that instance. On MVStore that instance is the one held in the catalog page. Creating collections in a burst interleaved with writes - which is what a data migration does - put the serializer's HashSet.writeObject and the catalog's HashSet.add on the same set at the same time, ending in MVStoreException: Could not serialize {mapNames=[...]} and MVStore.panic, after which the store is closed and every later operation throws.MapMetaData now copies, so a write always puts a new set and the instance a serializer may be reading is never touched - the rule WriteOperations.update already followed by cloning a document before merging into it.An index scan skips documents removed between the lookup and the fetch (#1302)
(id, null) row. With a residual filter that row reached Filter.apply and threw NullPointerException from the filter's document.get(); without one the cursor handed the caller a null element. The by-id fast path had the same window between containsKey and get.IndexedStream now prefetches and drops ids whose document is gone, FilteredStream treats a row without a document as a non-match, and the by-id path does one get. skip() still walks ids without fetching them, so under a concurrent remove a page boundary can still count an id whose document is gone - which is the same indeterminacy the removal itself introduces.clear() and dropIndex() reach every layout map an index occupies (#1295)
IndexManager.close(), clearAll() and dropIndexDescriptor() acted only on the map name recorded in IndexMeta, which is the classic one. That already missed the composite map of a non-unique index: after collection.clear() its rows survived, and a query on that index returned the ids of the cleared documents alongside the new ones - two live documents, four results. It also broke ChangeIdField, whose createIndex found the previous index map still populated and rebuilt over it.The first concurrent read of an index after a restart no longer fails dropping its legacy map (#1309, #1315)
ComparableIndexer created those instances with an unsynchronized check-then-act, so threads arriving together could each get an instance of their own and each run the migration. On MVStore the second drop asked MVMap.getName() for a map that was already gone, got null, and failed with NullPointerException in NitriteMVStore.removeMap; the same race also threw from Attributes.set through NitriteMap.updateLastModifiedTime.computeIfAbsent, so there is one index instance and one map wrapper per name. NitriteMVMap keeps the name it was opened with and acts only when its compare-and-set wins, so drop() and close() run once; removeMap ignores a null name and no longer creates an empty map only to remove it.Issue1315Test.Concurrent first reads of a map on an in-memory MVStore no longer fail inside H2 (#1311)
ObjectDataType picks the delegate that compares serialized keys on first use, through an unsynchronized field, and SerializedObjectType.compare checks delegates by identity. Threads making a map's first key comparison together could each install their own and fail with UnsupportedOperationException: Can not compare. The code is the same in h2 2.4.240 and 2.5.250.NitriteMVStore now settles each ObjectDataType key type once when it opens the map, on the one thread running computeIfAbsent. Nothing is serialized. It can go once H2 fixes the type upstream.A unique index no longer rejects a document over a key that document already holds (#1295)
addNitriteIds treated any existing id under the key as a violation, so it counted the writer's own id against it. Another document under the key is a violation; the same document again is not - which is what a unique index over an array field with a repeated element does, and what an index rebuild or a replayed write does.One collection's long write no longer stalls getCollection for every other collection (#1292)
CollectionFactory.getCollection held one lock for the whole factory and, while holding it, called isDropped() and isOpen() on the registered collection. Both take that collection's read lock. While a long write held a collection's write lock - an index rebuild, a large remove(filter), an update on a big document - the one caller asking for that collection blocked inside the factory lock, and from then on every getCollection call for every other collection queued behind it.update() for over three hours, and 349 other threads parked in CollectionFactory.getCollection, most of them wanting unrelated collections.MVStoreModuleBuilder.pageSplitSize defaults to 16 KB, not 16 bytes (#1293)
MVStore.Builder.pageSplitSize, which takes bytes. Its value was 16, the same literal used for cacheSize (megabytes) and cacheConcurrency (a count), so every leaf page split as soon as it held more than one entry.16; 36,096 pages at depth 12 with MVStore's own persistent-store default of 16 KB; 13,327 pages at depth 6 with 64 KB, the largest value that survives H2's (cacheSize / cacheConcurrency) >> 4 clamp.Closing an MVStore no longer trips H2's version assertion over an iterator opened on a dropped map
close() or drop() releases the versions of its iterators. An iterator opened through a wrapper that was already closed or dropped - which a transaction rolling back a dropped collection, and a migration renaming one, both do - was out of reach of every map the store still knew, so only the garbage collector released it. A store closed before that happened failed under -ea with AssertionError: 0 != 1 from MVStore.closeStore, and without assertions closed with an older version still pinned.DroppedMapIteratorOnCloseTest.An update that leaves an indexed value unchanged no longer rewrites the index (#1297)
DocumentIndexWriter.updateIndexEntry treated an index as affected whenever the update document carried the indexed field, and then removed and rewrote the entry. An update that writes the whole document back - the common upsert shape - carries every indexed field with its old value, so every index was rewritten on every update for nothing. On the pre-4.4.0 list layout that was a copy of the whole per-key id list twice per index per write.A unique index stores one id per key instead of a one-element list (#1295)
value -> [id] layout: a CopyOnWriteArrayList per key that never held more than one element, allocated and copied on every write, plus a size check standing in for the uniqueness test. It now stores the id itself (value -> id) in a map of its own, named with a |unique suffix, and enforces uniqueness by comparing the stored id with the writer's.IndexMap exposes the single-id map to the scanner and the filters as one-element lists, so the read path is unchanged. The map has its own name because the RocksDB adapter decodes values by the declared type of the map they live in.Equality and range index scans stream instead of materializing every id (#1298)
NitriteIndexer.findByFilter returns a LinkedHashSet of every matching id, so find(k = v).firstOrNull() built the whole match set before handing back one row, and a bounded page paid for the entire result. On a non-unique index over a low-cardinality field that set is a large fraction of the collection on every lookup.NitriteIndex.findNitriteIdStream and NitriteIndexer.findByFilterStream return null, so every other index type, plugin indexer and plan shape keeps the materialized path unchanged. The covered-count shortcut that lets size() answer without fetching documents is kept by counting the streamed ids on demand.Same-type numbers compare without going through BigDecimal (#1299)
Numbers.compare converted both operands to BigDecimal on every call, two allocations per comparison, and every index key comparison lands there. On a production store with numeric index keys the conversion was the hottest frame in a multi-hour index rebuild.longs; two doubles or two floats compare as themselves once NaN and the infinities have taken the existing special-case path, with -0.0 and 0.0 equal as BigDecimal treats them; two BigDecimals or two BigIntegers use their own compareTo. Every mixed pairing keeps the exact BigDecimal conversion, so cross-type equality such as Integer 1 against Float 1.0f is unchanged.Updating a document leaves its old page obsolete, so a store written to far more often than it grows accumulates chunks that are mostly dead. autoCompactFillRate(0) had disabled reclamation since #41, and the file climbed while the live data stayed small — a reported ~800MB around 100 live documents.
#41 was a cursor reading a chunk that compaction had already reclaimed (Chunk 162 no longer exists), which is why compaction was turned off rather than fixed. Every iterator and cursor now registers an MVStore version for its lifetime, so chunks it is reading cannot be collected underneath it, and compaction is safe to enable again. Abandoned iterators are released by a Cleaner and by close()/drop() on the map.
New MVStoreModuleBuilder.autoCompact, defaulting to true. Set it false for the previous behaviour.
Reclamation on close is guaranteed; reclamation while running is not. close() compacts synchronously and always shrinks the file. MVStore's background compaction runs on a one-second tick and attempts its work under a try-lock, so a writer holding the store lock makes it skip that round: how much a long-running application gets depends on its idle time and on how many cores are available to schedule the thread on. An application that holds the database open for months should not rely on the background half alone — see the File Growth and Compaction docs.
Measured over 25 live documents and 100k updates: the file held at ~213KB against 299KB and climbing, and close() left 57KB against 299KB.
DBValue folded every number to a double before using it as an index key. That is what makes Integer(5) and Double(5.0) the same key (#178), which matters on stores that compare the encoded key rather than calling compareTo — the RocksDB adapter is one.
A double stops stepping by one above 2^53. Around 8.7e17 the nearest doubles are 128 apart, so ids closer than that folded onto one key: a unique index rejected an id it had never seen, and a non-unique lookup returned rows belonging to a different id. Snowflake ids, TSIDs and ULID-style identifiers all live there.
The fold now applies only where the value survives it. Cross-type equality is unchanged for every value a double holds, which is the range #178 is about.
Important
Behaviour change. A Long and a BigInteger holding the same value above 2^53 are no longer the same key on a byte-comparing store. They only matched before because both had been rounded onto the same double — the same rounding that would equally have matched a different id. An index built by an earlier version over such values holds the folded keys, those entries were already colliding, and it should be rebuilt with rebuildIndex().
Paging coverage for the plan shapes a source-level skip has to decline — an or plan, a collection with removals, and an indexed filter ordered by that same index. From #1283.
Thanks to @DarkAtra and @mfleisch, whose reports and patches this release is built on.
Full changelog: https://github.com/nitrite/nitrite-java/blob/main/CHANGELOG.md
nitrite-bridge — inspect a running Nitrite database from a desktop client. The Nitrite adapter for the dbinspect wire protocol: collections and handed-in repositories as browsable stores, schema inferred by sampling documents and always flagged as inferred, paging over FindOptions skip/limit/orderBy, the JSON filter DSL, and watch over Nitrite's collection subscription. The engine-neutral core it plugs into is org.dizitart:dbinspect-bridge, which has no database in its dependency tree at all.
allowWrite, and whole-store snapshot behind allowSnapshot — both false unless the embedding application asks, and absent from the reported capabilities while they are. A row is addressed by _id; an update is partial; changes: 0 means the row was not there. _id inside an update's values is refused, because Nitrite merges an update document and it would rewrite the identity of the row it just matched.regex is off unless allowRegex is set, and length-capped with a nested-quantifier check when it is on. java.util.regex backtracks and a match cannot be interrupted, so the default is the mitigation that matters.Nitrite.getCollection creates a collection that does not exist, so an unchecked name would let a paired client make one; writes go through the same allow-list.<modules>: dbinspect-bridge is not on Maven Central until the dbinspect release, so a build of a clean checkout would fail on it. Build it on its own, with dbinspect-bridge installed locally.The bridge can open a transaction (docs/PROTOCOL.md §3.1). A transaction is a second adapter over the same database rather than a mode on the first, so one connection's uncommitted documents can never reach another connection's reads. Nitrite's transaction lives above the storage engine — a TransactionStore buffers the writes and a journal replays them on commit — so this works identically on MVStore, RocksDB and in memory.
capabilities.transactions follows allowWrite: that flag is the permission, this reports what the engine can undo. The transactional twin reports it false, because Nitrite does not nest one.listStores counts through the transaction, since a total that left out the rows the person just staged is not read-your-own-writes.find(filter, skipBy(n).limit(m)) now advances the storage iterator without decoding what it passes over.
BoundedStream skipped by calling next() on the iterator beneath it, and that iterator is below the document layer: it deserialised every skipped document in order to throw it away. Page latency was therefore linear in the page number - about 9 µs per skipped row on both persistent stores - so a 50 000-row collection paged at 200 rows took 3 ms for the first page and 350 ms for the two-hundredth. It is an ordinary browse, not an edge case.SkippableIterator, implemented by the storage-level iterators that can do better and by nothing else. BoundedStream uses it when it is there and falls back to the loop when it is not, so a stream that has to look at a document to know whether to skip it - a collection-scan filter, a blocking sort - is unaffected and still correct.MVMap.getKey(n) is a O(log n) descent rather than a scan; the skip seeks with it and continues from a cursor. Measured over 50 000 rows, 200 to a page: p95 583.9 ms → 5.8 ms, and the first and last pages now cost the same 0.1 ms.next() would have paid for each row.markDirty() flipped isDirty on the IndexMeta that get() returned and never put it back, so nothing told the store the entry had changed and the new value was not guaranteed to be written. Both beginIndexing() and endIndexing() go through it, so the marker was unreliable in both directions: an index left half-built by a crash could come back reading as clean, and a completed one could stay marked dirty and be rebuilt on the first write after every open.A sorted, limited find no longer fetches the whole collection when the sort field is indexed. find(ALL, orderBy("createdAt", Descending).limit(20)) asked for 20 rows and cost what draining every stored document costs. SortedDocumentStream collects the entire result set before BoundedStream gets to drop 99% of it — and the cost is the decode, not the comparison, so it scales with document size as well as count. An index on the sort field bought nothing: the index was only ever used to filter, never to order, and page 50 cost exactly what page 1 cost because the work finished before the skip applied.
When the query has no filter, one sort field, a limit, and a simple unique or non-unique index on exactly that field, the sort keys are now read from that index — which already stores them — and only the documents actually returned are fetched. Measured on MVStore over 2000 rows each carrying a 150-element list, a sorted page went from ~115 ms to ~2 ms, and stopped growing with the collection.
The index is used only when it holds exactly one entry per stored document. A multi-valued field is indexed once per element, which is detected by a duplicate-id check and an entry-count check, and falls back to the blocking sort. Ordering — including where nulls sort and how ties break — is identical either way: the same comparator runs over keys taken from the index instead of from the documents.
Where documents are small and cheap to read, the index walk replaces a decode that was nearly free, so a sorted page over lean rows can cost a few hundred microseconds more than before. The trade is deliberate: the loss is bounded and sub-millisecond, the win grows without bound with document size.
FindPlan.getSortIndexDescriptor()NitriteIndex.readSortKeys(long) and NitriteIndexer.readSortKeys(IndexDescriptor, NitriteConfig, long) — both default-implemented to return null, so existing indexer plugins are unaffectedDocumentSorter.compareValues(Object, Object, Collator)Full Changelog: v5.0.0...v5.1.0