Protobag: A bag o' Serialized Protobuf Messages
With built-in support for time-series data
Quickstart & Demo
See this python noteboook
for a demo of key features.
Or you can drop into a Protobag development shell using a clone of this repo
and Docker; FMI see:
./pb-dev --help
Summary
Protobuf is a popular data
serialization library for C++, Python, and several other languages. In
Protobuf, you can easily define a message type, create a message instance,
and then serialize that message to a string or file.
But what if you want to store multiple messages in one "blob"? You could
simply use repeated and create one giant message, but perhaps you don't in
general have the RAM for this approach. Well then, you could append multiple
messages into one big file, and delimit the boundaries of each message using
the number of bytes in the message itself. Then you'd have something that
looks exactly like the infamous
TFRecords
format, which is somewhat performant for whole-file streaming reads, and has
a very long list of downsides. For example, you can't even seek-to-message
in a TFRecords file, and you either need a large depenency (tensorflow) or
some very tricky custom code to even just do one pass over the file to count
the number of messages in it. A substantially better solution is to simply
create a tar archive of string-serialized Protobuf messages--
enter Protobag.
A Protobag file is simply an archive (e.g. a Zip or Tar file, or even just a
directory) with files that are string-serialized Protobuf messages. You can
create a protobag, throw away the Protobag library itself, and still
have usable data. But maybe you'll want to keep the Protobag library around
for the suite of tools it offers:
Protobag provides the "glue" needed to interface Protobuf with the
fileystem and/or an archive library, and Protobag strives to be fully
cross-platform (in particular supporting deployment to iOS).
Protobag optionally indexes your messages and retains message Descriptors
(employing the Protobuf
"self-describing message" technique)
so that readers of your Protobags need not have your Protobuf message
definitions. One consequence is that, with this index, you can convert
any protobag to a bunch of JSONs.
Protobag includes features for time-series data and offers a
"(topic/channel) - time" interface to data similar to those offered in
ROS and
LCM, respectively.
Batteries Included
Protobag uses libarchive as an archive
back-end to interoperate with zip, tar, and other archive formats. We
chose libarchive because it's highly portable and has minimal dependencies--
just libz for zip and nothing for tar. Protobag also includes vanilla
DirectoryArchive and
MemoryArchive back-ends for
testing and adhoc use.
If you want a simple "zip and unzip" utility, Protobag also includes those as
wrappers over libarchive. See
ArchiveUtil.
Development
Discussion of Key Features
Protobag indexes Protobuf message Descriptors
By default, protobag not only saves those messages but also
indexes Protobuf message descriptors so that your protobag readers don't
need your proto schemas to decode your messages.
Wat?
In order to deserialize a Protobuf message, typically you need
protoc-generated code for that message type (and you need protoc-generated
code for your specific programming language). This protoc-generated code is
engineered for efficiency and provides a clean API for accessing message
attributes. But what if you don't have that protoc-generated code? Or you
don't even have the .proto message definitions to generate such code?
In Protobuf version 3.x, the authors added official support for
the self-describing message paradigm.
Now a user can serialize not just a message but Protobuf Descriptor data that
describes the message schema and enables deserialzing the message
without protoc-generated code-- all you need is the protobuf library itself.
(This is a core feature of other serialization libraries
like Avro).
Note: dynamic message decoding is slower than using protoc-generated code.
Furthermore, the protoc-generated code makes defensive programming a bit
easier. You probably want to use the protoc-generated code for your
messages if you can.
Protobag enables all messages to be self-describing messages
While Protobuf includes tools for using self-describing messages, the feature
isn't simply a toggle in your .proto file, and the API is a bit complicated
(because Google claims they don't use it much internally).
protobag automatically indexes the Protobuf Descriptor data for your messages
at write time. (And you can disable this indexing if so desired). At read
time, protobag automatically uses this indexed Descriptor data if the user
reading your protobag file lacks the needed protoc-generated code to
deserialize a message.
What if a message type evolves? protobag indexes each distinct message type
for each write session. If you change your schema for a message type between
write sessions, protobag will have indexed both schemas and will use the
proper one for dynamic deserialization.
For More Detail
For Python, see:
protobag.build_fds_for_msg() -- This method collects the descriptor data
needed for any Protobuf Message instance or class.
protobag.DynamicMessageFactory::dynamic_decode() -- This method uses
standard Protobuf APIs to deserialize messages given only Protobuf
Descriptor data.
For C++, see:
BagIndexBuilder::DescriptorIndexer::Observe() -- This method collects the
descriptor data needed for any Protobuf Message instance or class.
DynamicMsgFactory -- This utility uses uses standard Protobuf APIs to
deserialize messages given only Protobuf Descriptor data.
Cocoa Pods
You can integrate Protobag into an iOS or OSX application using the CocoaPod ProtobagCocoa.podspec.json
podspec included in this repo. Protobag is explicitly designed to be cross-platform (and utilize only C++
features friendly to iOS) to facilitate such interoperability.
Note: before pushing, be sure to edit the "version" field of the ProtobagCocoa.podspec.json file
to match the version you're pushing.
pod repo push SCCocoaPods ProtobagCocoa.podspec.json --use-libraries --verbose --allow-warnings
C++ Build
Use the existing CMake-based build system.
In c++ subdir:
mkdir build && cd build
cmake ..
make -j
make test
Python Build
The Python library includes a wheel that leverages the above C++ CMake build system.
In python subdir:
python3 setup.py bdist_wheel