Relational Workflows: where database schemas define executable data pipelines.
Repo datajoint/datajoint-python, read from GitHub 10 min ago. datajoint has not joined Take My Repo.
DataJoint 2.3.6 fixes five security issues; upgrade promptly.
This is a security release on the 2.3 line. It contains 2.3.5, the fixes below, and a higher minimum PyMySQL version (see Dependencies). Every earlier release has at least one of these issues. Releases before 2.0, including 0.14, are no longer supported and will not receive a fix. The bug fixes merged since 2.3.5 will come in 2.3.7.
Upgrade with pip install --upgrade "datajoint>=2.3.6". The conda-forge package follows within about an hour of PyPI.
CVE IDs for these advisories are requested and will be added to each advisory when assigned.
alter() part affects 0.12.0 and later.<filepath@> value that points outside its store is now refused. This means a path with a .. part, or, for a file store, an absolute path outside the store's location.
DataJointError.DataJointError.dj.config.allow_filepath_outside_store = True or the environment variable DJ_ALLOW_FILEPATH_OUTSIDE_STORE=true. This turns the check off for every <filepath@> value, so use it only with data you trust.<filepath@>. The error message says so.<blob@>, <npy@>, <attach@> and <object@> values. A path with a .. part or a NUL (zero) byte raises DataJointError. On a file store, DataJoint also checks where the full path resolves, so a path must stay inside location even when location is empty or .. There, a path that starts with file:, local: or ~ is refused if it would resolve outside location; otherwise it stays a name inside the store. Paths that DataJoint creates itself always pass.location is a symbolic link, for example /tmp on macOS, which links to /private/tmp. Then an absolute path through the link's target, such as one returned by Path.resolve(), is refused. Give the path relative to the store, or under location exactly as written.<filepath@> path inside a file store's location is now saved relative to the store. The fetched .path is that relative path, not the string you inserted._hash/ and _schema/ by default) now cleans up the path first and ignores case. Paths such as ./_hash/x or _HASH/x now raise ValueError.. or .., or that contains /, \ or a NUL byte, raises DataJointError on fetch, and nothing is written. Such names, including ones with a backslash, can still be inserted; the error comes on fetch.
: is also refused. For example, an attachment named 12:00.txt that was inserted from Linux or macOS cannot be fetched on Windows. Fetch such attachments on Linux or macOS.name_1.ext, name_2.ext, ...). A regular file with the same content is still reused.<object@> path with a .. part is refused, on insert and on fetch.ObjectRef.download() saves a single file into a folder, it uses only the text after the last / or \ of the stored name. On Linux and macOS, a file stored as we\ird.txt is saved as ird.txt.<object@> folder now checks every entry before anything is written. If any entry would land outside destination/<object name>/, the download raises DataJointError and writes no files. Entries whose names contain a backslash, or a : on Windows, are also refused, as are keys with an empty or . segment (such as a//b or a/./b), which earlier releases downloaded with the extra segment collapsed. Downloaded files keep the same layout as before. With fsspec older than 2024.3.1, downloading an empty folder object now raises FileNotFoundError instead of creating an empty folder.TLS encrypts the connection between DataJoint and the database server.
use_tls=True or a use_tls dict now requires TLS. If the server does not offer TLS, the connection fails before the user name and password are sent. In 2.3.5, use_tls=True could connect without encryption.
use_tls=True encrypts the connection but does not verify the server's certificate. To verify it, pass use_tls={"ca": "/path/to/ca.pem"}.use_tls=False.ca, cert, key and password keys of a use_tls dict.
ca, DataJoint verifies the server's certificate and host name."check_hostname": False. Then only the certificate is verified.use_tls=True still requires TLS without verification. Before, a dict did the same.use_tls=None, has changed on both backends.
use_tls=False.pg_hba.conf does not accept TLS connections from this client. pg_hba.conf is the file that controls which connections the server accepts.use_tls=False now turns TLS off. Before, the setting never reached the driver. So on PostgreSQL, and on MySQL with PyMySQL 1.2 or later, the connection used TLS whenever the server offered it. Now a server that requires encrypted connections refuses the connection, such as MySQL with require_secure_transport=ON. Remove use_tls=False or set use_tls=True.dj.Instance and dj.Connection now read database.use_tls from the configuration, or the DJ_USE_TLS environment variable, when use_tls is not given. dj.conn() already did this.
false now turns TLS off for them.true now makes them refuse a server without TLS.use_tls= to override the configuration.dj.conn(use_tls=True) or dj.conn(use_tls={...}) now reconnects an existing connection with that setting, unless the connection already meets it. Before, the argument was ignored.
True. A dict is met only by a connection made with the same dict.DataJointError.dj.conn(use_tls=False) does not change an existing connection. Use reset=True.dj.conn() without use_tls.A value in a dict, record-array or DataFrame restriction must be one of these:
None;datetime or datetime.time;Decimal or a list;uuid attribute;Other values now raise DataJointError. Before, they were written into the query unquoted, and most of them did not give a correct query. They include:
memoryview;numpy.datetime64;datetime.timedelta (PyMySQL returns it for MySQL time columns);Convert such values first. For example, turn a numpy.datetime64 into a datetime.datetime, and a 0-d array into a number with .item(). If pandas converted a DataFrame's date columns to datetime64, use df.astype(object).
A JSON attribute key can end with a return type, as in attr.path:type. That type must now be a standard type: integer, floating-point, decimal, boolean, character, binary, JSON, UUID, date or time. It may have a length or precision, a time-zone qualifier, a character set and a collation.
DataJointError in restrictions, proj() and index declarations. Examples are char(10) ascii, money and citext.A key that starts with an attribute name but is not a valid JSON path, such as params.bad path, raises DataJointError. Keys that name no attribute are still ignored.
List values, and dict values given for a JSON path, are now written as quoted text and compared with their exact text.
Restriction values now stay inside their quotes whatever the server's string settings. These settings control how the server reads a backslash in quoted text: sql_mode on MySQL and standard_conforming_strings on PostgreSQL. The quoting change itself does not change which rows match under default settings.
''), not with a backslash. This holds under any sql_mode, including NO_BACKSLASH_ESCAPES. Under NO_BACKSLASH_ESCAPES, a value that contains a backslash, a line break or a NUL byte may not match the rows it should.E'...' form. In that form, a backslash is read the same way whatever the server settings.Foreign-key references (-> ...) in table definitions are now read as table names and never run as Python code. This applies when a table is declared and in alter().
Parent, module.Parent and Master.Part. Each may be in parentheses or followed by ()..proj() is supported with string arguments, such as Parent.proj(new_id='parent_id').Other forms now raise a DataJointError that lists the supported forms:
... or **dict passed to .proj();r, u or f prefix;'a' 'b';Rewrite such references in a supported form, for example by writing the names in a projection as string literals.
If the error says a name is not defined, make sure the parent table's class is defined or imported in the module where the table is declared. For alter(), you can instead pass the namespace that defines it with context=.
In describe() output, line breaks become spaces. This applies to table and column names, comments, default values, types and index columns. In a projected reference, a backslash or double quote in a column name is now escaped.
A compressed blob whose data expands beyond the size recorded in its header is now refused before it is fully expanded. This does not limit the memory that a blob can use up to its recorded size.
A corrupt compressed blob raises DataJointError instead of zlib.error. Code that catches zlib.error should catch DataJointError.
These changes apply to Linux and macOS.
The configuration search now skips some files and folders, and logs a warning for each one.
datajoint.json in the working directory and its parent directories..secrets directory, /run/secrets/datajoint and the secret files in them.An item is skipped when both of these are true:
/tmp, or below such a directory that lacks the sticky bit. The sticky bit is the t in drwxrwxrwt on /tmp. It stops users from removing or renaming each other's files.Files in a shared group directory, including one under a sticky /scratch, are still loaded. When a skipped datajoint.json is found, the search stops there, and files in parent directories are not used.
These setups can be affected:
emptyDir written by a helper container that runs as a different user.chmod -R 777./mnt/c), when running as root or as another WSL user.To use a skipped file, do one of these:
DJ_CONFIG_FILE to its path;dj.config.load(path);chown;chmod o-w.Supply skipped secrets through environment variables. The skip is logged as a warning, so DJ_LOG_LEVEL=ERROR hides it.
A warning is now logged when DataJoint uses a configuration file, secret file or secrets directory that every user can write. Restrict it with chmod o-w.
/run/secrets/datajoint with mode 1777.The new environment variable DJ_CONFIG_FILE names the configuration file to load at import, with no search.
.secrets directory next to that file is also used, under the rules above.~ is expanded.import datajoint raises FileNotFoundError.DJ_IGNORE_CONFIG_FILE=true turns it off along with the search.On Windows, the configuration search is unchanged.
<npy@> arrays from remote stores opened with mmap_mode are now cached in a directory for each user: <tempdir>/datajoint_mmap_<uid>, with mode 0700, so only that user can read it. Here <tempdir> is the system's temporary directory, such as /tmp, and <uid> is your numeric user ID.
<tempdir>/datajoint_mmap is no longer used and can be deleted.<tempdir>/datajoint_mmap, which is inside each user's own temporary folder.pymysql >= 1.1.1, up from >= 0.7.2, because of CVE-2024-36039 in PyMySQL.Fixes deletes, drops, and alter() renames that removed data they should have kept.
A patch release on the 2.3 line, recommended for every 2.3 deployment. There are no API changes.
A cascade delete removes only the rows it reaches (#1589, #1590, #1591; #1593):
(T & dj.Top(n)).delete() and delete_quick() start from the rows the Top selects. Before, they started from every row matching the other conditions.part_integrity="cascade", a master joins the delete only if one of its part rows does.On earlier releases, restrict the delete by the keys it should remove: (T & (T & cond).keys()).delete().
status enum, so dropping one jobs table used to remove status from all the others in the schema.alter() keeps an attribute's values when it renames a core-type attribute (#1631; #1635). On MySQL the rename is a CHANGE. On PostgreSQL, alter() refuses the rename before changing anything; renaming there comes in 2.4.alter() resolves foreign keys in the module that declared the table, not the caller's (#1602; #1618).populate(reserve_jobs=True) keeps other workers' reservations and error records (#1594; #1597). Cleanup checks keys against the full key_source; the restrictions only limit which jobs are added. Success and error records of deleted rows go on every refresh.completed_time is recorded to the millisecond, so it can no longer precede reserved_time (#1600; #1614).dj.errors.AccessError, as error 1142 already did (#1586; #1599). Contributed by @Bhanu73120.requires-python is now >=3.10,<3.16 (#1598). On 3.15 only, deepdiff stays below 9 until its dependencies publish 3.15 wheels.(Subject & (Session & "quality = 0")).delete(), raises or deletes nothing, and so does one whose restriction contains % (#1610, #1636). Restrict by the keys instead: (T & (T & cond).keys()).delete().Diagram.restrict() ignores a dj.Top seed and shows every matching row. This affects the preview only, not deletes.Published to PyPI on 2019-04-13, built from this commit. The repository's version.py still read 0.11.1; the version was set only in the uploaded build.
Changes since 0.11.1: v0.11.1...v0.11.2
The archived CHANGELOG listed a later 0.11.2 entry (Jul 25, 2019: "Fix #628 - incompatibility with pyparsing 2.4.1"). That fix was released as 0.11.3.
Unofficial maintenance release. DataJoint no longer supports versions before 2.0. This release only fixes installation with setuptools 82 and later; the full test suite was not run.
setuptools pin and the otumat dependency (#1571, @tabedzki): plugin discovery now uses importlib.metadata instead of pkg_resources, which setuptools 82 removed, and otumat's two helper functions are copied into datajoint.plugin. setuptools and otumat are no longer runtime dependencies. Plugin behavior is unchanged.Full Changelog: v0.14.9...v0.14.10
A patch release on the 2.3 line, focused on where data came from before it reached the pipeline.
Rows that enter the pipeline can now record where they came from. With provenance.capture enabled, a dj.Manual table carries a hidden _prov attribute, declared with the table and filled on insert. It is off by default, like jobs.add_job_metadata and for the same reason: it changes the DDL of every entry table declared afterwards, so that is a deployment's decision rather than a library default. Inside a pipeline provenance is structural — a computed row cannot exist without its declared upstream, so the foreign-key graph is the lineage — but at the boundary that runs out, and until now each pipeline invented its own record of external origin. _prov gives the boundary one shape, so that "which rows have no recorded origin" is a query rather than an audit. No author writes it: insert() takes no provenance argument, and the content comes from deployment configuration (provenance.source), from the connection, and — for an insert inside a make() — from the ingesting table and key, which is what makes the fan-out ingestion pattern traceable without a foreign key. A table declared before capture was enabled records nothing until dj.deploy.add_prov_column adds the slot. See Extrinsic Provenance at Entry Tables and Record Data Origin.
Three tiers gain a second name. dj.Entry, dj.Ingest and dj.Compute join dj.Manual, dj.Imported and dj.Computed. Each pair is one class rather than a subclass, so a table declared either way is identical — same SQL prefix, same tier detection, same describe() and diagram — and both names are permanent, with no deprecation and nothing to migrate. dj.Lookup and dj.Part are unchanged. The documentation keeps the original names for now; adopting the new ones as primary is slated for 2.4 (#1546).
Codecs receive connection context explicitly. encode() and decode() now take a context argument carrying schema, table, field and config, so key means what it means everywhere else — the primary key. Passing the calling connection's config is what matters: without it, path and backend resolution fall back to the global dj.config, which in a process holding connections for several users belongs to none of them and resolves a different store silently. Codecs written before 2.3.4 keep working unchanged; DataJoint passes context only to codecs whose signature declares it.
Two PostgreSQL defects in the hidden columns are fixed. Every platform-managed column — _job_start_time, _job_duration, _job_version, _prov, _singleton — is now declared in DataJoint notation and compiled through the type system, so each backend's mapping comes from one place. _job_start_time had been declared as a bare timestamp and so took microsecond precision on PostgreSQL where MySQL got datetime(3), and migrate.add_job_metadata_columns built its ALTER with backtick-quoted identifiers, which is a syntax error there — the retrofit had never run on PostgreSQL at all. Existing tables are untouched; only new declarations and retrofits change.
Upgrading from 2.3.3? No API breaks, and no DDL change: every new setting is off by default, so an unchanged schema declares under 2.3.4 exactly what it declared under 2.3.3. Reading a hidden attribute still requires SQL —
to_arrays('_prov')andproj('_prov')raise, and a supported accessor arrives in 2.4 (#1562).
Full details in What's New in DataJoint 2.3.
context argument, additive to key(#1556)@dimitri-yatsenko_prov attribute for extrinsic provenance at Entry tables(#1555)@dimitri-yatsenkoFull Changelog: v2.3.3...v2.3.4
A patch release on the 2.3 line, focused on storage credentials, diagram rendering, and type validation.
S3 stores no longer require static credentials. access_key and secret_key are now optional in an s3 store spec — omit both and the AWS credential chain resolves an ambient identity (EC2 instance profile, IRSA, ECS task role, SSO), matching how the gcs and azure protocols already behaved. This is what lets a pipeline run under an assumed role with no long-lived keys in its configuration. Setting exactly one of the two is now rejected at validation with a clear message instead of failing later inside the AWS client. See Configure Storage.
Diagrams were redesigned. dj.Diagram output moves to the DataJoint brand palette — Manual green, Lookup grey, Imported blue, Computed orange, with tier shapes unchanged — and renamed foreign keys are now drawn in amber, distinct from the Computed tier. Line weight now encodes cardinality and only cardinality: thick for one-to-one, thin for one-to-many. It previously also tried to signal the master-part relationship, which made the two unreadable together. A new display.diagram_theme setting takes auto (default), light, or dark; auto emits a single SVG that adapts to the viewer's light or dark mode. See Read Diagrams and Configuration.
Attribute types are validated at declaration and insert. Four related gaps closed, the most serious being that inserting a NumPy array into a native blob attribute silently stored the array's text representation, with no error on insert and none on fetch. decimal(M,D) unsigned is also accepted again after being rejected in 2.x, and misspelled native types are now caught at declaration rather than producing invalid DDL at the server.
Also: braces in a table or attribute comment no longer crash declaration, alter() no longer loses an added attribute's declared type, file-protocol stores are safe on Windows, and non-unique foreign-key columns are indexed on PostgreSQL.
Upgrading from 2.3.2? No API breaks. Two of the fixes above do reject input that previously passed silently — a misspelled native type at declaration, and a NumPy array inserted into a native
blob. Both cases were already broken rather than working: the first produced invalid DDL at the server, the second corrupted the stored value. Anything affected was losing data.
Full details in What's New in DataJoint 2.3.
Full Changelog: v2.3.2...v2.3.3