The GAP package distribution
The GAP package distribution is managed via this repository: specifically, it
contains the metadata of all the GAP packages in the distribution. We also
upload snapshots of the package distribution to appropriate release tags
on this repository.
Package lists
High-level status dashboard
| Test |
GAP master |
| Released packages |
|
Instructions for package authors
How to submit package updates
We automatically detect package updates provided these rules are followed:
- The
PackageInfoURL in the PackageInfo.g file of the current version of
your package in the package distribution is valid and points to a copy of
that file from the new version of your package.
- In this
PackageInfo.g file, the ArchiveURL is valid (i.e. when one adds
a file extension from ArchiveFormats it points to an archive containing
your package).
We scan the PackageInfoURL for all packages in the distribution once every hour for updates. When a new
version is detected this way, the package is downloaded and a new pull request
for the update is opened on this repository. A bunch of CI tests are then
started. Assuming they pass, a human will merge that PR, which means your
update is accepted. If a problem is detected, we will instead contact you to
discuss how to resolve it.
In the event that your package is to be moved to a new website, there are two ways
to go about this that ensure we will still be able to pick up updates:
- If you still have access to the old location, simply upload the new
PackageInfo.g
file (containing the new URLs and of course also a new version number) in both
the old and the new location. The system then will as usually automatically detect it,
and once the update is accepted and merged, the system
will use only the new PackageInfoURL.
- If this is not possible, you can also submit the update as if it was a new package,
as described in the next section.
How to submit a new package
See https://www.gap-system.org/packages/submit/ for how to submit
a new package, what to include, and the requirements a package must meet.
Instructions for maintainers of the package distribution
Adding a new package
Whether a package is accepted is decided as described at
https://www.gap-system.org/packages/submit/: submissions are discussed on
the gap@gap-system.org mailing list. A submission made as an issue in this
repository must be forwarded to that list first.
Once a package is accepted, anyone with write access to this repository can
add it, by creating a pull request for it in one of two ways.
Via a GitHub workflow:
- Go to https://github.com/gap-system/PackageDistro/actions/workflows/scan-for-updates.yml
- Click "Run workflow" once to open a popup menu, and enter the URL of the
PackageInfo.g file of the package into the field for a space separated
list of PackageInfo.g URLs.
- Click on the new green "Run workflow" button to actually trigger the workflow.
- After 2-3 minutes this should create a new pull request for each package you listed.
Or by hand, in a clone of this repository:
- Run
tools/import_packages.py URL, where URL is the URL of the
PackageInfo.g file of the package, or the path of a local copy. This needs
GAP with the packages json and crypting, and the Python packages
described under Requirements.
- This creates the file
packages/NAME/meta.json. Commit it on a new branch
and open a pull request.
Once the PR is created, a bunch of CI tests are started. Once they are completed,
a report is added to the PR which indicates whether the new package breaks something
in GAP or other packages, and whether its tests pass. If all looks good, the
PR may be merged by any maintainer. Afterwards, report the outcome on the
mailing list, and close the issue of the submission if there is one.
Removing a package
To remove a package from the distribution, delete its directory
packages/NAME in a pull request. The report added to the PR shows whether
the tests of other packages fail without it.
Testing GAP pull requests against all packages
Maintainers can also run the full PackageDistro test suite against a pull
request on gap-system/gap.
- Go to
https://github.com/gap-system/PackageDistro/actions/workflows/test-gap-pr.yml
- Click "Run workflow".
- Paste the GAP pull request URL into the
gap-pr-url field.
- Optionally restrict the package build or test glob.
The workflow resolves the pull request automatically, checks out the exact
head revision, compares the results against the latest report for the pull
request's target branch, and posts a summary comment back to the GAP pull
request.
How everything works
The following is an overview of the internals of how everything works.
Metadata
For each package in the GAP package distribution, there is a subdirectory of
the packages directory whose name equals that of the package with all
letters turned to lower case. In that directory is a file called meta.json
which contains the metadata for that package as JSON.
For example, the metadata of ACE package is contained in packages/ace/meta.json.
The content of the meta.json file for package is obtained by reading its
PackageInfo.g file in a GAP script, then turning this into JSON data where
possible (basically integers, strings, lists and records are converted to
their natural JSON counterparts; but e.g. functions are obviously not
convertible this way and thus are simply ignored). This is basically the same
form as used by the GAP release scripts to encode which packages are bundled
with a GAP release, except that there it is one big file, while here we have
one file per packages.
In addition to this, we also store the SHA256 checksum of the PackageInfo.g
file resp. of the first package archive under the keys PackageInfoSHA256 resp.
ArchiveSHA256. Here, with "the first package archive", we mean that we
take the archive with first format extension specified by the ArchiveFormats field of
the package info record.
For example, the SHA256 in this excerpt of a meta.json file refers to the
.tar.gz file
"ArchiveFormats": ".tar.gz .zip",
"ArchiveSHA256": "15c516d89863916ef8f4b0d5c68f5b79cb41b75741fc3b604a6a868569dcda38",
"ArchiveURL": "https://github.com/homalg-project/ToricVarieties_project/releases/download/2022-03-04/ToricVarieties",
Note that ArchiveFormats is not necessarily in the same order as in the
package's PackageInfo.g: when importing a package we move .tar.gz to the
front if the package offers it.
Not everything a PackageInfo.g declares is carried over, either: the
AcceptDate, CommunicatedBy and Status fields record the outcome of the
package refereeing process, which is no longer run. Nothing ever verified what
a package claimed there, so these fields are dropped on import and do not
appear in meta.json or in the package-infos.json of a release. Packages may
still declare them -- older versions of GAP expect them -- they simply have no
effect here.
GitHub Workflows
All of this is driven by several GitHub Workflows
which automate most of the maintenance on the GAP package distribution. For
details on how they operate, have a look at the .yml files in
.github/workflows.
These workflows are in parts implemented by calling scripts in the tools directory.
Requirements
The scripts in the tools directory are written in Python 3 or GAP. For the Python
scripts, you must make sure their prerequisites are installed, e.g. by invoking the
following command once from the root of this repository:
python -m pip install -r tools/requirements.txt
For information about what each script does, please consult its source, which
should have comments explaining what it does and how to invoke it.
If you need to make changes to the Python code, note that we are using the
black code formatter. Pull requests modifying
or adding Python code must run black on all Python files in order to pass the
test suite and be merged.
Other directories
The various scripts create and/oruse a bunch of auxiliary directories to store
and exchange data:
_archives is used to store copies of all package archives
_pkginfos stores copies of the PackageInfo.g
_releases: TODO
What do we test, and how?
In a nutshell, when a new package update is received, we validates its
metadata (including validating URLs in it), and run the testsuites of GAP and
of all GAP packages in the package distribution. If any of these detects an
issue, this is reported.
Some more details:
- if a package update is detected, we fetch its latest metadata and validate it
- This can be replicated manually by doing the following in an up-to-date clone
of the PackageDistro repository (substitute
example by the name of your package)
./tools/scan_for_updates.py example
./tools/validate_package.py example
This downloads the metadata from the PackageInfoURL, and hence only works once
you made a release. To try out an update beforehand, pass the directory holding
the PackageInfo.g of your working copy (or that file itself) instead of the
package name, and the metadata is read from there:
./tools/scan_for_updates.py path/to/example
./tools/validate_package.py example
Note that the source archive is still fetched from ArchiveURL: until you
publish it, scan_for_updates.py warns and records FAIL as its checksum, and
validate_package.py fails outright because it cannot download the archive it
is supposed to check.
- The checks this perform include
- check output of GAP function
ValidatePackageInfo
- verify version is greater than previous version and does not contain
dev
- verify release date is not less than previous release date
- verify release date is not in the future
- verify URLs are valid (currently
PackageInfoURL, README_URL, source archives)
- validates source archives, including:
- must not contain absolute paths, or rather: all files in the archive must be
contained in a single parent subdirectory
- must contain
PackageInfo.g
- must not contain symlinks
- clone the GAP repository and build GAP
- this (and all steps after this) is currently done only for the latest
GAP
master branch; testing the latest stable-X.Y branch is disabled
- compile all packages that require it
- actually we skip
xgap and also don't test it for technical reasons
- start GAP, execute
LoadAllPackages(), run GAP's testinstall test suite
- for each package
pkgname in the distribution
- start GAP, do
LoadPackage("pkgname") then TestPackage("pkgname")
- start GAP, do
LoadPackage("pkgname" : OnlyNeeded) then TestPackage("pkgname")
- unfortunately quite some packages have issues with that as their test
suites actually require more than the minimal set of dependencies. We
currently are lenient there and include custom treatment for a bunch of
packages (for details see the "Run tests with OnlyNeeded" step in
.github/workflows/test-all.yml)
- start GAP, do
LoadPackage("pkgname" : OnlyNeeded) and check whether GAP
printed any syntax errors or warnings for files of pkgname while doing so
- such messages do not make loading fail, so nothing else notices them, but
they look alarming to users and almost always indicate a genuine mistake
in the package, which we then report upstream