Import OpenStreetMap data into a PostgreSQL/PostGIS database
Repo osm2pgsql-dev/osm2pgsql, read from GitHub 23 h ago. osm2pgsql-dev has not joined Take My Repo.
This is a bugfix release which fixes a segfault that may be encountered when using the new diff expire on tables with multiple expire outputs.
It also fixes installation of man pages (thanks @sebastic).
Quick and efficient expire and re-rendering of tiles with changed data is important for OSM. Historically osm2pgsql didn't do a great job figuring out which tiles are affected by some change. Quite some work went into this release to improve that situation. Most importantly polygons are now not expired by bounding box any more but the shape of the polygon is taken into account. For
large polygons with weird shapes, this can make quite a difference.
A larger refactoring was needed to be able to support "diff expire": When an object changes, we used to always expire all tiles intersecting the old or the new geometry of an object. We can now expire tiles based on the symmetric difference between old and new geometry. A small change in, say, a large lake will then only affect the tile(s) covering that small change, not the whole
area. This feature has to be enabled using the diff_expire = true setting in the table definition.
The number of tiles to be expired can be quite large if the input geometries are large or if there are many geometries. Numbers of tiles in the billions can crash osm2pgsql because it runs out of memory. Such large numbers can also overwhelm any kind of re-rendering mechanism run after osm2pgsql to bring tiles up to date. In day-to-day processing this should not happen, but it can happen due to vandalism or misconfiguration.
To protect against this problem, this version introduces limits on the number of tiles that can be affected by a single geometry and the overall number of tiles that an expire output will generate for each run of osm2pgsql. Defaults for these limits are quite high to not negatively affect existing setups. It is recommended to review and adapt the limits for each expire output based on your configuration and expire processing.
Overall the code related to expire processing is much cleaner and some bugs in corner-cases were removed.
The newly added tool osm2pgsql-test-style allows you to test your Lua style file using a BDD-style testing language. osm2pgsql has tested its own styles and functionality with such BDD tests for a long time. We've cleaned up the code of the framework behind these tests and moved it into its own stand-alone file, so it can now be used externally.
It is sometimes useful to process tagged nodes together with the ways they are on. Examples are barrier tags on highways (such as a lift gate) or a ford in a river. On some maps, the size and orientation of the "lift gate" or the "ford" symbol depends on the type and geometry of the way it is on.
Doing the processing for this is possible in PostGIS, but not easy and not cheap. So we added some functionality to osm2pgsql to help with this: When processing nodes, you instruct osm2pgsql in the Lua config file to store the Ids of some nodes (for instance with the "ford=yes" tag) in the "Id cache". Later, when processing the ways this information can be retrieved and used to store the node together with the way geometry or do other processing.
Thanks to @dch0ph for bringing up this issue and testing the implementation.
We are adding more and more functionality around geometries to osm2pgsql accessible from the Lua config file. It is more convenient and often much faster to do this processing while the data is being imported instead of inside the database later on.
:as_point(n) can now be used to create a point geometry from an OSM way object. The parameter n is the 1-based index of the member nodes.spherical_area() function was changed to use the spheroid instead of a sphere, it now returns the same value as ST_Sphere(geography, use_spheroid = true).spherical_length() function was added.== operator.n_points() function to get the number of points in any geometry.There are lots of smaller fixes, cleanups, improvements and other changes:
max_tiles_per_run to limit the number of tiles processed per run to spread the update load over a longer time.double precision, and timestamp (with or without time zone). Thanks to @amandasaurus for the double precision PR.osm2pgsql-expire is now installed on make install.This release adds two powerful building blocks that can be used to implement all sorts of new features. The new "Locators" and the "deleted callbacks" features seem simple by themselves, but they can be used as basis for many features that have been requested for years.
This release introduces a new feature: Locators. They help with a common problem, namely how to quickly find out in which region an OSM object is. The region can be a country or any other area. A locator is initialized with one or more regions, each region has a name and a polygon or bounding box. A geometry of an OSM object can then be checked against this region list to figure out in which region(s) it is located. This check is much faster than it would be to do this inside the database after import.
Locators can be used for all sorts of interesting features, for instance:
For details see the manual and look at the example config files provided in the flex-config/locator directory.
The Lua callback functions process_node(), process_way(), and process_relation() (and their untagged companions) only get called for new or changed OSM objects. Deleted objects are usually handled behind the scenes and for most use cases this is enough.
But sometimes it is useful to also know when an object was deleted. That's why we have new process_deleted_node/way/relation() callbacks now. These open up a lot of new processing options for working with changes. Some users do advanced processing of OSM data inside the database after the data is imported with osm2pgsql. This is much easier now because you don't need database triggers anymore to detect and process deleted objects.
We have added a osm2pgsql-expire command. It is currently marked as experimental so it might change without notice. You can use it to turn entries in expire files (that look like ZOOM/X/Y) into a GeoJSON file visualizing the tiles. When given an OSM data file it can generate the tile output (in expire file format or as GeoJSON). This tool is intended as a debugging aid.
process_node/way/relation() callback even if they had no tags. This is now fixed, the process_untagged_*() callbacks are called instead.--username and --user command line options in all commands, there was an inconsistency between osm2pgsql and osm2pgsql-replication before.This is a bug fix release which contains the following:
This is a bugfix and maintenance release. The main visible change is that osm2pgsql now works better together with connection poolers and similar database middleware: prepared statements now use protocol level prepare instead of PREPARE sql commands and there is a new option to explicitly create a primary key on flex tables.
Other fixes and features:
This release fixes some small issues with 2.0.0:
--schema parameter from osm2pgsql-replication to osm2pgsqlosm2pgsql_find_changed_ways function in the right schema (thanks @falko17)