vitalif
/
imposm3
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

update docs

master
Oliver Tonnhofer 2015-05-07 16:40:50 +02:00
parent 7ddba3382c
commit 615520b51f
5 changed files with 162 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/conf.py
View File

@ -44,7 +44,7 @@ master_doc = 'index'
# General information about the project.
project = u'Imposm3'
copyright = u'2014, Oliver Tonnhofer'
copyright = u'2015, Oliver Tonnhofer'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@ -127,7 +127,7 @@ html_theme = 'sphinxdoc'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied

7
docs/index.rst
View File

@ -1,7 +1,7 @@
Imposm3
=======
Imposm3 is an importer for OpenStreetMap data. It reads PBF files and can import the data into PostgreSQL/PostGIS databases.
Imposm3 is an importer for OpenStreetMap data. It reads PBF files and imports the data into PostgreSQL/PostGIS databases.
It is designed to create databases that are optimized for rendering/tile/map-services.
It is developed and supported by `Omniscale <http://omniscale.com>`_ and is released as open source under the `Apache Software License 2.0 <http://www.apache.org/licenses/LICENSE-2.0.html>`_. Imposm3 is a rewrite of Imposm 2 with even better performance and support for (minutely) diff updates.
@ -11,7 +11,7 @@ Features
--------
Custom database schemas
It creates tables for different data types. This allows easier styling and better performance for rendering in WMS or tile services.
It creates separate tables for different feature types. This allows easier styling and better performance for rendering in tile or WMS services.
Multiple CPU/core support
Imposm is parallel from the ground up. It distributes parsing and processing to multiple CPUs/cores.
@ -29,7 +29,7 @@ Generalized tables
It can automatically create tables with lower spatial resolutions, perfect for rendering large road networks in low resolutions for example.
Limit to polygons
It can limit imported geometries to polygons from Shapefiles or GeoJSON.
It can limit imported geometries to polygons from GeoJSON.
HStore support
Don't know which tags you will be needing? Store all tags in an `HStore column <http://www.postgresql.org/docs/9.3/static/hstore.html>`_.
@ -55,6 +55,7 @@ Contents
.. toctree::
:maxdepth: 2
install
tutorial
mapping

14
docs/install.rst Normal file
View File

@ -0,0 +1,14 @@
Installation
============
Binary
------
There are no official releases, but you find development builds at <http://imposm.org/static/rel/>.
These builds are for x86 64bit Linux and require *no* further dependencies. Download, untar and start `imposm3`.
(Note: These binaries require glibc >= 2.15 at the moment. Ubuntu 12.04 is recent enough, Debian 7 not.)
Source
------
Please see `the README for build instructions <https://github.com/omniscale/imposm3/#installation>`_.

106
docs/mapping.rst
View File

@ -1,7 +1,7 @@
Data Mapping
============
The data mapping defines which `OSM feature types <http://wiki.openstreetmap.org/wiki/Map_Features>`_ should be imported in which table. The mapping is described with a JSON file.
The data mapping defines which `OSM feature types <http://wiki.openstreetmap.org/wiki/Map_Features>`_ should be imported in which table. The mapping is a JSON file.
See `example-mapping.json <https://raw.githubusercontent.com/omniscale/imposm3/master/example-mapping.json>`_ for an example.
@ -9,7 +9,7 @@ See `example-mapping.json <https://raw.githubusercontent.com/omniscale/imposm3/m
Tables
------
The most important part is the ``tables`` definition. Each table is a JSON object with the table name as the key. Each table has a ``type`` a mapping definition and ``columns``.
The most important part is the ``tables`` definition. Each table is a JSON object with the table name as the key. Each table has a ``type``, a mapping definition and ``columns``.
``type``
@ -22,6 +22,7 @@ The most important part is the ``tables`` definition. Each table is a JSON objec
~~~~~~~~~~~
``mapping`` defines which OSM key/values an element needs to have to be imported into this table. ``mapping`` is a JSON object with the OSM `key` as the object key and a list of all OSM `values` to be matched as the object value.
You can use the value ``__any__`` to match all values.
To import all polygons with `tourism=zoo`, `natural=wood` or `natural=land` into the ``landusages`` table:
@ -47,8 +48,6 @@ To import all polygons with `tourism=zoo`, `natural=wood` or `natural=land` into
}
``columns``
~~~~~~~~~~~
@ -62,8 +61,8 @@ This is the name of the resulting column.
``type``
^^^^^^^^
This defines the data type that Imposm should use for this column. There are two different classes of types. `Value types` are types that convert OSM values to database types. Examples are ``string`` for street or place names, ``bool`` for ``yes``, ``no``, ``true`` or ``false``.
`Element types` are types that dependent on the OSM lement (node/way/relation). Examples are ``geometry`` for the geometry, ``mapping_key`` and ``mapping_value`` for the actual key and value that was mapped.
This defines the data type that Imposm should use for this column. There are two different classes of types. `Value types` are types that convert OSM tag values to a specific database type. Examples are ``string`` for street or place names, or ``bool`` for ``yes``, ``no``, ``true`` or ``false`` values.
`Element types` are types that dependent on the OSM element (node/way/relation). Examples are ``geometry`` for the geometry, ``mapping_key`` and ``mapping_value`` for the actual key and value that was mapped.
See :ref:`column_types` for documentation of all types.
@ -79,7 +78,6 @@ See :ref:`column_types` for documentation of all types.
Some column types require additional arguments. Refer to the documentation of the type.
Example
~~~~~~~
@ -134,6 +132,37 @@ The mapping below will create a ``tracks`` table with the following columns:
}
}
``mappings``
~~~~~~~~~~~~
An OSM element is only inserted once even if a mapping matches multiple tags. Sometime it's convenient to have a geometry multiple times, e.g. a way with ``rail=tram`` and ``highway=secondary``.
``mappings`` allows to define multiple sub-mappings. Each sub-mapping requires a name and a separate mapping dictionary. The elements will be inserted into the table for each match of a sub-mapping.
.. code-block:: javascript
:emphasize-lines: 5-12
{
"tables": {
"transport": {
"type": "linestring",
"mappings": [
"rail": {
"rail": ["__any__"]
},
"roads": {
"highway": ["__any__"]
}
]
},
...
}
}
}
.. _column_types:
@ -157,7 +186,7 @@ Same as ``bool`` but stores a numeric ``1`` for ``true`` values, and ``0`` other
``string``
^^^^^^^^^^
The value as-is. Missing values will be inserted as an empty string and not as ``null``.
The value as-is. Note that missing values will be inserted as an empty string and not as ``null``. This allows SQL queries like ``column NOT IN ('a', 'b')``.
``direction``
@ -179,7 +208,7 @@ Element types
``id``
^^^^^^
The ID of the OSM node, way or relation. Relation IDs are negated (-1234 for ID 1234) to avoid collisions with relation IDs.
The ID of the OSM node, way or relation. Relation IDs are negated (-1234 for ID 1234) to prevent collisions with way IDs.
``mapping_key``
@ -213,18 +242,69 @@ Like `geometry`, but the geometries will be validated and repaired when this tab
``pseudoarea``
^^^^^^^^^^^^^^
Area of polygon geometries in square meters. This area is calculated in the webmercator projection, so it is only accurate at the equator and gets off the more the geometry move to the poles. It's still good enough to sort features by area for rendering purposes.
Area of polygon geometries in square meters. This area is calculated in the webmercator projection, so it is only accurate at the equator and gets off the more the geometry moves to the poles. It's still good enough to sort features by area for rendering purposes.
``wayzorder``
^^^^^^^^^^^^^
Calculate the z-order of an OSM highway or railway. Returns a numeric value that represents the importance of a way where ``motorway`` is the most important (9), and ``path`` or ``track`` are least important (0). ``bridge`` and ``tunnel`` will modify the value by -10/+10. ``layer`` will be multiplied by ten and added to the value. E.g. ``highway=motorway``, ``bridge=yes`` and ``layer=2`` will return 39.
Calculate the z-order of an OSM highway or railway. Returns a numeric value that represents the importance of a way where ``motorway`` is the most important (9), and ``path`` or ``track`` are least important (0). ``bridge`` and ``tunnel`` will modify the value by -10/+10. ``layer`` will be multiplied by ten and added to the value. E.g. ``highway=motorway``, ``bridge=yes`` and ``layer=2`` will return 39 (9+10+2*10).
``hstore_tags``
^^^^^^^^^^^^^^^
Stores all tags in a HStore column. Requires the PostGIS HStore extension. This will only insert tags that are referenced in the ``mapping`` or ``columns`` of any table. See :ref:`tags` on how to import all availabel tags.
.. TODO
.. "hstore_tags": {"hstore_tags", "hstore_string", HstoreString, nil},
.. "wayzorder": {"wayzorder", "int32", WayZOrder, nil},
.. "zorder": {"zorder", "int32", nil, MakeZOrder},
.. "string_suffixreplace": {"string_suffixreplace", "string", nil, MakeSuffixReplace},
Generalized Tables
------------------
Generalized tables allow you to create a copy of an imported table with simplified/generalized geometries. You can use these generalized tables for rendering low map scales, where a high spatial resolution is not required.
Each generalize table is a JSON object with the new table name as the key. Each generalize table has a ``source`` and a ``tolerance`` and optionally an ``sql_filter``.
``source`` is the table name of another Imposm table from the same mapping file. You can also reference another generalized table, to create multiple generalizations of the same data.
``tolerance`` is the `resolution` used for the Douglas-Peucker simplification. It has the same unit as the import `-srid`, i.e. meters for EPSG:3857 and degrees for EPSG:4326. Imposm uses `PostGIS ST_SimplifyPreserveTopology <http://postgis.net/docs/ST_SimplifyPreserveTopology.html>`_.
The optional ``sql_filter`` can be used to limit the rows that will be generalized. You can use it to drop geometries that are to small for the target map scale.
::
"generalized_tables": {
"waterareas_gen_50": {
"source": "waterareas",
"sql_filter": "ST_Area(geometry)>50000.000000",
"tolerance": 50.0
},
.. _tags:
Tags
----
Imposm caches only tags that are required for a ``mapping`` or for any ``columns``. This keeps the cache small as it does not store any tags that are not required for the import. You can change this if you want to import all tags, e.g with the ``hstore_tags`` column type.
Add ``load_all`` to the ``tags`` object inside your mapping JSON file. You still manually can exclude single tags with the ``exclude`` option.
To load all tags exepct ``created_by`` and ``source``::
"tags": {
"load_all": true,
"exclude": [
"created_by",
"source"
]
},

67
docs/tutorial.rst
View File

@ -1,7 +1,16 @@
Tutorial
========
The import process is separated into multiple steps. You can combine most steps in one command, but we will explain each one in detail here.
The ``imposm3`` command has multiple sub-commands. This tutorial will explain the most important sub-commands: ``import`` and ``diff``.
Use the name of the sub-command as the second argument to start it. For example, to print the installed version number call the ``version`` sub-command::
$ imposm3 version
0.1dev-20150507-7ddba33
Preparation
^^^^^^^^^^^
Create database
---------------
@ -23,7 +32,7 @@ You also need to make sure that the user is allowed to access the database from
host all all 127.0.0.1/32 md5
(Don't forget to reload PostgreSQL after changes.)
(Don't forget to reload PostgreSQL after these changes.)
The following command should print the PostGIS version, if everything was successful::
@ -32,36 +41,40 @@ The following command should print the PostGIS version, if everything was succes
Please `refer to the PostGIS <http://postgis.net/docs/index.html>`_ and `PostgreSQL documentation <http://www.postgresql.org/docs/9.3/interactive/manage-ag-createdb.html>`_ for more information.
Importing
^^^^^^^^^
The import process is separated into multiple steps.
You can combine most steps in one command, but we will explain each one in detail here.
Reading
-------
The first step is the reading of the OpenStreetMap data. Building the way and relation geometries requires random access to all nodes and ways, but this is not supported by the OSM PBF data format. Imposm needs to store all nodes, ways and relations in an intermediary data store that allows random access to these elements. It does this on-disk to keep the memory usage of Imposm low. Having lot's of memory will still speed the import up, because your OS will use all free memory for caching of these files.
The first step is the reading of the OpenStreetMap data. Building the way and relation geometries requires random access to all nodes and ways, but this is not supported by the OSM PBF data format. Imposm needs to store all nodes, ways and relations in an intermediary data store that allows random access to these elements. It does this on-disk to keep the memory usage of Imposm low. Having lots of memory will still speed the import up, because your OS will use all free memory for caching of these files.
Imposm uses LevelDB key-value databases for this, which are fast and compact.
Imposm needs to know which OSM elements you want to have in your database. You can use the provided ``mapping.json`` file for this tutorial, but you should read :doc:`mapping` for more information on how to define your own mapping.
To read an extract::
imposm3 import -mapping mapping.json -read germany.osm.pbf
Cache files
~~~~~~~~~~~
Imposm stores the cache files in `/tmp/imposm3`. You can change that path with ``-cachedir``. Imposm can merge multiple OSM files into the same cache (e.g. when combining multiple extracts) with the ``-appendcache`` option or it can overwrite existing caches with ``-overwritecache``. Imposm will fail to ``-read`` if it finds existing cache files and you don't specify ``-appendcache`` or ``-overwritecache``.
Imposm stores the cache files in `/tmp/imposm3`. You can change that path with ``-cachedir``. Imposm can merge multiple OSM files into the same cache (e.g. when combining multiple extracts) with the ``-appendcache`` option or it can overwrite existing caches with ``-overwritecache``. Imposm will fail to ``-read`` if it finds existing cache files and if you don't specify either ``-appendcache`` or ``-overwritecache``.
Writing
-------
The second step is the writing of OpenStreetMap features into the database. It reads the features from the cache from step one, builds all geometries and imports them into the according tables. It overwrites existing tables, :ref:`see below <production_tables>` how to work with existing datasets.
The second step is the writing of OpenStreetMap features into the database. It reads the features from the cache from step one, builds all geometries and imports them into the according tables. It overwrites existing tables, :ref:`see below <production_tables>` for more information on how to update your database in production.
After the import, it creates the generalized tables and indicies.
You need to tell Imposm the connection parameters of your database. The ``-connection`` option takes a URL in the format ``postigs://username:password@host:port/databasename``.
You need to tell Imposm the connection parameters of your database. The ``-connection`` option takes a URL in the format ``postgis://username:password@host:port/databasename``.
In our example:
::
@ -70,20 +83,20 @@ In our example:
You can combine reading and writing::
imposm3 import -mapping mapping.json -read -write -connection postgis://osm:osm@localhost/osm hamburg.osm.pbf
imposm3 import -mapping mapping.json -read hamburg.osm.pbf -write -connection postgis://osm:osm@localhost/osm
Limit to
~~~~~~~~
You can limit the imported geometries to polygon boundaries. You can load the limit-to polygons from GeoJSON files. Linestrings and polygons will be clipped exactly at the limit to geometry. The geometries need to be in EPSG:4326.
You can limit the imported geometries to polygon boundaries. You can load the limit-to polygons from GeoJSON files. Line strings and polygons will be clipped exactly at the limit to geometry. The GeoJSON needs to be in EPSG:4326.
::
imposm3 import -mapping mapping.json -connection postgis://osm:osm@localhost/osm -read europe.osm.pbf -write -limitto germany.geojson
``-limitto`` also controls which elements are stored in the internal cache. You can configure a buffer around the ``-limitto`` geometry with the ``-limittocachebuffer`` to add more elements to your cache. This is necessary for getting complete polygons and linestrings at the boundaries of your ``-limitto`` geometry.
``-limitto`` also controls which elements are stored in the internal cache. You can configure a buffer around the ``-limitto`` geometry with the ``-limittocachebuffer`` to add more elements to your cache. This is necessary for getting complete polygons and line strings at the boundaries of your ``-limitto`` geometry.
Config file
~~~~~~~~~~~
@ -133,17 +146,17 @@ You can combine reading, writing and optimizing::
Deploy production tables
------------------------
Since Imposm overwrites existing tables on import, it is recommended to use different schemas for import and for production.
Since Imposm overwrites existing tables on import (``-write``), it is recommended to use different schemas for import and for production.
Imposm imports all tables into the ``import`` schema by default. For example, after the import the table ``osm_roads`` is accessible as ``import.osm_roads`` and not as ``osm_roads`` or ``public.osm_roads``.
.. note:: Database schemas are a feature of a few databases including PostgreSQL to define multiple namespaces for tables. Don't mistake this for database schemas (as in data model) which are discussed in doc:`mapping`.
Imposm can `deploy` all imported tabels by updating the schema of the tables.
To move all tables form ``import`` the default schema ``public``::
Imposm can `deploy` all imported tables by updating the schema of the tables.
To move all tables form ``import`` to the default schema ``public``::
imposm3 import -mapping mapping.json -connection postgis://osm:osm@localhost/osm -deployproduction
This will also move all existing Imposm tables from the ``public`` to the ``backup``.
This will also remove all existing Imposm tables from ``backup`` and it will moves tables from the ``public`` to the ``backup`` schema.
You can revert a deploy (moving ``public`` tables to ``import`` and ``backup`` tables to ``public``)::
@ -161,7 +174,7 @@ Other options
Projection
~~~~~~~~~~
Imposm uses the the web mercator projection (``EPSG:385``) for the imports. You can change this with the ``--srid`` option. At the moment only EPSG:3857 and EPSG:4326 are supported.
Imposm uses the the web mercator projection (``EPSG:3857``) for the imports. You can change this with the ``-srid`` option. At the moment only EPSG:3857 and EPSG:4326 are supported.
Diff
@ -173,6 +186,24 @@ Imposm needs to cache a few more information to be able to update the database f
imposm3 import -config config.json -read hamburg.osm.pbf -write -diff
Read :doc:`diff` for more information.
Read :ref:`diff` for more information.
.. note:: Each diff import requires access to the cache files from this initial import. So it is a good idea to set ``-cachedir`` to a more premanent location then `/tmp/`.
.. note:: Each diff import requires access to the cache files from this initial import. So it is a good idea to set ``-cachedir`` to a premanent location instead of `/tmp/`.
.. _diff:
Updating
^^^^^^^^
Imposm allows you to update an existing database by importing changes from an `OSM changes file <http://wiki.openstreetmap.org/wiki/OsmChange>`_. Changes files contain all edits made to the OSM dataset in a defined time-range. These files are `available at planet.openstreetmap.org <http://wiki.openstreetmap.org/wiki/Planet.osm/diffs>`_.
The ``diff`` sub-command requires similar options as the ``import`` sub-command. You can pass one or more XML changes files to ``diff``, instead of a PBF file for the ``-read`` option.
To update an existing database with three change files::
imposm3 diff -config config.json changes-1.osc.gz changes-2.osc.gz changes-3.osc.gz
Remember that you have to make the initial import with the ``-diff`` option. See above.
.. note:: You should not make changes to the mapping file after the initial import. Changes are not detected and this can result aborted updates or incomplete data.