imposm3/README.md

Imposm 3
========

Imposm is an importer for OpenStreetMap data. It reads PBF files and
imports the data into PostgreSQL/PostGIS. It can also update the
DB from diff files.

It is designed to create databases that are optimized for rendering (i.e. generating tiles or for WMS services).

Imposm 3 is written in Go and it is a complete rewrite of the previous Python implementation.
Configurations/mappings and cache files are not compatible with Imposm 2, but they share a similar architecture.


It is released as open source under the [Apache License 2.0][].

[Apache License 2.0]: http://www.apache.org/licenses/LICENSE-2.0.html


The development of Imposm 3 was sponsored by [Omniscale](http://omniscale.com/) and development will continue as resources permit.
Please get in touch if you need commercial support or if you need specific features.


Features
--------

* High-performance
* Diff support
* Custom database schemas
* Generalized geometries


### In detail


- High performance:
  Parallel from the ground up. It distributes parsing and processing to all available CPU cores.

- Custom database schemas:
  Creates tables for different data types. This allows easier styling and better performance for rendering in WMS or tile services.

- Unify values:
  For example, the boolean values `1`, `on`, `true` and `yes` all become ``TRUE``.

- Filter by tags and values:
  Only import data you are going to render/use.

- Efficient nodes cache:
  It is necessary to store all nodes to build ways and relations. Imposm uses a file-based key-value database to cache this data.

- Generalized tables:
  Automatically creates tables with lower spatial resolutions, perfect for rendering large road networks in low resolutions.

- Limit to polygons:
  Limit imported geometries to polygons from Shapefiles or GeoJSON, for city/state/country imports.

- Easy deployment:
  Single binary with only runtime dependencies to common libs (GEOS, GDAL, SQLite and LevelDB)

- Support for table namespace (PostgreSQL schema)


### Performance ###

Imposm 3 is much faster then Imposm 2 and osm2pgsql.

* Makes full use of all available CPU cores
* Bulk inserts into PostgreSQL with `COPY FROM`
* Efficient intermediate cache for reduced IO load during ways and relations building


Some import times from a Hetzner EX 4S server (Intel i7-2600 CPU @ 3.40GHz, 32GB RAM and 2TB software RAID1 (2x2TB 7200rpm SATA disks)) for imports of a 20.5GB planet PBF (2013-06-14) with generalized tables:

* 6:30h in normal-mode
* 13h in diff-mode

osm2pgsql required between 2-8 days in a [similar benchmark (slide 7)](http://www.geofabrik.de/media/2012-09-08-osm2pgsql-performance.pdf) with a smaller PBF file (~15GB).

Import of Europe 11GB PBF with generalized tables on the same server:

* 2:20h in normal-mode

Benchmarks with SSD are TBD.


Current status
--------------

Imposm 3 is in alpha stadium and there is no official release yet.
The import itself is working however and it was already used for production databases.

### Missing ###

* Documentation
* Updating generalized tables in diff-mode
* Automatic download of diff files
* Tile expire list for re-rendering updated areas
* Background mode for diff-import (update DB in background)
* Diff import into custom PG schemas
* Support for other projections than EPSG:3857
* Improve parallelization of diff import

Installation
------------

### Dependencies ###

#### Compiler

You need [Go >=1.1](http://golang.org).

#### C/C++ libraries

Other dependencies are [libleveldb][], [libgeos][], [libgdal][] and [libsqlite3][].
Imposm 3 was tested with recent versions of these libraries, but you might succeed with older versions.
GEOS >=3.2 is recommended, since it became much more robust when handling invalid geometries.

[libleveldb]:
[libgeos]:
[libgdal]:
[libsqlite3]:

#### Go libraries

Imposm3 uses the following libraries. `go install` will fetch these:

- <https://github.com/jmhodges/levigo>
- <https://github.com/mattn/go-sqlite3>
- <https://code.google.com/p/goprotobuf/proto>
- <https://code.google.com/p/goprotobuf/protoc-gen-go>
- <https://github.com/olt/pq>

### Installation ###

Create a new `GOPATH`

    mkdir imposm
    cd imposm
    export GOPATH=`pwd`

Get Imposm 3 and all dependencies:

    mkdir src
    git clone https://github.com/omniscale/imposm3 src/imposm3
    go install imposm3

Done. You should now have an imposm3 binary in `$GOPATH/bin`.

Go compiles to static binaries and so Imposm 3 has no runtime dependencies to Go.
Just copy the `imposm3` binary to your server for deployment. The C/C++ libraries listed above are still required though.



Usage
-----

`imposm3` has multiple subcommands. Use `imposm3 import` for basic imports.

For a simple import:

    imposm3 import -connection postgis://user:password@host/database -mapping mapping.json -read /path/to/osm.pbf -write

You need a JSON file with the target database mapping. See `example-mapping.json` to get an idea what is possible with the mapping.

Imposm creates all new tables inside the `import` table schema. So you'll have `import.osm_roads` etc. You can change the tables to the `public` schema:

    imposm3 import -connection postgis://user:passwd@host/database -mapping mapping.json -deployproduction


You can write some options into a JSON configuration file:

    {
        "cachedir": "/var/local/imposm3",
        "mapping": "mapping.json",
        "connection": "postgis://user:password@localhost:port/database",
    }

To use that config:

    imposm3 import -config config.json [args...]

For more options see:

    imposm3 import -help

Sorry, that's all documentation for the moment.


Support
-------

There is a [mailing list at Google Groups](http://groups.google.com/group/imposm) for all questions. You can subscribe by sending an email to: `imposm+subscribe@googlegroups.com`

For commercial support [contact Omniscale](http://omniscale.com/contact).

Development
-----------

The source code is available at: <https://github.com/omniscale/imposm3/>

You can report any issues at: <https://github.com/omniscale/imposm3/issues>

### Test ###

#### Unit tests ####

    go test imposm3/...


#### System tests ####

There is a system test that imports and updates OSM data and verifies the database content.
This test is written in Python and requires `nose`, `shapely` and `psycopg2`.

WARNING: It uses your local PostgeSQL database (`import` schema), if you have one. Change the database with the standard PGXXX environment variables.

    cd test
    nosetests test.py