The creation of a world map based on OpenStreetMap data

the Backstory

When developing a project for online navigation (and offline in the future) there was a need to develop their own online maps. Services from Google, Bing, etc. have limited use of their maps for commercial purposes. Free, like OpenStreetMap, did not satisfy the requirements of the customer.
There was a question about creating online maps (tile maps) for the whole world. Style close to Google and Skobbler. Data based on OpenStreetMap.

Limitations

Known file systems have limitations in number of files and directories. Therefore, the storage of each tile (tile maps) separately depends on these restrictions. For instance, the 14th Zuma need 267 583 488 tiles (~4¹⁴).
View the maximum number of files in Linux: df-i
The solution is to store the tiles in the database.

Iron and axis

Server generation and load data: Supermicro server. 24 cores at 2GHz, 64Gb RAM, HDD 2.7 Tb. With Ubuntu 13.04 on the Board.
A server which is using the results of generation are much more modest: 8 cores at 1.8 GHz, 27Gb RAM, 130Gb SSD.

Installation and configuration

To focus on the installation of will not. As it is well described in other articles (see links below).
Installed:
the

the
• PostgreSQL 9.1
the
• PostGIS
the
• Mapnik
the
• TileMill with OSM Bright (default style)
the
• Osm2pgsql
the
• SQLite 3
the
• Node.js
the
• nginx

Development

In order not to reinvent the wheel is turned to open source. The best option for storing tiles in a database format MBTiles. It's a simple SQLite database allowing to store millions of tiles in a single file.
To develop and edit their own style, a good editor, TileMill, besides it allows to save maps in MBTiles format.
First you need to load a separate area (country) and work styles. It is desirable to study several countries with different alphabets: Latin, Cyrillic, Arabic, kanji. So you don't face the unpleasant surprise of some missing names or incorrect display style for different countries. To download the world and work styles do not suggest, as the time of dynamic generation in the editor is increased.

loading the data into PostgreSQL

Tuning PostgreSQL

To speed up the process of downloading will be doing some changes to the file /etc/postgresql/9.1/main/postgresql.conf
the max_connections = 150 is evaluated empirically, depending on the number of processors. Defaults to 100.
shared_buffers = 7GB — <25% of system RAM. For this option you need to increase kernel.shmmax.
temp_buffers = 512MB — used to access temporary tables, you need to utility osm2pgsql slim mode.
work_mem = 3GB — used for sorting “ORDER BY” and Association “JOIN”.
maintenance_work_mem = 16GB — commands used for VACUUM, CREATE INDEX, ALTER TABLE ADD FOREIGN KEY.
random_page_cost = 3.0 is the setting load of the processors. Be careful with this option!
effective_cache_size = 42GB up to 66% of the RAM.
checkpoint_segments = 50
autovacuum = off

Loading with osm2pgsql

Data to download take on planet.osm.org.
To load the data in PostgreSQL was used osm2pgsql utility. To download around the world has been used slim mode, it allows to limit the RAM usage when it creates temporary tables to store intermediate results.
Before downloading the fix file styles which in Ubuntu to be in /usr/local/share/osm2pgsql/default.style. For the non-Latin names I have added three fields: int_name (original), name:en (name in English), is_in:country (country).
The command to download around the world:
the

time sudo -u postgres osm2pgsql -r pbf -sc-d gis -C 40000 --number-processes 24 --cache-strategy dense --unlogged planet-latest.osm.pbf

I will explain the individual settings:
-r pbf — used pbf format;
-sc — mode slim-mode (s) by creating new tables (c), to add use a;
--number-processes 24 — the number of parallel processes, the number of CPU-s;
Download around the world took 28 hours.

Create your own styles in TileMill

the Latin name

As I wrote above first I downloaded PostgreSQL in several individual countries. So how in the world are used many alphabets need to be processed and the labels are not written in the Latin alphabet (after the original in brackets to indicate the name in the Latin alphabet). To do this, the table planet_osm_point create a field nonlatin:
the

ALTER TABLE planet_osm_point ADD COLUMN nonlatin boolean;
UPDATE planet_osm_point SET nonlatin='1' WHERE name similar to '%[^\x20-\x7e]+%';

Edit styles

To have the default styles, you need to install OSM Bright for TileMill.
TileMill is able to work with three data sources: files, SQLite, PostGIS. For the design of map styles used CartoCSS, if you are familiar with CSS, you'll find a lot in common.
Be careful, some filters (queries to PostGIS PostgreSQL) from OSM Bright is quite heavy and require a lot of time to complete. So you need to optimize the SQL query for acceleration.
Sample Latin and non-Latin names in queries:
the

SELECT ..,
CASE WHEN nonlatin=true AND "name:en" IS NOT NULL THEN CONCAT("name:en",' (',name,')') ELSE name END as en_name
...

Generate MBTiles

In the select Export TileMill/MBTiles. Specify the name of the file, Zuma (approximation of maps), the coordinates of the center and border.
Borders all over the world: -180.0, -85, 180.0, 85
For convenience, generate card private sumami.
the

the
• Zuma 0-10, the total generation time is 13 hours
the
• Zoom 11, total time 7 hours
the
• Zoom 12, 18 hours
the
• Zoom 13, 3 days
the
• Zoom 14, 9 days

Zuma 15, 16, 17, decided to leave the dynamic generation. To speed to use caching.

some MBTiles files

The unification of the different MBTiles files, you can use the command:
the

echo '.dump' | sqlite3 file1.mbtiles | sqlite3 file2.mbtiles

The result will be file2.mbtiles.

Usage

MBTiles

To start using MBTiles Node.js:
the

nohup node /usr/share/tileserver/mbtiles-server/server.js /media/data/mbtiles/filename.mbtiles PORTNUMBER &

Where PORTNUMBER is the port number which will be available tiles (tiles) of the map.

NGINX configuration File

To access the map using nginx.
An example configuration file:
the

server {
listen 80;
server_name your_server_map_tiles_name.com;

#dinamically generation
location ^~ /15/ {
rewrite ^/15/(.*)$ http://localhost:20008/tile/projectName/15/$1 last;
}
...
#using MBTiles via runned port
location / {
proxy_pass http://localhost:PORTNUMBER;
proxy_set_header Host $Host;
proxy_set_header X-Real-IP $remote_addr;
}
}

Thus at your_server_map_tiles_name.com/zoom/x/y.png is available in tiles of your map.

Features and suggestions

Best MBTiles file to place on the SSD to speed up loading.
For the iPhone application uses a non-standard Retina tiles with a size of 512 × 512 pixels.
To reduce the load for a dynamic generation (Zuma 15-17) use caching in Redis.

Result

You can see the result here: tourstart.org/drive
Or download the iPhone app: itunes.apple.com/app/tourstart/id586049610?mt=8

the Articles used in the process

habrahabr.ru/post/144675
switch2osm.org/serving-tiles/manually-building-a-tile-server-12-04
wiki.openstreetmap.org/wiki/Osm2pgsql
github.com/mapbox/mbtiles-spec
www.mapbox.com/tilemill/docs/linux-install
www.mapbox.com/tilemill/docs/guides/osm-bright-mac-quickstart
www.mapbox.com/carto/api/2.1.0

Sorry. To embed in the hub OpenStreetMap enough karma.
I will be glad to answer your questions.

Article based on information from habrahabr.ru