Mimirsbrunn

• What's a Geocoder
• Getting Started
  • Prerequisites
  • Installing
  • Running
  • Testing
  • Where to Go From There
• Development
  • Contributing
  • Versioning
  • Authors
• Processes
• Who Uses It ?
• Alternatives
• Resources
• License
• Acknowledgments

Mimirsbrunn (also called Mimir) is an independent geocoding and reverse-geocoding system written in Rust, and built upon Elasticsearch. It can handle addresses, streets, points-of-interest (POI), administrative regions and public transport stops.

What's a Geocoder ?

Usually geocoding refers to "the process of transforming a physical address description to a location on the Earth's surface". However Mimir is more a geoparser than a geocoder since it can resolve any ambiguous toponym to its correct location.

In other words, a geocoder reads a description (possibly incomplete) of a location, and returns a list of candidate locations (latitude / longitude) matching the input.

Geocoding is traditionally used for autocompleting search fields used in geographic applications. For example, here is a screenshot of Qwant Maps, where the user enters a search string 20 rue hec mal, and mimir returns possible candidates in a dropdown box.

Getting Started

For an introduction to the project, you can have a look at the Introduction to Mimirsbrunn.

These instructions will give you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on deploying the project on a live system.

Prerequisites

Mimirsbrunn is a rust project, so you need to have a rust development environment. Instructions on how to do that are found on the rust website.

Additionally, mimirsbrunn relies on cosmogony to create so called cosmogony files which are used for indexing administrative regions.

For running end to end or unit tests, you need the docker engine available.

Installing

Debian

You can install Mimirsbrunn from debian packages available as build artifacts on the repository homepage. [FIXME Where are .deb ?]

Docker

You can also fetch official images from DockerHub here:

• navitia/mimirsbrunn:latest
• navitia/bragi:latest

If you want to get the commit ref used to build the latest image you can run the following command:

docker inspect --format='{{index .Config.Labels "org.label-schema.vcs-ref"}}' navitia/mimirsbrunn:latest

Manually

You can build Mimirsbrunn manually, as the following instructions explain.

You need to retrieve the project and build it using the rust compiler:

git clone https://github.com/hove-io/mimirsbrunn.git
cd mimirsbrunn
cargo build --release

Running

Having built the project, you can now perform a sanity check. This will serve as a test that the program works, and as a basic example of the two main functionalities: indexing and querying.

So we'll index administrative regions into Elasticsearch, and then search for some of them. We'll focus on one country, lets say... Denmark!

1. Download OSM file.

First we download the OSM file for Denmark, from the geofabrik server, and store the file locally.

2. Generate cosmogony file.

If you haven't installed cosmogony yet, you need to do so now, by following the instructions here. You can then transform the original OSM PBF file for Denmark (The following command must be typed in the directory of the cosmogony project)

cargo run --release -- \
  generate \
  --input /path/to/denmark-latest.osm.pbf \
  --output denmark.jsonl.gz

3. Start an Elasticsearch docker container.

We'll start an Elasticsearch version 7.13, available on ports 9200 and 9300.

docker run --name mimirsbrunn-test \
  -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
  -d docker.elastic.co/elasticsearch/elasticsearch:7.13.0

You can check that Elasticsearch has correctly started (maybe wait for about 10/20s for Elasticsearch to be available):

curl 'http://localhost:9200'
{
  "name": "...",
  "cluster_name": "docker-cluster",
  ...
k  "tagline": "You Know, for Search"
}

4. Prepare Elasticsearch

We use templates, such that if you create an index that starts with a certain prefix, if a templates is configured for that prefix, it will create the index with the values found in that template. So, before creating any index, you need to import all the templates in Elasticsearch.

cargo run --bin ctlmimir -- \
  --setting "elasticsearch.url='http://localhost:9200'" \
  --config-dir ./config \
  --run-mode testing \
  run

You should see all the index templates by querying your Elasticsearch. For example, the template for administrative regions:

curl 'http://localhost:9200/_index_template/mimir-admin*'
{
  "index_templates": [
    {
      "name": "mimir-admin",
      "index_template": {
        "index_patterns": [
          "munin_admin*"
        ],
        ...
        "composed_of": [
          "mimir-base",
          "mimir-dynamic-mappings"
        ],
        "priority": 10,
        "version": 3
      }
    }
  ]
}

5. Index cosmogony into Elasticsearch

The result of building the mimirsbrunn project includes several binaries located in /target/releases, one of which is used to index cosmogony files:

cosmogony2mimir uses several configuration files found in the source code, and they work fine by default. In the following command, we use a setting to make sure that cosmogony2mimir will target the Elasticsearch container we just started: (See here for more details about using cosmogony2mimir)

cargo run --bin cosmogony2mimir -- \
  --config-dir ./config \
  --run-mode testing \
  --setting "elasticsearch.url='http://localhost:9200'" \
  --setting langs=['en', 'da'] \
  --input <path/to/denmark.jsonl.gz> \
  run

You can follow in the mimirsbrunn/logs directory.

5. Check Elasticsearch

The previous step created an index for administrative regions, and so you should be able to query your Elasticsearch like so.

curl 'http://localhost:9200/_cat/indices'
health status index                                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   munin_admin_fr_20211104_152535_346903898 FrWbs7PiRi26w-cbsIXjbg   1   4       1841            0     16.3mb         16.3mb

6. Start Bragi

Now you can start bragi.

cargo run --bin bragi -- \
  --config-dir ./config \
  --setting "elasticsearch.url='http://localhost:9200'" \
  --run-mode testing \
  run

7. Query Bragi

You can now query bragi for a city that contains 'hay' (like 'L'Haÿ-les-Roses' for example).

http "localhost:5000/api/v1/autocomplete?q=hay&type[]=city"

Testing

Since this is a rust project, we are well instrumented to run all sorts of tests:

• style
• lint
• unit tests
• end to end / integration.
• benchmark

You can run them all at once, and this in the way it is carried out in the CI pipeline, with

make check

See this page for a more in depth introduction to testing this project.

Where to Go from There

Maybe you find that some the results you get are not ranked correctly, and want to adjust the way Elasticsearch is configured. So you may want to learn how Elasticsearch templates are configured and how to get there.

You want to know more about indexing data into Elasticsearch.

You want to know more about bragi and how to query Elasticsearch via mimir, start here.

Development

You can find more developer oriented documentation here

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use Semantic Versioning for versioning. For the versions available, see the tags on this repository.

Authors

Mimirsbrunn is a project initially started by Guillaume Pinot and Antoine Desbordes for Navitia.

See also the list of contributors who participated in this project.

Processes

processes

Who Uses It ?

• Navitia
• Qwant Maps

If you use it too, feel free to open a pull request, we'll be happy to add your project here!

Alternatives

• pelias
• photon
• addok

TODO: add a bit more detail on all the projects

All those projects use quite the same APIs, and you can compare their results using geocoder-tester.

For a more visual comparison, you can also use a comparator.

Resources

• A french presentation of Mimirsbrunn

License

This project is licensed under the AGPLv3 GNU Affero General Public License - see the LICENSE.md file for details

Acknowledgments

• Billie Thompson - Provided README Template - PurpleBooth