Spatial Versions Save

Ported to Neo4j 4.4.3

This version of Neo4j changed many dependencies and was built with JDK17. We were not able to use JDK17 to build Spatial but did need to upgrade many dependencies to get things working.

One particularly big upgrade was the move from Junit4 to Junit5 which required changes to every single test class.

The lastest version of Geotools does not support JDK17 and so we will hold off upgrading completely to JDK17 before Geotools has completed their port.

Ported to Neo4j 4.3

Neo4j 4.3 has an issue with leaking RelationshipTraversalCursor, and we needed to do some workarounds to avoid this issue, usually by exhausting the iterator, which can have a higher performance cost in some cases. We updated the version number to 0.28.1 to reflect these bug-fixes / workarounds.

Version 0.28 begins work on trying to deal work multiple OSM imports. For this release, we disallow importing the same OSM file twice, unless you specify two separate layers, and we also ensure that the indexes are not mixed up between the layers, something that was the case in all previous releases.

To achieve this, two main changes were made:

• Use unique labels for indexing within multiple OSM layers. Essentially when adding nodes of a particular label, we also add a label with a unique name for indexing purposes. The unique name is made of the original name plus a hex suffix made of the MD5 hash of the layer name. For example, the layer geom1 will have an MD5 hash of its name 9ECE5459EA0D46FC556E5E3F454A0795. Then when adding an OSM node we label the node with both:

  • OSMNode
  • OSMNode_9ECE5459EA0D46FC556E5E3F454A0795

• Remove use of old reference nodes, reducing deadlocks. This change is not backwards compatible, as the spatial model is different. Layers created in earlier versions would not be readable in this one. To protect against that we throw exceptions on all SpatialDatabaseService. layer finding methods, if they detect the old format. The exception requests that the database be upgraded. This can be done in the Java API with the upgradeFromOldModel method, or via procedures using spatial.upgrade.

Port of the 0.27 branch to Neo4j 4.2. This is a smaller change than the port to 4.1 and a much smaller change that that to 4.0. Please read the release notes of those releases to understand the full scope of the changes.

For 4.2, the change was more subtle. Mostly only internal API's around the use of Path instead of File. One change that could be noticed by users is the IndexManager.IndexAccessMode class. In the 0.27.0 and 0.27.1 versions, we used OverridenAccessMode to take the users existing access mode and simply add on the rights to create tokens and indexes. In 0.27.2 we instead use RestrictedAccessMode to restrict the users access right to the built-in AccessModel.Static.SCHEMA and then boost to enable index and token writes. The difference is subtle and should only be possible to notice in Enterprise Edition.

Port to Neo4j 4.1.7. This is very much like the port to 4.0, and involves the same changes and considerations. Please read the release notes for that release for a more complete view of the extensive changes required to port to Neo4j 4.x.

In addition, some smaller changes were required to support 4.1.7. Mostly some internal API changes:

• Transaction type names
• IndexSeek API change
• Stricter checks on parameters containing Node objects

This last change means that if we return a Node from one transaction, we cannot pass it into a spatial procedure in another transaction. Users have to either lookup the node again in the new transaction, or use some new procedures we provide here:

• spatial.addNode.byId
• spatial.addNodes.byId
• spatial.removeNode.byId
• spatial.removeNodes.byId

Also the removeNode procedures now return nodeId instead of node

Since this library was originally written in 2010 when Neo4j was still releasing early 1.x versions, it made use of internal Java API's that were deprecated over the years, some as early as Neo4j 2.x. When Neo4j 4.0 was released, it entirely removed a number of deprecated API's, and completely changed the Transaction API. This meant that the spatial library needed a major refactoring to work with Neo4j 4.x:

• The library previously depended on outer and inner transactions, so code that was called within a transaction (like procedures) could be common with code that was not (Java API). The removal of this support required that all internal API's needed to include parameters for the current transaction, and only the specific surface designed for embedded use not.
• The library made use of Lucene based explicit indexes in many places. The removal of support for explicit indexes required new solutions in sevaral places:
  • The OSMImporter used instead normal Neo4j schema indexes (introduced in 2.0), but these can only be created in separate index transactions. Due to the new transaction model this required stopping the import transaction, starting an index transaction, and then restarting the import transaction. All of this is incompatible with procedures, which already have an incoming, non-stoppable transaction. The solution was to run the actual import in another thread, retaining the original batch-processing capabilities, but requiring modifying the security model of the procedure context.
  • The ExplicitIndexBackedPointIndex needed to be modified to instead use a schema index. This required similar tricks as employed in the OSMImporter.
• Neo4j 4.0 runs only in Java 11, and GeoTools only supported up to Java 1.8 until recently. This required an upgrade to the GeoTools and JTS libraries to version 24.2. This in turn required a re-write of the Neo4jDataStore interface since the older API had long been deprecated, and was entirely unavailable in newer versions.

Consequences of this port:

• The large number of changes mean that the 0.27.0 version should be considered very alpha.
• Many API's have changed and client code might need to be adapted to take the changes into account.
• The new GeoServer API is entirely untested, besides the existing unit and integration tests.
• The need to manage threads and create schema indexes results in the procedures requiring unrestricted access to internal API's of Neo4j.

This last point means that you need to set the following in your neo4j.conf file:

dbms.security.procedures.unrestricted=spatial.*

A small release including support for spatial.removeNode and spatial.removeNodes procedures for removing nodes from layers (and their underlying indexes). These procedures do not delete the nodes.

A small release including support for spatial.removeNode and spatial.removeNodes procedures for removing nodes from layers (and their underlying indexes). These procedures do not delete the nodes.

This release supports Neo4j 3.5. In addition, the primary change introduced in the 0.26.x branch of releases is the addition of a new GeomertyEncoder for point data, called the NativePointEncoder. This can be used as a replacement for the SimplePointEncoder. The SimplePointEncoder encodes points as two double properties on a node, usually 'x' and 'y' or 'longitude' and 'latitude'. The NativePointEncoder works on Neo4j 3.4 and later with the built-in support for Point data types.

All procedures that supported the SimplePointEncoder now have versions for the NativePointEncoder. For example:

CALL spatial.addLayerWithEncoder('geom','NativePointEncoder','pos:mbr')

And then

CREATE (n {pos:point(latitude:13.2,longitude:56.7)}) CALL spatial.addNode('geom',n)

Or perhaps:

CALL spatial.addNativePointLayerGeohash('points')

And then:

CREATE (n {location:point(latitude:13.2,longitude:56.7)}) CALL spatial.addNode('points',n)

The primary change in the release is the addition of a new GeomertyEncoder for point data, called the NativePointEncoder. This can be used as a replacement for the SimplePointEncoder. The SimplePointEncoder encodes points as two double properties on a node, usually 'x' and 'y' or 'longitude' and 'latitude'. The NativePointEncoder works on Neo4j 3.4 and later with the built-in support for Point data types.

All procedures that supported the SimplePointEncoder now have versions for the NativePointEncoder. For example:

CALL spatial.addLayerWithEncoder('geom','NativePointEncoder','pos:mbr')

And then

CREATE (n {pos:point(latitude:13.2,longitude:56.7)}) CALL spatial.addNode('geom',n)

Or perhaps:

CALL spatial.addNativePointLayerGeohash('points')

And then:

CREATE (n {location:point(latitude:13.2,longitude:56.7)}) CALL spatial.addNode('points',n)