IllegalStateException when import big OSM file with spatial.importOSM

Hi everyone,

I'm currently working on a project that integrates spatial data into a Neo4j database. Previously, I successfully imported a small OSM file using the spatial.importOSM procedure. The file was obtained via Overpass and corresponds to a specific municipality.

Now, I’m trying to import a larger OSM file (~2 GB) representing a broader region, downloaded from the GeoFabrik website. However, I'm encountering an issue during the import process. The following exception is thrown:

Failed to invoke procedure `spatial.importOSM`: Caused by: java.lang.IllegalStateException: Expected index to come online within a reasonable time.

From the Neo4j logs, I can see the following:

About to create node index
Created index OSM-prov_Alicante_info.osm-OSMNode_70A4C151ADA71E6238BC3DC9F5BBFBB4-node_osm_id
About to populate node index

For context, I'm using Neo4j version 4.14.

Has anyone experienced a similar issue or have suggestions for resolving this? I’d appreciate any help or advice.

Thank you in advance!

I tried to divide the large file into small pieces but i still get IllegalStateException running via Python Script.

```Python

neo4j.exceptions.ClientError: {code: Neo.ClientError.Procedure.ProcedureCallFailed} {message: Failed to invoke procedure `spatial.importOSMToLayer`: Caused by: org.neo4j.graphdb.NotInTransactionException: The transaction has been closed.}

And the logs of the database are the following:

Geometry statistics for 3 geometry types: Point: 1878 LineString: 15445 Polygon: 52 Tag statistics for 4 types: all: TagStats[all]: [ISO3166-2, access, addr:city, addr:housenumber, addr:postcode, addr:street, admin_level, alt_name, alt_name:es, amenity, area, area:highway, barrier, bench, bicycle, bicycle_road, bin, boat, border_type, boundary, brand, bridge, bridge:structure, building, building:levels, cables, capacity, colour, construction, covered, crossing, crossing:island, crossing:markings, crossing:signals, crossing_ref, cycleway:both, cycleway:left, cycleway:oneway, cycleway:right, cycleway:right:oneway, description, destination, destination:ref, direction, fee, flag, foot, footway, ford, frequency, from, golf, hazmat, highway, idee:name, incline, ine:municipio, informal, intermittent, isced:level, junction, kerb, landuse, lane_markings, lanes, layer, leaf_type, leisure, line_attachment, line_management, lit, location, location:transition, material, maxheight, maxspeed, maxspeed:type, maxweight:signed, motor_vehicle, motorcar, mtb:scale:uphill, name, name:ar, name:be, name:be-tarask, name:bg, name:br, name:bs, name:ca, name:cs, name:cy, name:da, name:de, name:el, name:en, name:eo, name:es, name:et, name:etymology:wikidata, name:eu, name:fi, name:fr, name:fy, name:ga, name:gl, name:hr, name:hsb, name:hu, name:hy, name:ia, name:is, name:it, name:ja, name:kk, name:la, name:lb, name:lt, name:lv, name:mi, name:mk, name:nl, name:nn, name:pl, name:pt, name:ro, name:ru, name:sk, name:sl, name:sq, name:sr, name:sv, name:tr, name:uk, name:ur, name:vi, name:zh, name:zh-Hans, name:zh-Hant, natural, network, noexit, noname, official_name, official_name:ar, official_name:be, official_name:ca, official_name:en, official_name:es, official_name:fr, old_ref, oneway, oneway:bicycle, operator, operator:type, operator:wikidata, par, parking, phone, place, playground, population, population:date, power, public_transport, public_transport:version, ref, ref:colour, ref:colour_tx, ref:ine, ref:nuts, religion, restriction, roof:levels, roof:shape, route, segregated, service, shelter, short_name, short_name:ar, short_name:bg, short_name:ca, short_name:cs, short_name:de, short_name:en, short_name:es, short_name:fr, short_name:hu, short_name:lt, short_name:nl, short_name:pl, short_name:pt, short_name:sl, short_name:sr, short_name:sv, short_name:uk, side, sidewalk, smoothness, source, sport, surface, tactile_paving, to, toll, tourism, tracktype, traffic_sign, tunnel, type, usage, voltage, water, waterway, website, wheelchair, width, wikidata, wikipedia] node: TagStats[node]: [barrier, bicycle, crossing, crossing:island, crossing:markings, crossing:signals, crossing_ref, direction, ford, golf, highway, kerb, line_attachment, line_management, location:transition, material, name, noexit, place, power, public_transport, ref, side, tactile_paving, traffic_sign] way: TagStats[way]: [access, admin_level, amenity, barrier, bicycle, border_type, boundary, bridge, building, construction, covered, cycleway:left, cycleway:right, cycleway:right:oneway, destination, fee, foot, footway, golf, hazmat, highway, incline, intermittent, junction, landuse, lane_markings, lanes, layer, leaf_type, leisure, lit, location, maxspeed, name, natural, noname, old_ref, oneway, oneway:bicycle, operator, par, parking, playground, power, ref, ref:colour, ref:colour_tx, segregated, service, sidewalk, smoothness, source, sport, surface, tracktype, tunnel, voltage, water, waterway, wheelchair] relation: TagStats[relation]: [ISO3166-1, ISO3166-1:alpha2, ISO3166-1:alpha3, ISO3166-1:numeric, ISO3166-2, access, addr:city, addr:housenumber, addr:postcode, addr:street, admin_level, alt_name, alt_name:az, alt_name:ba, alt_name:be, alt_name:be-tarask, alt_name:bg, alt_name:br, alt_name:bs, alt_name:ca, alt_name:ce, alt_name:ckb, alt_name:cs, alt_name:cv, alt_name:cy, alt_name:da, alt_name:de, alt_name:dsb, alt_name:en, alt_name:eo, alt_name:es, alt_name:et, alt_name:fi, alt_name:fo, alt_name:fr, alt_name:gl, alt_name:grc, alt_name:gv, alt_name:hi, alt_name:hsb, alt_name:hu, alt_name:id, alt_name:ka, alt_name:kk, alt_name:ku, alt_name:la, alt_name:lez, alt_name:lt, alt_name:lv, alt_name:lzh, alt_name:mk, alt_name:mn, alt_name:mrj, alt_name:ms, alt_name:nn, alt_name:no, alt_name:pa, alt_name:pl, alt_name:ro, alt_name:ru, alt_name:sc, alt_name:sco, alt_name:sh, alt_name:sk, alt_name:sl, alt_name:sr, alt_name:sv, alt_name:tr, alt_name:tt, alt_name:uk, alt_name:uz, alt_name:vi, alt_name:yi, alt_name:yue, amenity, ascent, border_type, boundary, brand, building, building:levels, building:part, coat_of_arms, colour, construction, contact:website, currency, default_language, denomination, descent, description, description:en, destination, diagram, distance, driving_side, flag, from, golf, group, highway, idee:name, ine:ccaa, ine:municipio, ine:provincia, int_name, isced:level, landuse, leaf_type, leisure, name, name:ace, name:af, name:als, name:am, name:an, name:ang, name:ar, name:arc, name:arz, name:ast, name:ay, name:az, name:az-Arab, name:azb, name:ba, name:bar, name:bat-smg, name:bcl, name:be, name:be-tarask, name:bg, name:bi, name:bn, name:bo, name:bpy, name:br, name:bs, name:bug, name:bxr, name:ca, name:cbk-zam, name:cdo, name:ce, name:ceb, name:ch, name:chr, name:chy, name:ckb, name:cnr, name:co, name:crh, name:cs, name:csb, name:cu, name:cv, name:cy, name:da, name:de, name:diq, name:dsb, name:dv, name:dz, name:ee, name:el, name:eml, name:en, name:eo, name:es, name:et, name:eu, name:ext, name:fa, name:fi, name:fo, name:fr, name:frp, name:frr, name:fur, name:fy, name:ga, name:gaceria, name:gag, name:gan, name:gd, name:gl, name:glk, name:gn, name:grc, name:gu, name:gv, name:hak, name:haw, name:he, name:hi, name:hif, name:hr, name:hsb, name:ht, name:hu, name:hy, name:ia, name:id, name:ie, name:ig, name:ik, name:ilo, name:io, name:is, name:it, name:iu, name:ja, name:jam, name:jbo, name:jv, name:ka, name:kaa, name:kab, name:kbd, name:kg, name:kk, name:kk-Arab, name:kl, name:km, name:kn, name:ko, name:koi, name:krc, name:ku, name:kv, name:kw, name:ky, name:la, name:lad, name:lb, name:lez, name:li, name:lij, name:lmo, name:ln, name:lo, name:lt, name:ltg, name:lv, name:lzh, name:mdf, name:mg, name:mhr, name:mi, name:min, name:mk, name:ml, name:mn, name:mr, name:mrj, name:ms, name:mt, name:mwl, name:my, name:mzn, name:na, name:nah, name:nan, name:nap, name:nb, name:nds, name:nds-nl, name:ne, name:new, name:nl, name:nn, name:no, name:nov, name:nrm, name:nv, name:oc, name:or, name:os, name:pa, name:pag, name:pam, name:pap, name:pcd, name:pdc, name:pih, name:pl, name:pms, name:pnb, name:pnt, name:ps, name:pt, name:qu, name:rm, name:rmy, name:rn, name:ro, name:roa-tara, name:ru, name:rue, name:rw, name:sa, name:sah, name:sc, name:scn, name:sco, name:se, name:sh, name:sk, name:sl, name:sm, name:smn, name:sms, name:so, name:sq, name:sr, name:sr-Latn, name:srn, name:ss, name:st, name:stq, name:su, name:sv, name:sw, name:szl, name:ta, name:te, name:tet, name:tg, name:th, name:tk, name:tl, name:to, name:tok, name:tpi, name:tr, name:ts, name:tt, name:ty, name:tzl, name:udm, name:ug, name:uk, name:ur, name:uz, name:vec, name:vep, name:vi, name:vls, name:vo, name:vro, name:wa, name:war, name:wo, name:wuu, name:xal, name:xmf, name:yi, name:yo, name:yue, name:za, name:zea, name:zh, name:zh-Hans, name:zh-Hant, name:zu, natural, network, nohousenumber, note, note:ISO3166-1, note:description, note:name, note:url, official_name, official_name:ar, official_name:be, official_name:ca, official_name:cs, official_name:de, official_name:el, official_name:en, official_name:eo, official_name:es, official_name:fa, official_name:fr, official_name:gaceria, official_name:gl, official_name:hu, official_name:pt, official_name:ur, official_name:zh, official_name:zh-Hans, official_name:zh-Hant, old_name, operator, operator:2021-08-31, operator:type, operator:wikidata, osmc:symbol, parking, phone, pilgrimage, pipeline:type, place, population, population:date, public_transport:version, ref, ref:colour, ref:colour_bg, ref:colour_tx, ref:ine, ref:nuts, ref:nuts:1, ref:nuts:2, ref:nuts:3, religion, restriction, roundtrip, route, route_master, short_name, short_name:an, short_name:ar, short_name:ast, short_name:be, short_name:bg, short_name:bs, short_name:ca, short_name:cnr, short_name:co, short_name:cs, short_name:cy, short_name:da, short_name:de, short_name:el, short_name:en, short_name:eo, short_name:es, short_name:et, short_name:eu, short_name:ext, short_name:fi, short_name:fr, short_name:frp, short_name:fy, short_name:ga, short_name:gl, short_name:gn, short_name:grc, short_name:hr, short_name:hu, short_name:ia, short_name:is, short_name:it, short_name:kbp, short_name:kk, short_name:lb, short_name:li, short_name:lt, short_name:lv, short_name:mi, short_name:mk, short_name:mt, short_name:mwl, short_name:nap, short_name:nl, short_name:pa, short_name:pcd, short_name:pl, short_name:pt, short_name:ro, short_name:ru, short_name:sc, short_name:sk, short_name:sl, short_name:sr, short_name:sv, short_name:tr, short_name:tt, short_name:uk, short_name:vls, short_name:wa, short_name:zea, short_name:zh, short_name:zh-Hans, short_name:zh-Hant, social_facility, social_facility:for, source, source:ISO3166-1, source:name, source:name:oc, source:note:name, source_ref, sport, to, toll:conditional, type, waterway, website, wikidata, wikipedia] ./import/provAlicante_1_1.osm[1059294]: Re-indexing with GraphDatabaseService: community single [DatabaseLayout{databaseDirectory=/var/lib/neo4j/data/databases/neo4j, transactionLogsDirectory=/var/lib/neo4j/data/transactions/neo4j}] (class: class org.neo4j.kernel.impl.factory.GraphDatabaseFacade) Mismatching vertices size for Polygon:Node[198394]: 1 != 2 Mismatching vertices size for LineString:Node[199079]: 2 != 4 Mismatching vertices size for LineString:Node[199676]: 27 != 44 Mismatching vertices size for LineString:Node[199715]: 2 != 8 Mismatching vertices size for LineString:Node[199720]: 4 != 10 Mismatching vertices size for LineString:Node[199740]: 2 != 12 Mismatching vertices size for LineString:Node[199745]: 8 != 11 Mismatching vertices size for LineString:Node[199763]: 13 != 14 Mismatching vertices size for LineString:Node[199779]: 4 != 6 Mismatching vertices size for LineString:Node[199786]: 4 != 6 Mismatching vertices found 100 times Mismatching vertices found 200 times

My docker-compose.yml is the following:

services:
  neo4j:
    image: neo4j_custom_image
    container_name: neo4j_bd     
    restart: no
    volumes:
      - "./data:/var/lib/neo4j/data"
      - "./plugins:/var/lib/neo4j/plugins:rw"
      - "./import:/var/lib/neo4j/import:rw"
    environment:Y
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4J_dbms_security_procedures_unrestricted=apoc.*,spatial.*
      - NEO4J_dbms_security_procedures_allowlist=apoc.*,spatial.*
      - NEO4J_apoc_import_file_enabled=true
      - NEO4J_dbms_memory_heap_initial__size=25G
      - NEO4J_dbms_memory_heap_max__size=25G
      - NEO4J_dbms_memory_pagecache_size=60G
      - NEO4J_dbms_jvm_additional='-XX:+ExitOnOutOfMemoryError'
      - NEO4J_dbms_transaction_timeout=10d
      - NEO4J_dbms_lock_acquisition_timeout=10d
      - NEO4J_dbms_transaction_bookmark__ready__timeout=30d                    
    build: .

Any suggestion or what could be wrong?