Batchimport exception - Missing group

Neo4j Graph Platform ETL-Tool
1

Howdy all,

I'm trying to import data from Postgres, and I seem to be hitting the following error:

 Stderr: 'org.neo4j.internal.batchimport.input.HeaderException: Group 'public._product_collection_filters' not found
. Available groups are: [ <redacted for brevity>

This is obviously caused by the table _product_collection_filters. At first I thought it might be a strange case of snake case to camelcase or similar, but the only references to product_collection_filters of any kind are in:

  • the mapping.json, within the sql it generates, under a table key, under an id-space key, and under a name key (as PRODUCT_COLLECTION_FILTERS)
  • the headers file (REL_PRODUCTCOLLECTIONFILTERS_34d0ff28-4caf-4698-9a4d-edf456417552_headers.csv), in the END_ID header - :END_I D(public._product_collection_filters)
  • the actual data csv export, in what would be the :TYPE column, with value PRODUCT_COLLECTION_FILTER

The command I'm running to run the mapping is:

      ${etl_tool} generate-metadata-mapping \
        --rdbms:url ${RDBMS_URL} \
        --rdbms:user ${RDBMS_USER} \
        --rdbms:password ${RDBMS_PASSWORD} \
        --output-mapping-file /data/mapping.json
        --exclusion-mode-tables ${TABLES_EXCLUSION_MODE} \
        --tables \"-l ${TABLES}\"

and the export command is:

      ${etl_tool} export \
        --rdbms:url ${RDBMS_URL} \
        --rdbms:user ${RDBMS_USER} \
        --rdbms:password ${RDBMS_PASSWORD} \
        --neo4j:user ${NEO4J_USER} \
        --neo4j:password ${NEO4J_PASSWORD} \
        --using bulk:neo4j-import \
        --import-tool ${NEO4J_HOME}/bin \
        --csv-directory /data/csv \
        --options-file /data/options.json \
        --mapping-file /data/mapping.json \
        --destination ${NEO4J_HOME}/data/databases/neo4j/ \
        --exclusion-mode-tables ${TABLES_EXCLUSION_MODE} \
        --tables \"-l ${TABLES}\"
        --quote '"' \
        --force

I have saved a copy of the CSV data & mapping and the command that the etl tool generates to import the CSVs. I'm not sure where the logs are saved to have those available; I've looked at /var/lib/neo4j/import/import.report and /var/lib/neo4j/logs but there's nothing there.

Additional info:

  • Neo4j version v4.0.4 (Docker)
  • ETL tool version v1.5.0 (I've packaged this into the same Docker image)
  • Importing from Postgres

Also, I wanted to try excluding this table but it looks like the mapping & export commands don't seem to be respecting the table inclusions/exclusions I've provided. Not sure if it's worth creating an issue or new post for this?

2

It would be good to get two GH issues for those.

What it's looking for is a node-import csv file that has public._product_collection_filters as it's group (in round parens after :ID.

Really odd, both of them.