cancel
Showing results for 
Search instead for 
Did you mean: 

Can't map postgres to Neo4j graph using Neo4j ETL tool?

steves
Node Clone

Hey there 🙂

I am trying to map my Cloudquery database (postgres) to Neo4j using ETL Tool.

Project created with the following version:

Version 4.4.5

Neoj4 Desktop version 1.4.15.

I am on OSX Monterey 12.5 (21G72), Apple M1 Pro.

For the postgres (where cloudquery is pouring the data) I am using postgres:13.7-alpine3.16 docker image.

Selected RDBMS instance (connection succeeded) and my project. Within the main.log I get:

[2022-07-28 14:27:13.192] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:27:13.844] [info] Process [87889] exit with code '1', signal 'null'
[2022-07-28 14:27:43.264] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:27:43.372] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:28:23.273] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:28:23.382] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:03.278] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:03.386] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:39.909] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:29:40.771] [info] Process [88036] exit with code '0', signal 'null'
[2022-07-28 14:29:43.282] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:43.338] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:44.217] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.NeoIntegrationCli, generate-metadata-mapping, --rdbms:url, jdbc:postgresql://localhost:5432/postgres?ssl=false, --rdbms:password, pass, --rdbms:user, postgres, --output-mapping-file, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.214] [info] Process [88039] exit with code '0', signal 'null'
[2022-07-28 14:30:04.216] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.util.FileUtils, readfile, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.616] [info] Process [88058] exit with code '143', signal 'null'
[2022-07-28 14:30:23.288] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:30:23.596] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:03.292] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:03.344] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:43.296] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:43.459] [info] Online check response: 200 version: 1.4.15

It seems I do get the mapping file but the mapping process takes ages without finishing.

Please advise what should I do to make it work?

Can I upload manually the mapping file? How can I force it to do the mapping? Is it ok it takes so long?

3 REPLIES 3

david_allen
Neo4j
Neo4j

@michael_hunger if you have a chance to look at this when you're back please do; I spoke with @steves by discord.  Doesn't seem to be a clear error here so some debugging questions to gather more information might be a way to proceed.

It is an issue with the posgres JDBC driver that the fetching the metadata for the mapping takes so long, it's not the case for other databases.
It's a know issue that they don't want to fix. 

So you'd need to have patience unfortunately. It might be faster if you have a local copy of the database and fetch the mapping from there.

You cannot upload the mapping file directly, but if you use the command line version of the etl-tool, you can pass in the mapping file as a parameter:

https://neo4j.com/labs/etl-tool/1.5.0/#_export_command

https://neo4j.com/labs/etl-tool/1.5.0/neo4j-etl/#_online_batch_import_via_java_bolt_driver_for_incre...

steves
Node Clone

Thanks a lot @michael_hunger !

Would you be so kind and post an A-Z tutorial or how-to to convert RDBMS to Neo4j graph? Maybe using Hop?

I will be happy to see an example. If it's possible it will help me and others.

Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.