Hi all, I'm trying to use neo4j admin import to ingest parquet files. In my test import, i used the simple example given by the tutorial at Neo4j-admin import - Operations Manual
import pyarrow as pa
import pyarrow.parquet as pq
# Movies
movies_table = pa.table({
'movieId:ID': ['tt0133093', 'tt0234215', 'tt0242653'],
'title': ['The Matrix', 'The Matrix Reloaded', 'The Matrix Revolutions'],
'year': [1999, 2003, 2003],
':LABEL': ['Movie', 'Movie;Sequel', 'Movie;Sequel']
})
pq.write_table(movies_table, '/home/ubuntu/neo4j/import/movies.parquet')
# Actors
actors_table = pa.table({
'personId:ID': ['keanu', 'laurence', 'carrieanne'],
'name': ['Keanu Reeves', 'Laurence Fishburne', 'Carrie-Anne Moss'],
':LABEL': ['Actor', 'Actor', 'Actor']
})
pq.write_table(actors_table, '/home/ubuntu/neo4j/import/actors.parquet')
# Relationships
roles_table = pa.table({
':START_ID': ['keanu', 'keanu', 'keanu', 'laurence', 'laurence', 'laurence', 'carrieanne', 'carrieanne', 'carrieanne'],
':END_ID': ['tt0133093', 'tt0234215', 'tt0242653', 'tt0133093', 'tt0234215', 'tt0242653', 'tt0133093', 'tt0234215', 'tt0242653'],
':TYPE': ['ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN', 'ACTED_IN']
})
pq.write_table(roles_table, '/home/ubuntu/neo4j/import/roles.parquet')
and tried to ingest these three parquet files on my server using
docker run --rm \
-v {$HOME}/neo4j/data:/data \
-v {$HOME}/neo4j/import:/import \
-v {$HOME}/neo4j/logs:/logs \
neo4j:2025.03.0-community-bullseye \
neo4j-admin database import full --overwrite-destination --input-type=parquet --verbose neo4j \
--nodes /import/movies.parquet \
--nodes /import/actors.parquet \
--relationships /import/roles.parquet
But always got the NullPointerException
Executing command line: /opt/java/openjdk/bin/java -cp /var/lib/neo4j/plugins/*:/var/lib/neo4j/conf/*:/var/lib/neo4j/lib/* -XX:+UseParallelGC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.nio.maxCachedBufferSize=1024 -Dio.netty.tryReflectionSetAccessible=true -XX:+ExitOnOutOfMemoryError -Djdk.tls.ephemeralDHKeySize=2048 -XX:FlightRecorderOptions=stackdepth=256 -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --enable-native-access=ALL-UNNAMED -Dlog4j2.disable.jmx=true -Dfile.encoding=UTF-8 org.neo4j.cli.AdminTool database import full --overwrite-destination --input-type=parquet --verbose neo4j --nodes /import/movies.parquet --nodes /import/actors.parquet --relationships /import/roles-pq.parquet
neo4j 2025.03.0
VM Name: OpenJDK 64-Bit Server VM
VM Vendor: Eclipse Adoptium
VM Version: 21.0.7+6-LTS
JIT compiler: HotSpot 64-Bit Tiered Compilers
VM Arguments: [-XX:+UseParallelGC, -XX:-OmitStackTraceInFastThrow, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.nio.maxCachedBufferSize=1024, -Dio.netty.tryReflectionSetAccessible=true, -XX:+ExitOnOutOfMemoryError, -Djdk.tls.ephemeralDHKeySize=2048, -XX:FlightRecorderOptions=stackdepth=256, -XX:+UnlockDiagnosticVMOptions, -XX:+DebugNonSafepoints, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --enable-native-access=ALL-UNNAMED, -Dlog4j2.disable.jmx=true, -Dfile.encoding=UTF-8]
Configuration files used (ordered by priority):
/var/lib/neo4j/conf/neo4j-admin.conf
/var/lib/neo4j/conf/neo4j.conf
--------------------
Starting to import, output will be saved to: /logs/neo4j-admin-import-2025-05-19.13.05.42.log
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:233)
at java.base/java.util.KeyValueHolder.<init>(KeyValueHolder.java:61)
at java.base/java.util.Map.entry(Map.java:1721)
at org.neo4j.internal.batchimport.input.parquet.ParquetInput.lambda$new$7(ParquetInput.java:114)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at org.neo4j.internal.batchimport.input.parquet.ParquetInput.<init>(ParquetInput.java:120)
at org.neo4j.importer.FileImporter.importInput(FileImporter.java:202)
at org.neo4j.importer.FileImporter.doImport(FileImporter.java:181)
at org.neo4j.importer.ImportCommand$Base.doExecute(ImportCommand.java:525)
at org.neo4j.importer.ImportCommand$Full.execute(ImportCommand.java:806)
at org.neo4j.cli.AbstractAdminCommand.wrappedExecute(AbstractAdminCommand.java:207)
at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:97)
at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:37)
at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
at picocli.CommandLine.execute(CommandLine.java:2174)
at org.neo4j.cli.AdminTool.execute(AdminTool.java:94)
at org.neo4j.cli.AdminTool.main(AdminTool.java:82)
which is not debug-able and I honestly had no clue. But if I leave out the relationships parquet and only ingest the nodes, it actually worked, so my guess is that there is something wrong with the relationships parquet file.
docker run --rm \
-v {$HOME}/neo4j/data:/data \
-v {$HOME}/neo4j/import:/import \
-v {$HOME}/neo4j/logs:/logs \
neo4j:2025.03.0-community-bullseye \
neo4j-admin database import full --overwrite-destination --input-type=parquet --verbose neo4j \
--nodes /import/movies.parquet \
--nodes /import/actors.parquet
Executing command line: /opt/java/openjdk/bin/java -cp /var/lib/neo4j/plugins/*:/var/lib/neo4j/conf/*:/var/lib/neo4j/lib/* -XX:+UseParallelGC -XX:-OmitStackTraceInFastThrow -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.nio.maxCachedBufferSize=1024 -Dio.netty.tryReflectionSetAccessible=true -XX:+ExitOnOutOfMemoryError -Djdk.tls.ephemeralDHKeySize=2048 -XX:FlightRecorderOptions=stackdepth=256 -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --enable-native-access=ALL-UNNAMED -Dlog4j2.disable.jmx=true -Dfile.encoding=UTF-8 org.neo4j.cli.AdminTool database import full --overwrite-destination --input-type=parquet --verbose neo4j --nodes /import/movies.parquet --nodes /import/actors.parquet
neo4j 2025.03.0
VM Name: OpenJDK 64-Bit Server VM
VM Vendor: Eclipse Adoptium
VM Version: 21.0.7+6-LTS
JIT compiler: HotSpot 64-Bit Tiered Compilers
VM Arguments: [-XX:+UseParallelGC, -XX:-OmitStackTraceInFastThrow, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.nio.maxCachedBufferSize=1024, -Dio.netty.tryReflectionSetAccessible=true, -XX:+ExitOnOutOfMemoryError, -Djdk.tls.ephemeralDHKeySize=2048, -XX:FlightRecorderOptions=stackdepth=256, -XX:+UnlockDiagnosticVMOptions, -XX:+DebugNonSafepoints, --add-opens=java.base/java.nio=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --enable-native-access=ALL-UNNAMED, -Dlog4j2.disable.jmx=true, -Dfile.encoding=UTF-8]
Configuration files used (ordered by priority):
/var/lib/neo4j/conf/neo4j-admin.conf
/var/lib/neo4j/conf/neo4j.conf
--------------------
Starting to import, output will be saved to: /logs/neo4j-admin-import-2025-05-19.13.13.36.log
Neo4j version: 2025.03.0
Importing the contents of these files into /data/databases/neo4j:
Nodes:
/import/movies.parquet
/import/actors.parquet
Available resources:
Total machine memory: 1008GiB
Free machine memory: 571.8GiB
Max heap memory : 26.67GiB
Max worker threads: 10
Configured max memory: 490.8GiB
High parallel IO: true
Nodes, started 2025-05-19 13:13:38.311+0000
[*Nodes:?? 514.0MiB---------------------------------------------------------------------------] 0 ∆ 0
Done in 513ms
Prepare node index, started 2025-05-19 13:13:38.873+0000
[*:514.0MiB-----------------------------------------------------------------------------------] 0 ∆ 0
Done in 92ms
Relationships, started 2025-05-19 13:13:38.969+0000
[*Relationships:?? 514.0MiB-------------------------------------------------------------------] 0 ∆ 0
Done in 11ms
Node Degrees, started 2025-05-19 13:13:38.995+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 1ms
Relationship --> Relationship 1/1, started 2025-05-19 13:13:39.008+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
RelationshipGroup 1/1, started 2025-05-19 13:13:39.010+0000
[*>:??-----------------------------------------------|>---------------------------------------] 0 ∆ 0
Done in 2ms
Node --> Relationship, started 2025-05-19 13:13:39.016+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 3ms
Relationship <-- Relationship 1/1, started 2025-05-19 13:13:39.023+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
Count groups, started 2025-05-19 13:13:39.030+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
Gather, started 2025-05-19 13:13:39.033+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
Write, started 2025-05-19 13:13:39.036+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
Node --> Group, started 2025-05-19 13:13:39.039+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
Node counts and label index build, started 2025-05-19 13:13:39.336+0000
[*>-----------------------------------------------------------------|LABEL INDEX-----|COUNT:51]10.0K ∆10.0K
Done in 10ms
Relationship counts and relationship type index build, started 2025-05-19 13:13:39.371+0000
[*>:??----------------------------------------------------------------------------------------] 0 ∆ 0
Done in 0ms
IMPORT DONE in 1s 124ms.
Imported:
6 nodes
0 relationships
15 properties
Peak memory usage: 514.0MiB
Could someone kindly enlighten me on where I made a mistake?