Impossible to export from embedded database

ontologiae · September 28, 2021, 5:28pm

neo4j 3.5 (The version 4 never worked in embedded context, it does not creates links between nodes... Tried everything....)

Hi,
I have a huge database with lots of differents type of nodes and relationships.
I'm trying desperately to export my database to an importable file, whatever it is, csv, cypher, I don't care, I just want it is importable by a standalone server.

I tried a LOT of things :

Impossible to registers apoc procedure, so I directly used the method inside APOC package and did this :

 def testExport(db: GraphDatabaseService) = {

    val myMap    = Map("format" -> "plain".asInstanceOf[Object])
    val exporter = new ExportCypher(db)
    //val exporterCSV = new ExportCSV(db) // other experiment
    //val s        = exporter.query("match (n) return n", "test.cypher", myMap.asJava) // other experiment
    val s        = exporter.all( "test.cypher", myMap.asJava)
    println(s.iterator.asScala.toList)
  }

I ALWAYS had

[error] (run-main-0) org.neo4j.graphdb.NotFoundException: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error] org.neo4j.graphdb.NotFoundException: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error]         at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
[error]         at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
[error]         at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
[error]         at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
[error]         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
[error]         at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
[error]         at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
[error]         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:563)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.buildStatementForNodes(AbstractCypherFormatter.java:134)
[error]         at apoc.export.cypher.formatter.CreateCypherFormatter.buildStatementForNodes(CreateCypherFormatter.java:18)
[error]         at apoc.export.cypher.formatter.CreateCypherFormatter.statementForNodes(CreateCypherFormatter.java:57)
[error]         at apoc.export.cypher.MultiStatementCypherSubGraphExporter.exportNodesUnwindBatch(MultiStatementCypherSubGraphExporter.java:131)
[error]         at apoc.export.cypher.MultiStatementCypherSubGraphExporter.export(MultiStatementCypherSubGraphExporter.java:106)
[error]         at apoc.export.cypher.ExportCypher.doExport(ExportCypher.java:127)
[error]         at apoc.export.cypher.ExportCypher.exportCypher(ExportCypher.java:116)
[error]         at apoc.export.cypher.ExportCypher.all(ExportCypher.java:51)
[error]         at org.deepanalyzer.GraphUtil$.testExport(GraphUtil.scala:252)
[error]         at org.deepanalyzer.GraphBuilder$.calculHotSpotWalkInGraph(GraphBuilder.scala:194)
[error]         at org.deepanalyzer.GraphBuilder$.genereGraph(GraphBuilder.scala:541)
[error]         at org.deepanalyzer.Main$.main(Main.scala:128)
[error]         at org.deepanalyzer.Main.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at org.neo4j.kernel.impl.core.NodeProxy.singleNode(NodeProxy.java:836)
[error]         at org.neo4j.kernel.impl.core.NodeProxy.getLabels(NodeProxy.java:657)
[error]         at apoc.export.util.FormatUtils.getLabelsAsStream(FormatUtils.java:102)
[error]         at apoc.export.util.FormatUtils.getLabelsSorted(FormatUtils.java:98)
[error]         at apoc.export.cypher.formatter.CypherFormatterUtils.getNodeIdProperties(CypherFormatterUtils.java:58)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.lambda$buildStatementForNodes$5(AbstractCypherFormatter.java:127)
[error]         at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
[error]         at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:563)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[error]         at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
[error]         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
[error]         at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
[error]         at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
[error] Caused by: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at org.neo4j.kernel.impl.core.NodeProxy.singleNode(NodeProxy.java:836)
[error]         at org.neo4j.kernel.impl.core.NodeProxy.getLabels(NodeProxy.java:657)
[error]         at apoc.export.util.FormatUtils.getLabelsAsStream(FormatUtils.java:102)
[error]         at apoc.export.util.FormatUtils.getLabelsSorted(FormatUtils.java:98)
[error]         at apoc.export.cypher.formatter.CypherFormatterUtils.getNodeIdProperties(CypherFormatterUtils.java:58)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.lambda$buildStatementForNodes$5(AbstractCypherFormatter.java:127)
[error]         at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
[error]         at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:563)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[error]         at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
[error]         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
[error]         at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
[error]         at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Precision : my code only add node and never remove them...
Precision2 : All the transactions are closed and another is launch to make the export

When I close the database and reopen it with newEmbeddedDatabase (I'm desperate, I try...) I get
throw java.lang.RuntimeException: Export to files not enabled, please set apoc.export.file.enabled=true in your neo4j.conf because with newEmbeddedDatabase I can't define config parameter

How can I give it the right config items ?
I tried a CALL dbms.setConfigValue('apoc.export.file.enabled', 'true') inside but I get
throw org.neo4j.graphdb.QueryExecutionException: There is no procedure with the name dbms.setConfigValue registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

The exportCSV work (why ??) but how I can import a single CSV file, I can do nothing with this

So, do you have, please, any solutions ?

In advance Thank you

michael.hunger · September 28, 2021, 8:29pm

The easiest thing to do is the following.

shut down your application
take the database folder and copy it into a server install
start up the server to make sure that the db is up and running, potentially need to configure active database
shut down server
do neo4j-admin dump --database --to file.dump

then you can use that dump in other contexts e.g. importing it into a 4.x database, uploading to aura, use in desktop, seed a docker instance etc.

ontologiae · September 28, 2021, 9:49pm

Hi Michael,
It's precisely the process I'm doing for 2 years now and I would like to avoid that, because I have 80 neo4j different database to build..
What I don't understand is why I lost data when I shut down the database. When I reopen it, there's only 5% of the nodes that remaining. I carefully verify all the transactions

ontologiae · September 28, 2021, 10:10pm

So I tried the old good method you describe, and guess what, 95% of the data disapeared...

michael.hunger · September 28, 2021, 11:53pm

But you copy the whole database folder which includes all neostore.transaction.* files?

And you shut your embedded application cleanly down before doing that?

ontologiae · September 29, 2021, 7:32am

It's exactly what I did : I copy the whole directory, I even tried to define the directory as neo4j's server directory, and I always get 5% of the data remaining.
I shutdown programmatically the database before

ontologiae · September 29, 2021, 3:59pm

Michael, sorry, I missed a tx.close somewhere, I looked at everywhere but forgot just one.
And now, it works !
Thank you !

For the record the code needed to makes it work :

new GraphDatabaseFactory()
      .newEmbeddedDatabaseBuilder(new java.io.File(path))
      .setConfig(GraphDatabaseSettings.read_only, "true")
      .setConfig("apoc.export.file.enabled", "true")
      .setConfig("dbms.directories.import", "/home/path/")
      .setConfig("dbms.security.procedures.whitelist", "apoc.*")
      .newGraphDatabase()

val proceduresService = db.asInstanceOf[GraphDatabaseAPI].getDependencyResolver().resolveDependency(classOf[Procedures]);
    proceduresService.registerProcedure(classOf[apoc.export.cypher.ExportCypher])

Topic		Replies	Views
Apoc.export.csv.all prepending 'import\' to specified filename, causing failure Neo4j Graph Platform apoc	1	600	October 9, 2019
Apoc.load.csv in embedded db is not working General apoc , cypher	3	296	October 29, 2021
Export and Import CSV Procedures & APOC	6	1491	February 2, 2022
Enabling apoc export? Procedures & APOC apoc , configuration , export	3	578	July 12, 2023
Unable to execute APOC procedure on embedded graph database Other Databases & ETL	1	380	December 21, 2020

Impossible to export from embedded database

Related topics