Spark connector with reactive neo4j driver

Neo4j Graph Platform Import / Export
1

Hello,
Neo4j vers. 4.4.5 community, Spark 3.3 in Azure Databricks.

My intention is to use Spark to export data from Neo to external data source.
Amount of data in Neo4j might be considered as large.

Question: which neo4j driver is used by spark connector? Can I control it to use reactive driver?

Currently my neo4j often crashes at pretty random places.

Queries are like below, but some of them are a bit more complex (getting values from 3rd level of node)

data= spark.read.format("org.neo4j.spark.DataSource") \
    .option("query", """
        MATCH (p:Node1)
        MATCH (p)-[:RELATION]->(m:Node2)
        RETURN p.id  AS pId, m.id as mId
    """)

maybe there could be better approach applied to export such data and use in Spark?

2

Hello, the Spark connector uses the "regular"/synchronous driver APIs, and this is not something that can be changed.

Can you share the logs of such crashes? This is likely a symptom of a different problem.

3

Hello and thank you for confirming my suspections about spark connector.
As I do kubectl logs to neo4j pod, I do not see any logs. I will provide such in next week (the devops guy will be then).

I will give more details about process. We have set of queries like in main post which work without any filter or with small filters on node properties.
Neo4j runs with 3GB memory limit 1.0 core.
Number of "central" Node objects is 1,7milion, this node is linked to other nodes and so on.

We had one big query working on relatively small number of "central" nodes but was more in a "Cypher style", meaning many matches and optional matches, With statements and so on.

After removing that big query from pipeline, neo4j stopped crashing but became 3 times slower.

We noticed that we can use apoc.export.csv.data to export data. When exporting whole DB to single file, there was no pick on memory chart and process completed quickly.
When using that with queries like in first post, there was memory pick.
Do you know if this plugin is fully memory optimized also when exporting data with queries?

For sure I will perform more tests and let you know