cancel
Showing results for 
Search instead for 
Did you mean: 

Neo4j Connector with Spark

deepika
Node Link

I want to load the data into neo4j using spark. I have installed apache spark and adding neo4j connector using python code.

df.write.format("org.neo4j.spark.DataSource")
.option("url","url" )
.option("authentication.type", "basic").option("authentication.basic.username", "xx")
.option("authentication.basic.password","xxxx")
.option("query", "MERGE (city:City{locationId:event.LOCALITY_ID}) ON CREATE SET city.city = event.LOCALITY_NAME")
.mode("Overwrite")
.save()

I am using neo4j apache connector.
neo4j-connector-apache-spark_2.12-4.0.1_for_spark_3.jar

However when i am executing , i am getting below error.

21/08/06 11:29:48 ERROR SchemaService: Query not compiled because of the following exception:
org.neo4j.driver.exceptions.ClientException: Variable event not defined (line 2, column 29 (offset: 71))
"MERGE (city:City{locationId:event.LOCALITY_ID}) ON CREATE SET city.city = event.LOCALITY_NAME"

Can some one assist here.

2 REPLIES 2

poem_daga
Node Link

Hello,

I am facing the same issue while executing following Query:

MERGE (account:AccountSS {entityKey: event.entityKey, communityId: event.communityId, appId: event.appId, nativeType: event.nativeType})   MERGE (ap:DBAccessProfileSS {dataCollection: event.dataCollection, name: event.name, description: event.description, displayName: event.displayName})  MERGE (ap)<-[:IS_RELATED_TO_SS]-(account)
in the middle of logs

21/08/19 15:20:11 ERROR SchemaService: Query not compiled because of the following exception:
org.neo4j.driver.exceptions.ClientException: Variable event not defined (line 2, column 38 (offset: 80))
"MERGE (account:AccountSS {entityKey: event.entityKey, communityId: event.communityId, appId: event.appId, nativeType: event.nativeType}) MERGE (ap:DBAccessProfileSS {dataCollection: event.dataCollection, name: event.name, description: event.description, displayName: event.displayName}) MERGE (ap)<-[:IS_RELATED_TO_SS]-(account)"
^
at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
at org.neo4j.driver.internal.InternalResult.blockingGet(InternalResult.java:128)
at org.neo4j.driver.internal.InternalResult.consume(InternalResult.java:117)

Though query is executed successfully at the end. But there are exceptions in the log.

at the end of logs

21/08/19 15:20:12 INFO Neo4jDataWriter: Writing a batch of 11 elements to Neo4j,
for jobId=236e007e-85e5-4103-875c-0bc46f21f60f and partitionId=0
with query: WITH $scriptResult AS scriptResult
UNWIND $events AS event
MERGE (account:AccountSS {entityKey: event.entityKey, communityId: event.communityId, appId: event.appId, nativeType: event.nativeType}) MERGE (ap:DBAccessProfileSS {dataCollection: event.dataCollection, name: event.name, description: event.description, displayName: event.displayName}) MERGE (ap)<-[:IS_RELATED_TO_SS]-(account)
21/08/19 15:20:12 INFO DataWritingSparkTask: Committed partition 0 (task 4, attempt 0, stage 4.0)

deepika
Node Link

Neo4j told this issue is related to the driver. we have used the latest connector jar file and this error is gone.