Issue with relationship property creation in Spark submit job: Please provide a valid WRITE query

Integrations
, , , , ,
1

Hello all,

I'm using data warehouse connector to move data from Snowflake to Neo4j (AuraDB free). There are no issues when I create a relationship in a query construct, but when I try to add a property to the relationship I get an error:

Exception in thread "main" java.lang.IllegalArgumentException: Please provide a valid WRITE query
at org.neo4j.spark.util.ValidationUtil$.isTrue(ValidationUtil.scala:26)
at org.neo4j.spark.util.ValidateWrite.validate(Validations.scala:114)
at org.neo4j.spark.util.Validations$.$anonfun$validate$1(Validations.scala:12)
at org.neo4j.spark.util.Validations$.$anonfun$validate$1$adapted(Validations.scala:12)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)

Working query:

match (s:Destination {EXPERIENCE : event.CODE}), (t:Experience {ID: event.ID}) WITH s,t MERGE (s)-[r:HAS_EXPERIENCE]->(t)

Query which causes error:

match (s:Destination {EXPERIENCE : event.CODE}), (t:Experience {ID: event.ID}) WITH s,t MERGE (s)-[r:HAS_EXPERIENCE{weight: event.WEIGHT}]->(t)

I use a Spark Submit job in Azure Databricks to run this with these settings for target:

"target" : {

"format" : "org.neo4j.spark.DataSource",

"options" : {

"url" : "${env:NEO4JURL}",

"authentication.basic.username" : "${env:NEO4JUSERNAME}",

"authentication.basic.password" : "${env:NEO4JPASSWORD}",

"query" : "match (s:Destination {EXPERIENCE : event.CODE}), (t:Experience {ID: event.ID}) WITH s,t MERGE (s)-[r:HAS_EXPERIENCE{weight: event.WEIGHT}]->(t)"

},

"mode" : "Overwrite"

}

:: modules in use:
net.snowflake#snowflake-ingest-sdk;0.10.3 from central in [default]
net.snowflake#snowflake-jdbc;3.13.15 from central in [default]
net.snowflake#spark-snowflake_2.12;2.10.0-spark_3.2 from central in [default]
org.apache.xbean#xbean-asm6-shaded;4.10 from central in [default]
org.apiguardian#apiguardian-api;1.1.0 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12;4.1.2_for_spark_3 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12_common;4.1.2 from central in [default]
org.neo4j#neo4j-cypher-dsl;2020.1.4 from central in [default]
org.neo4j.driver#neo4j-java-driver;4.4.5 from central in [default]
org.reactivestreams#reactive-streams;1.0.3 from central in [default]
:: evicted modules:
net.snowflake#snowflake-jdbc;3.13.14 by [net.snowflake#snowflake-jdbc;3.13.15] in [default]

2

Hello @arjanv :slightly_smiling_face:

Could you try this query?

MATCH (s:Destination {EXPERIENCE: event.CODE}) 
MATCH (t:Experience {ID: event.ID}) 
MERGE (s)-[r:HAS_EXPERIENCE]->(t) 
SET r.weight = event.WEIGHT

Regards,
Cobra

3

Hello @Cobra ,

Thanks for your reply, I tried your solution and it works. But when I want to set multiple items it doesn't.

MATCH (s:Destination {EXPERIENCE: event.CODE}) 
MATCH (t:Experience {ID: event.ID}) 
MERGE (s)-[r:HAS_EXPERIENCE]->(t) 
SET r.weight = event.WEIGHT, r.height = event.HEIGHT

This doesn't work and this also doesn't work:

SET r += {weight: event.WEIGHT, height: event.HEIGHT }

Any idea why? Does it only support a single property to set?

4

Hello @arjanv :blush:

It should work. What is the error?

Regards,
Cobra