Hello all,
I'm trying to setup the best solution for the integration of Snowflake and Neo4j. Idea is that we use SNowflake as a source to update the Neo4j graph with the correct nodes.
I followed the instructions on https://github.com/neo4j-contrib/neo4j-dwh-connector to setup a Spark Submit job. Using the connector for Spark 3.x which I uploaded in Databricks and the configuration as supplied in the github documentation. I changed the configuration to match the DBFS path of my uploaded connector and the same for the json config file which is very simple at the moment.
Databricks choosen configuration: Driver: Standard_DS3_v2, Workers: Standard_DS3_v2, 0 workers, 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
In the logs I can see which modules are loaded:
modules in use:
net.snowflake#snowflake-ingest-sdk;0.10.3 from central in [default]
net.snowflake#snowflake-jdbc;3.13.15 from central in [default]
net.snowflake#spark-snowflake_2.12;2.10.0-spark_3.2 from central in [default]
org.apache.xbean#xbean-asm6-shaded;4.10 from central in [default]
org.apiguardian#apiguardian-api;1.1.0 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12;4.1.2_for_spark_3 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12_common;4.1.2 from central in [default]
org.neo4j#neo4j-cypher-dsl;2020.1.4 from central in [default]
org.neo4j.driver#neo4j-java-driver;4.4.5 from central in [default]
org.reactivestreams#reactive-streams;1.0.3 from central in [default]
:: evicted modules:
net.snowflake#snowflake-jdbc;3.13.14 by [net.snowflake#snowflake-jdbc;3.13.15] in [default]
But then I receive an error:
Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ScalaRunTime$.wrapRefArray([Ljava/lang/Object;)Lscala/collection/immutable/ArraySeq;
at org.neo4j.dwh.connector.utils.DatasourceOptions$.(DatasourceOptions.scala:47)
at org.neo4j.dwh.connector.utils.CliUtils$.options(CliUtils.scala:33)
at org.neo4j.dwh.connector.utils.CliUtils$.parseArgs(CliUtils.scala:28)
at org.neo4j.dwh.connector.Neo4jDWHConnector$.main(Neo4jDWHConnector.scala:22)
at org.neo4j.dwh.connector.Neo4jDWHConnector.main(Neo4jDWHConnector.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:956)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1045)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1054)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Anyone an idea how to solve this?
Thanks in advance!