cancel
Showing results for 
Search instead for 
Did you mean: 

Data Warehouse Connector Snowflake issue in connecting

arjanv
Node Link

Hello all, 

I'm trying to setup the best solution for the integration of Snowflake and Neo4j. Idea is that we use SNowflake as a source to update the Neo4j graph with the correct nodes.

I followed the instructions on https://github.com/neo4j-contrib/neo4j-dwh-connector to setup a Spark Submit job. Using the connector for Spark 3.x which I uploaded in Databricks and the configuration as supplied in the github documentation. I changed the configuration to match the DBFS path of my uploaded connector and the same for the json config file which is very simple at the moment.

Databricks choosen configuration: Driver: Standard_DS3_v2, Workers: Standard_DS3_v2, 0 workers, 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

In the logs I can see which modules are loaded:
modules in use:
net.snowflake#snowflake-ingest-sdk;0.10.3 from central in [default]
net.snowflake#snowflake-jdbc;3.13.15 from central in [default]
net.snowflake#spark-snowflake_2.12;2.10.0-spark_3.2 from central in [default]
org.apache.xbean#xbean-asm6-shaded;4.10 from central in [default]
org.apiguardian#apiguardian-api;1.1.0 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12;4.1.2_for_spark_3 from central in [default]
org.neo4j#neo4j-connector-apache-spark_2.12_common;4.1.2 from central in [default]
org.neo4j#neo4j-cypher-dsl;2020.1.4 from central in [default]
org.neo4j.driver#neo4j-java-driver;4.4.5 from central in [default]
org.reactivestreams#reactive-streams;1.0.3 from central in [default]
:: evicted modules:
net.snowflake#snowflake-jdbc;3.13.14 by [net.snowflake#snowflake-jdbc;3.13.15] in [default]

 

But then I receive an error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ScalaRunTime$.wrapRefArray([Ljava/lang/Object;)Lscala/collection/immutable/ArraySeq;
at org.neo4j.dwh.connector.utils.DatasourceOptions$.<clinit>(DatasourceOptions.scala:47)
at org.neo4j.dwh.connector.utils.CliUtils$.options(CliUtils.scala:33)
at org.neo4j.dwh.connector.utils.CliUtils$.parseArgs(CliUtils.scala:28)
at org.neo4j.dwh.connector.Neo4jDWHConnector$.main(Neo4jDWHConnector.scala:22)
at org.neo4j.dwh.connector.Neo4jDWHConnector.main(Neo4jDWHConnector.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:956)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1045)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1054)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

Anyone an idea how to solve this?

Thanks in advance!

1 ACCEPTED SOLUTION

arjanv
Node Link

Apparently there was an issue with the connector version of Scala, I recreated the connector with this command: ./maven-release.sh package 2.12 3 , uploaded this file and now the connection is established. 

View solution in original post

2 REPLIES 2

arjanv
Node Link

To be complete hereby also the parameters of the Spark Submit job:

["--packages","org.neo4j:neo4j-connector-apache-spark_2.12:4.1.2_for_spark_3,net.snowflake:spark-snowflake_2.12:2.10.0-spark_3.2,net.snowflake:snowflake-jdbc:3.13.15","--class","org.neo4j.dwh.connector.Neo4jDWHConnector","dbfs:/FileStore/jars/0738d673_adde_4333_a6ee_5ddfff077c5a-neo4j_dwh_connector_2_13_1_0_0_for_spark_3-654d1.jar","-p","/dbfs/FileStore/tables/ImportData.json"]

arjanv
Node Link

Apparently there was an issue with the connector version of Scala, I recreated the connector with this command: ./maven-release.sh package 2.12 3 , uploaded this file and now the connection is established. 

Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.