Neo4j Spark Connector integration with pyspark Using Pycharm

Integrations
1

Hi,
I'm trying to read nodes from my local neo4jdb for practice purposes by using pyspark and neo4j connector. I've already downloaded the last version of neo4j-connector-apache-spark (2.12) and integrated it in pyspark as explained in the repo.

Spark version :- 3.0
neo4j-connector used :- neo4j-connector-apache-spark_2.12-4.0.1_for_spark_3

However when I try to perform a read using pycharm :-

code
code825×336 32.2 KB

I am getting the following error :-

py4j.protocol.Py4JJavaError: An error occurred while calling o38.load.
: java.lang.AbstractMethodError: Method org/neo4j/spark/DataSource.getTable(Lorg/apache/spark/sql/util/CaseInsensitiveStringMap;)Lorg/apache/spark/sql/connector/catalog/Table; is abstract
at org.neo4j.spark.DataSource.getTable(DataSource.scala)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:220)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:206)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Thanks for your attention!

2

Discussion continued on the Github issue.

Problem was the pyspark version that wasn't supported by the connector since it wasn't stable.

Link to the issue: Neo4j Spark Connector integration with pyspark · Issue #347 · neo4j-contrib/neo4j-spark-connector · GitHub

1 Like