Neo4j Spark Connector integration with pyspark Using Pycharm

Hi,
I'm trying to read nodes from my local neo4jdb for practice purposes by using pyspark and neo4j connector. I've already downloaded the last version of neo4j-connector-apache-spark (2.12) and integrated it in pyspark as explained in the repo.

Spark version :- 3.0
neo4j-connector used :- neo4j-connector-apache-spark_2.12-4.0.1_for_spark_3

However when I try to perform a read using pycharm :-

I am getting the following error :-

py4j.protocol.Py4JJavaError: An error occurred while calling o38.load.
: java.lang.AbstractMethodError: Method org/neo4j/spark/DataSource.getTable(Lorg/apache/spark/sql/util/CaseInsensitiveStringMap;)Lorg/apache/spark/sql/connector/catalog/Table; is abstract
at org.neo4j.spark.DataSource.getTable(DataSource.scala)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:220)
at scala.Option.map(Option.scala:230)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:206)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Thanks for your attention!

Discussion continued on the Github issue.

Problem was the pyspark version that wasn't supported by the connector since it wasn't stable.

Link to the issue: Neo4j Spark Connector integration with pyspark · Issue #347 · neo4j-contrib/neo4j-spark-connector · GitHub

1 Like