cancel
Showing results for 
Search instead for 
Did you mean: 

Head's Up! Site migration is underway. Phase 2: migrate recent content

NiFi goes Neo

martin_ebert
Node Clone

Hi community,
I've already tried Kettle and Neo. That went without a hitch.
Now I want to use NiFi and Neo together and encounter problems:

Neo processors are not included in the standard installation of NiFi, but there is a "nifi-graph-bundle" in the official NiFi Github Repo (https://github.com/apache/nifi/tree/master/nifi-nar-bundles).:

Following the instructions I come across some errors.

Since I haven't worked with Maven and Java yet, I don't know if it's the configuration.

Error message after executing "mvn -T 2.0C clean install" in the nifi root directory:
2X_8_88526dd404539173ce6a460dde622e31ee65b3aa.png

Can someone help me and even provide me with the required nar file? Initially I wanted to reach the graph database via JDBC, but of course the corresponding Execute SQL processor doesn't understand Cypher.

1 ACCEPTED SOLUTION

martin_ebert
Node Clone

NiFi 1.10.0 is out now.

#Bad News
The graph bundle is not included in the original nifi 1.10.0 but you can create the bundle by yourself.

#Good News
The processor works perfectly and only with URL, username and password.

You can choose:

The Controller Service is set up in seconds:

Processor is really simple:

Most importantly, it works:
2X_4_4c58090fe1d9c4145e0bb5501c9062bdd5f5111b.png

View solution in original post

28 REPLIES 28

martin_ebert
Node Clone

First of all, the graph bundle is coming but unfortunately not with nifi 1.9.2 but with 1.10.0 RC.
2X_d_da61f8be329b58af867bf954f3b0a6ec204b6119.png

I also found out what it takes to successfully create the package: Execute

mvn clean install -DskipTests

to compile tests, but skip running them.

So is there currently no way to work with Neo4j on nifi 1.9.2?

Hi @martin.ebert,

Thank you for this! Would you know if the graph bundle is compatible with Nifi 1.9.2?

Does this mvn clean install take a long time? It's going on for several hours in my laptop and often fails with connection reset error.

martin_ebert
Node Clone

NiFi 1.10.0 is out now.

#Bad News
The graph bundle is not included in the original nifi 1.10.0 but you can create the bundle by yourself.

#Good News
The processor works perfectly and only with URL, username and password.

You can choose:

The Controller Service is set up in seconds:

Processor is really simple:

Most importantly, it works:
2X_4_4c58090fe1d9c4145e0bb5501c9062bdd5f5111b.png

Which of the below nar files should be copied over to

\nifi-1.10\nifi-assembly\target\nifi-1.10.0-bin\nifi-1.10.0\lib

Got this done. After setting up nifi per the build instructions given in Github, just copy the below nar files to

nifi-graph-client-service-api-nar-1.10.0.nar
nifi-graph-nar-1.10.0.nar
nifi-neo4j-cypher-service-nar-1.10.0.nar
nifi-other-graph-services-nar-1.10.0.nar

\nifi-1.10\nifi-assembly\target\nifi-1.10.0-bin\nifi-1.10.0\lib

tom1
Node

Thanks for the guide.

I'm trying the same setup and can successfully query Neo4j using Cypher in the ExecuteGraphQuery when returning one node. However, when I query for multiple nodes I get an empty array back. For example, following this tutorial: https://neo4j.com/graphgist/first-steps-with-cypher

Running this query:

CREATE (you:Person {name:"You"})
RETURN you

Result:

[{"name":"You"}]

But a Cypher query with multiple nodes, gives an empty JSON array:

Query:

MATCH  (you:Person {name:"You"})
CREATE (you)-[like:LIKE]->(neo:Database {name:"Neo4j" })
RETURN you,like,neo

Result:
[{"you":{},"like":{},"neo":{}}]

Does anyone know how to solve this? we're using the graph bundle from 1.10

Hey @tom1,

What database version of Neo4j are you using? I have found an issue with particular versions of the Neo4j database not working properly with my NiFi implementation.

-yyyguy

Hi @yyyguy
We're using a Neo4j Docker image with version 3.5.15 and can connect fine:

That's good. Have you confirmed that the Neo4JCypherClientService controller service is Enabled?

I apologize for asking such basic questions. It is just that I have had this controller service sitting in an "Enabling" state without me realizing it.

-yyyguy

Yes, definitely enabled and we can run the Cypher queries fine:

Great. If you do a simple query like:

MATCH (n:Person)
RETURN n.name

it should return multiple results, however you are seeing no results returned? Is that correct?

-yyyguy

Yes, that query runs fine giving this:
2X_e_ef05583841b81166089e94df38a972180f940102.png

However, a query that returns multiple nodes in the Neo4j browser:

Gives an empty JSON array in the Flowfile content:
2X_0_090c6d2ba158d3446da2786e2d7b1e43408c8753.png

@tom1 can you run the query that I provided. I would like to see what the results are without the "you" filter. Also, can you provide a screenshot of the results in the LogAttribute processor on the success branch?

-yyyguy

Sure:

Gives this in the 'Success' branch:

Flowfiles all contain this:
2X_4_4fd2c3a461e7c31ade504ffc51e303558e0831eb.png

Ok, so in the next query can you please try this? Note: This assumes that you already have a Database node with name = "Neo4j"

MATCH (you:Person {name:"You"})
MERGE (you)-[:LIKE]->(neo:Database {name:"Neo4j" })
RETURN you.name, neo.name

-yyyguy

Now this query:

OK this works:

@tom1, it looks like if you do not have any properties associated with a relationship [like:LIKE], the query will fail.

Do you have any relationship properties that you plan to include in your database design? If not, do you think that is workable?

-yyyguy

BTW, it looks like you have created 3 relationships between the You node and the Neo4J node. MERGE is a better option perhaps.

-yyyguy

OK great - thank you for your help. We'll see if we can rework our schema and then try our query again

1.how did you passed the flowfile through the ExecuteGraphQuery processor
2.when i pass a query in process it gives "WRITER server connection not established error"

martin_ebert
Node Clone

Did anyone manage to ingest the FlowFile content? No matter if Parquet or json format.

I could well imagine that this could work with Load JSON. I can save the FlowFile content as an attribute if needed and reuse it directly in the Cypher statement.

I just want to save the intermediate step (save data locally and write it to Neo4j)

Don't suppose you got anywhere with this one?

Currently doing it ourselves with ExtractText and an attribute, but feels like a bit of a hack and would rather use the JSON in the flowfile content!

kevinneo4j
Node

hi

Thanks @yyyguy and @martin.ebert for information above. I have Apache Nifi 1.12 working against Neo4j 3.5.15.

However, when using Apache Nifi 1.12.0 I have problems connecting to the Neo4J version 4.0.4. The Neo4JCypherClientService throws a "org.neo4j.driver.v1.exceptions.ServiceUnavailableException. Connection to the database terminated"

You mentioned that you did have problems connecting with different versions of Neo4j. I was wondering if you had a chance to look at this and if there was a fix on the way.

Regards,
Kevin

Hi Kevin,
you talk about NiFi 1.11.x. 1.12 is not released yet. Anyways. I noticed the exact same error message. The neo4j nar files are not 4.0 compatible.

martin_ebert
Node Clone

Yes 3.x is working with NiFi 1.9.2.

Hello Martin,

Do you have any news regarding the compatibility of recent versions of neo4j and Nifi ? I tested the following versions :

Nifi 1.11.x + Neo 4.2.2 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.1.5 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.0.11 => ServiceUnavailableException
Nifi 1.11.x + Neo 3.5.26 => OK

Nifi 1.12.x + Neo 4.2.2 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.1.5 => ServiceUnavailableException
Nifi 1.11.x + Neo 4.0.11 => ServiceUnavailableException
Nifi 1.11.x + Neo 3.5.26 => OK

conker84
Graph Voyager

@karim.el I think that the best option to connect NiFi to the last Neo4j versions is via JDBC driver.