Curious how streaming of results works

[f1-outsourcing]

I am curious how to approach streaming of data to clients. I guess you have to structure your cypher query in such a way that the results are able to stream and be displayed when received. The other thing is of course how does neo4j stream, does it send data as fast as possible or is there some sort of communication where the client notifies it has processed x amount of samples and is now ready to receive another x amount of samples. Also interesting is, how does this go when two clients execute the same query, but at a different time. Can you then 'reuse' the data of the client that already received these results?

[1]

Can Neo4j 4.1 reactively stream time-ordered nodes?

Context: I want to see if it is possible to build a timeline feed where the user is streaming new posts from Neo4j continuously as he scrolls (possibly by intermittently having to click Load more...). Neo4j has since version 4.0 the possibility to stream query data using reactive streams (reactive programming). I also believe Neo4j recent versions have DateTime data types which nodes might have as a property (so modelling time with nodes by use of Time Trees is unnecessary/legacy). Such a prope…

Streaming results with python vs javascript

Hi! Im trying to make a big query to export results from a neo4j 4 database. When using python to make the query, it seems results doesnt start streaming until the query has finished, which can take a very long time. With javascript the results starts streaming pretty much at once. I would like the second behaviour, but not use javascript. Is it possible to make the python driver stream results as soon as they are available somehow? Have I missed some config? I dont provide any extra confi…

[grant_lodge]

Hi @f1-outsourcing
Our drivers and bolt protocol stream records one by one in batches.
The way this works under the hood is the driver issues a query that opens a cursor on the server and then sends requests to read from that cursor.

If you ever log driver logs at the debug level, you will see these messages pipelined so that the pulling starts as soon as the values are available.

--> RUN ...
--> PULL X
<-- SUCCESS 
<-- RECORD ...
<-- SUCCESS {has_more:true}

Generally, on the server, as soon as the records are ready, they are written to the network up until the end of a batch, at which point it sends a success and if there is any more data. When data is written to the network, the network may or may not flush depending on the size of the data written; once all data for the batch is written, the server will then flush.

When the driver sees the SUCCESS {has_more:true}, and has processed that batch(some variation in behavior of drivers here) it will send another PULL X, note: This is abstracted for users, and we just expose an iterator of records in the driver.

how does this go when two clients execute the same query, but at a different time. Can you then 'reuse' the data of the client that already received these results?

This is not something the driver does; if the reuse of results is viable really depends more on your model and whether those results are still valid for reuse, to make this work would require some sort of caching layer far beyond the scope of a driver.