CDC changes and reads from a cluster

Neo4j Graph Platform
,
1

Hi All,

We run a db-database on a three node cluster (self-managed). We've enabled CDC. We read the transactionlog via the cypher methods: db.cdc.query() from which we retrieve relevant changes for downstream processing.

Once in a while the downstream processing can't find the nodes that were affected and published on the transactionlog. We don't do any deletion on our database, so that is quite unexpected.

Manual querying afterwards (after we were signalled by our monitoring system), show the data in the graph, running the exact same query

So I have some questions:

  • Can have database write propagation be part of the problem
  • Can different type transactional behaviour create such problems?

Any other possible cause can I think of?

2

tldr: at times there are record changes published to the CDC log that refer to nodes that don't seem to exist when CDC actors query for those references nodes. What might cause this?

3

@daniel.portier

How are you querying the database to see the nodes post db.cd.query() call?

In a cluster it is possible whereby for a write to occur, we only need a majority of the cluster members to accept the write for the write to be committed. i.e. in a 3 node cluster once 2 members of the cluster agree to accept the write the txn will be committed and the 3rd node will eventually accept the write. But with the usage of bookmarks you should always be guarenteed to read your own writes. When using bookmarks, If your request is routed to a member which is not caught up to the bookmark then the transaction will not return until it is caught up.

For example this is described in the Neo4j Python driver at Coordinate parallel transactions - Neo4j Python Driver Manual. The other drivers have similar documentation

4

Hi Dana,

Thanks for your suggestion. I'll give that a try.

After reading the manual, especially the section Mix .execute_query() and sessions it is not exactly clear to me, if the driver.execute_query_bookmark_managersuggestion works in different processes, as well. Or that is per driver. Because the writers and readers are totally different processes (aws lambda's). But I'll give it a try.