Is CDC seq order safe to use for generate cypher?

Neo4j Graph Platform Cypher
1

In the CDC doc about seq

A number used for ordering changes that happened in the same transaction. The order of changes observed in the output does not necessarily correspond to the order in which changes were applied during the transaction.

What does this mean?

I want use seq order to generate cypher, so the order is important.
For ex:
If I run these 3 cypher in a transaction.

CREATE (tom:Person {name: "Tom Hanks"})
CREATE (movie1:Movie {title: "Movie 1"})
CREATE (tom)-[:ACTED_IN]->(movie1)

Any chance the seq be like this?

seq 0 CREATE (tom)-[:ACTED_IN]->(movie1)
seq 1 CREATE (tom:Person {name: "Tom Hanks"})
seq 2 CREATE (movie1:Movie {title: "Movie 1"})

If yes, my generated cypher will fail because tom and movie1 are not exist when run seq 0.

Or it always create node first then relationship?

In my test, I never see it create relationship first.

2

@abccbaandy

I cant really think how the relationship creation would occur 1st since for the relationship to be created the 2 nodes in seq 1 and seq 2 have to be created 1st. or maybe im misreading/misunderstanding

3

Hi @dana_canzano
This is why I ask this question.

I don't know if this is possible to happen. The quote seems the seq is not reliable?

4

@abccbaandy

The quote seems the seq is not reliable?

what quote? something as decscribed in a document under Neo4j documentation - Neo4j Documentation?

again not sure how u can create a realtionship and to which a relationship is predicated on the existence of a starting and end node, but yet the starting/end node have not been created

5

@abccbaandy

The events for a single transaction in CDC always follows this ordering:

  • node creation
  • relationship creation
  • node labels/props updates
  • relationship props updates
  • node deletions
  • relationship deletions

The comment relates to the fact that you could run something like the following

MATCH (n:Person {name: 'Forrest Gump'}) DELETE n;
CREATE (tom:Person {name: "Tom Hanks"})

so in this case you'd see the create event first, followed by the delete event - even though the node deletion was the first operation in the transaction

1 Like
6

A number used for ordering changes that happened in the same transaction. The order of changes observed in the output does not necessarily correspond to the order in which changes were applied during the transaction.

from

7

Thanks for reply @rory.steele
Then the seq seems not reliable for recreate cypher, right?

If I am doing some recreate biz logic:

MATCH(n {id:1}
DEL n

CREATE (newn {id:1})

CDC will give me:

seq 0 create id : 1
seq 1 del id:1

Right?

8

@abccbaandy

thank you and my apologies for not spending more effort to better understand.
But ill leave this to @rory.steele responses

9

Hi @abccbaandy

Yep - your event ordering is correct for that biz logic

The events will have the elementId though and you could use that for the deletes in the cypher you generate instead.

10

Can you explain more detail about how to use the elementId for this use case?

I guess you mean delete by elementId? Won't it delete other node by accident? Since elementId is not unique.

Also In my previous example:
The first generated cypher will fail because id 1 already exist.

Btw I thought elementId is not safe to use outside the transaction?
I even open a question for it but no response yet.

11

@abccbaandy

Could you explain a bit more about the generated cypher and what you plan to do with it? Might help me give some better answers :crossed_fingers:

On elementId - it is unique to an entity at a certain point in time, i.e. if there is a node/rel with that ID, then no other node/rel will have that ID in the database at the same time. If that entity gets deleted, then the elementId it had will get reused at some point in the future. This is why they shouldn't be used as persistent IDs in external systems

12

Sure, what I plan to do is undo/redo feature like many editors have, for ex: Microsoft word.

Sry, but I am not asking the explanation for the elementId, I am asking how the elementId help me in this case, because it's 100% outside the transaction.

13

For your use case - you're correct in that elementId isn't going to help you unfortunately

From what you've described , it looks like you'd need your ID property to be unique for the undo-redo to work with the order of the CDC events. Some monotonically increasing ID should work

14

This seems lots of work to do :frowning:

Any chance that CDC seq follow the execute order?

Or why it's order like that? Any tech reason?

15

The state that gets maintained during a transaction contains the logical changes that would happen rather than the physical operations that were performed.
As the CDC events are generated from this tx state before they are applied to the store, all execution order has been lost at this point.

1 Like
16

Hi, can you explain more detail how monotonically increasing ID way work?

I think the id should bind to the CDC event instead node/relationship, right?

But how can I do that?

The CDC metadata is the same per transaction.

17

Hi

So something like this pseudo code when creating entities

CREATE (newn {id:counter++})

where counter is always increasing. This way, a delete of the entity will always be for that entity alone. Managing that counter across multiple machines is a different problem though ...

Another option would be to use a UUID - you get the same outcome in that each entity would always get a unique ID so there would never be a risk of a delete acting on the wrong entity. Whilst much easier to implement, the property/value might not be something you want in your model

18

Oops, so your solution is still based on Node id.
I thought it will based on event.

Recreate biz logic just one of logic came in my mind when I found out the seq order issue.
Also the unique id way won't help if there is another property in the node that have unique constraint.
Now I guess you may suggest we always do delete event first when undo, but who knows if there is another issue to do that.

Anyway, without reliable order, I think we need to review all our biz logic to ensure the "order way" won't affect us :cry: