Hi guys,
Technical description :
- neo4j Community Edition version 5.20 and 5.19
- neo4j java driver version 5.20
- our java service, read data from kafka and merge/create index/node/relationship
- exemple of our cypher:
UNWIND $batch as row
MERGE (entity:kif { idRef: row.id })
ON CREATE
SET
entity.createdAt = localdatetime(row.createdAt),
entity.updatedAt = localdatetime()
WITH row, entity,hash
// remove old labels. They will be replaced by the new
CALL apoc.create.removeLabels(entity, labels(entity)) YIELD node as removedLabels
CALL apoc.create.addLabels(entity, row.labels) YIELD node as addedLabels
// remove and replace with new properties and labels
SET entity = {
idRef: row.id,
type: row.type,
raw: row.raw,
topic: row.topic,
app: row.app,
version: row.version,
domain: row.domain,
createdAt : entity.createdAt,
lastSyncAt: localdatetime(row.lastSyncAt),
updatedAt: localdatetime()
}
// add dependencies from
CALL apoc.do.when(row.dependencies is not null and size(row.dependencies) > 0, "
UNWIND row.dependencies as dependencies
call {
WITH row, entity, dependencies
// add dependencies
MERGE (entityDependency:kif { idRef: dependencies.id })
ON CREATE SET
entityDependency:unknown:kif,
entityDependency.createdAt = localdatetime(row.createdAt),
entityDependency.lastSyncAt = localdatetime(row.lastSyncAt),
entityDependency.updatedAt = localdatetime()
WITH row, entity, dependencies, entityDependency
// add relation between node and dependency
CALL apoc.merge.relationship(
entity,
dependencies.type,
null,
{createdAt:localdatetime(row.createdAt),lastSyncAt:localdatetime(row.lastSyncAt), updatedAt:localdatetime(), raw: dependencies.raw},
entityDependency,
{lastSyncAt:localdatetime(row.lastSyncAt),updatedAt:localdatetime(), raw: dependencies.raw}
) YIELD rel
RETURN 0 as ignored
} return *
", "", {row: row, entity: entity}) yield value as fromValue
....
Error Description :
After millions of ingestion, we encountered, randomly, this error
024-05-31 01:14:10.638+0000 ERROR [o.n.b.f.StateMachineImpl] Client triggered an unexpected error [Neo.DatabaseError.Transaction.TransactionStartFailed]: The database has encountered a critical er
ror, and needs to be restarted. Please see database logs for more details., reference 3d261b51-8666-4ceb-b28f-02a40a944955.
2024-05-31 01:14:10.638+0000 WARN [o.n.r.QueryRouter] Could not eagerly create kernel transaction due to: org.neo4j.graphdb.TransactionFailureException: The database has encountered a critical err
or, and needs to be restarted. Please see database logs for more details.
2024-05-31 01:14:10.638+0000 WARN [o.n.r.QueryRouter] Could not eagerly create kernel transaction due to: org.neo4j.graphdb.TransactionFailureException: The database has encountered a critical err
or, and needs to be restarted. Please see database logs for more details.
2024-05-31 01:14:10.631+0000 ERROR [o.n.b.f.StateMachineImpl] Client triggered an unexpected error [Neo.DatabaseError.Transaction.TransactionStartFailed]: The database has encountered a critical er
ror, and needs to be restarted. Please see database logs for more details., reference 66f56a69-4b42-4762-84a9-946602c7a226.
2024-05-31 01:14:10.631+0000 WARN [o.n.r.QueryRouter] Could not eagerly create kernel transaction due to: org.neo4j.graphdb.TransactionFailureException: The database has encountered a critical err
or, and needs to be restarted. Please see database logs for more details.
2024-05-31 01:14:10.630+0000 ERROR [o.n.b.f.StateMachineImpl] Client triggered an unexpected error [Neo.DatabaseError.Transaction.TransactionStartFailed]: The database has encountered a critical er
ror, and needs to be restarted. Please see database logs for more details., reference 9feea19e-cb2a-43b0-949b-02b3b0a2d9df.
org.neo4j.bolt.protocol.common.fsm.error.TransactionStateTransitionException: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
Caused by: org.neo4j.internal.id.indexed.IllegalIdTransitionException: Illegal addition ID state for range: 33554431 (IDs 4294967168-4294967231) transition
into: 00000000 00000000 00000000 00000000 00000000 00011111 11111111 11111111
from: 00000000 00000000 00000000 00000000 00000000 00011111 11110000 00000000
at org.neo4j.internal.id.indexed.IdRange.verifyMerge(IdRange.java:231) ~[neo4j-id-generator-5.20.0.jar:5.20.0]
at org.neo4j.internal.id.indexed.IdRange.mergeFrom(IdRange.java:158) ~[neo4j-id-generator-5.20.0.jar:5.20.0]
....
at org.neo4j.bolt.protocol.common.fsm.transition.transaction.streaming.StreamingStateTransition.process(StreamingStateTransition.java:62) ~[neo4j-bolt-5.20.0.jar:5.20.0]
at org.neo4j.bolt.protocol.common.fsm.transition.transaction.streaming.AutocommitStateTransition.process(AutocommitStateTransition.java:45) ~[neo4j-bolt-5.20.0.jar:5.20.0]
at org.neo4j.bolt.protocol.common.fsm.transition.transaction.streaming.StreamingStateTransition.process(StreamingStateTransition.java:55) ~[neo4j-bolt-5.20.0.jar:5.20.0]
at org.neo4j.bolt.protocol.common.fsm.transition.transaction.streaming.StreamingStateTransition.process(StreamingStateTransition.java:34) ~[neo4j-bolt-5.20.0.jar:5.20.0]
at org.neo4j.bolt.protocol.common.fsm.transition.transaction.TransactionalStateTransition.process(TransactionalStateTransition.java:42) ~[neo4j-bolt-5.20.0.jar:5.20.0]
... 11 more
Suppressed: java.lang.Exception: Illegal addition ID state for range: 33554431 (IDs 4294967168-4294967231) transition
into: 00000000 00000000 00000000 00000000 00000000 00011111 11111111 11111111
from: 00000000 00000000 00000000 00000000 00000000 00011111 11110000 00000000 | GB+Tree[file:/app/neo4j/data/databases/neo4j/neostore.propertystore.db.strings.id, layout:IdRangeLayout[version:1.2,
identifier:3735929182, fixedSize:true], generation:13385/13386]
Thanks for your help.