cancel
Showing results for 
Search instead for 
Did you mean: 

Neo4j enterprise edition memory issues

Hi,

I am running neo4j cluster with 3 cores and 1 read replica. We are ingesting around 72 million nodes data and getting below exception and warnings after couple of hour of data ingestion
ersion:6.0]
2022-05-11 21:03:10.247+0000 WARN [c.n.c.c.s.TxPuller] [neo4j/e86b6ca5] Unexpected exception when pulling transactions. [Address: ntd-neo4j-replica-0.ntd-neo4j-replica.ntd-vendors.svc.cluster.local:6000]
java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]
at com.neo4j.causalclustering.helper.OperationProgressMonitor.get(OperationProgressMonitor.java:45) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.CatchupClient.sendBlocking(CatchupClient.java:50) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.tx.TxPullClient.pullRawTransactions(TxPullClient.java:71) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.TxPuller.execute(TxPuller.java:122) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.TxPuller.pullTransactions(TxPuller.java:83) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.RemoteStore.pullTransactions(RemoteStore.java:186) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.RemoteStore.copy(RemoteStore.java:142) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.catchup.storecopy.StoreCopyProcess.replaceWithStoreFrom(StoreCopyProcess.java:79) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]
at com.neo4j.causalclustering.core.state.snapshot.StoreDownloader.bringUpToDate(StoreDownloader.java:52) ~[neo4j-causal-clustering-4.4.5.jar:4.4.5]

After coupl eof hours i am getting blow message continuously

2022-05-12 02:00:50.882+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 290 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:00:59.533+0000 INFO [c.n.c.c.t.TxPullRequestBatchingLogger] TxPullRequests batched latest 5 minutes:
[/172.16.7.244:39942] sent 145 TxPullRequests for DatabaseId{00000000} for transaction above id 79
2022-05-12 02:01:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 291 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:02:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 292 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:03:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 293 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:04:50.883+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 294 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:05:50.884+0000 INFO [c.n.c.c.r.RaftReplicator] [neo4j/e86b6ca5] Replication attempt 295 to leader RaftMemberId{725c8b48}: DistributedOperation{content=ReplicatedLeaseRequest{owner=RaftMemberId{725c8b48}, leaseId=3}, globalSession=GlobalSession{sessionId=457784f4-bd29-45fe-b46d-72207382fcb0, owner=RaftMemberId{725c8b48}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=0}}
2022-05-12 02:05:59.533+0000 INFO [c.n.c.c.t.TxPullRequestBatchingLogger] TxPullRequests batched latest 5 minutes:
[/172.16.7.244:39942] sent 153 TxPullRequests for DatabaseId{00000000} for transaction above id 79

Also the memory usage has reached allocated RAM limit of 24 GB. it is not coming down. All the queries are hung and ingest program is stuck.

Any help is appreciated.

Thanks,
Susheelstrong text

0 REPLIES 0
Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.