Batch Update Getting Slower

yunzhoun · July 10, 2020, 10:23am

Hi guys,

Following the popular article https://dzone.com/articles/tips-for-fast-batch-updates-of-graph-structures-wi. I'm doing my batch update. I'm using java API to update it and my query looks like:

UNWIND $props as row MERGE (n:Entity{eid:row.eid}) ON MATCH set n += row.properties

I have a 1 million records data set and I'm using a batch size as 10K, each time I send 10K records as a list of maps to my parameters. This approach worked fine at the beginning, but it got really slower or even stuck after 2 or 3 batch.

I have Index on Entity and eid, I used browser to test my query, and it looks like:

For the newly added node, when I tried to match them with label Entity and property eid, the index is also used. So I believe my problem is not due to index.

I tried to use smaller batch size as 1K, but the same problem occurred, after around 50 batch, the update got really slow.

What can I do to solve this problem? Any idea will be well appreciated.

Joel · July 10, 2020, 8:04pm

It looks like you have a Unique index on :Entity(eid), but it might be good to share the cypher for the index, to have a second set of eyes on it, just in case.

Beyond that we may need to understand more about your data and graph model (and maybe neo4j server) in order to provide suggestions, the challenge may well be something specific to your use case...

yunzhoun · July 13, 2020, 2:24am

Hi Joel,

Thanks for replying! I'm a bit confused by your wording "share the cypher for the index", could you please explain more?

The first column is the eid, and the last column is the properties I want to update to my server. I'm using java to process these data into lists of maps (depending on the batch size), and use them as parameters. Following is my code:

public void batchUpdate( final List<Map<String,Object>> maps )
    {
        try ( Session session = driver.session() )
        {

            String signal = session.writeTransaction( new TransactionWork<String>()
            {
                @Override
                public String execute( Transaction tx )
                {

                    Map<String,Object> params = new HashMap<>();
                    try {

                        params.put( "props", maps );
                    } catch (Exception e) {
                        System.out.println("File read problem");
                    }

                    StatementResult result = tx.run("UNWIND $props as row MERGE (n:Entity{eid:row.eid}) ON MATCH set n += row.properties", params);
                    return "done";
                }
            } );
            System.out.println( "one transaction " + signal );
        }
    }

Topic		Replies	Views
Seemingly simple batch query with 2500 rows takes over a minute to insert Cypher performance , cypher , batching	9	354	April 14, 2022
Batch update properties on relationships too slow Neo4j Graph Platform apoc	2	1175	July 29, 2020
Neo4j merge performance optimization Cypher performance , merge	4	1984	March 26, 2020
Very slow relationship creation (UNWIND) Cypher performance , cypher , relationship	8	2317	May 29, 2021
Generalised batch import of nodes and relationships Cypher	4	86	June 26, 2024

Batch Update Getting Slower

Related topics