# Struggling with apoc.periodic.iterate in a big Query from python code

**URL:** <https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165>\
**Category:** Cypher\
**Tags:** apoc, cypher, apocperiodiciterate\
**Created:** [November 21, 2018, 4:53pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165 "2018-11-21T16:53:31Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [November 21, 2018, 4:53pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/1 "2018-11-21T16:53:31Z")

</div>

Hello,

I've been struggling for days to optimize the following request:

For the context, i have a graph that contains User nodes with some properties and relationships.  
From python, i fetch these nodes and do some calculation with them and the result is the following data that i want to update neo4j with:

My data is as follow:

```auto
{'Labels': [{'date': '21/11/2018', 'rankings': [{'score': '0.00967110194468', 'lbl': '0', 'user': 'alpha'}, {'score': '0.00967110194468', 'lbl': '0', 'user': 'Betha'}], 'name'='dummy'},
	    {'date':...}]}

```

The following works on a small amount of Data so I tried to use apoc.periodic.iterate without success

```auto
query = "UNWIND $Labels as label"+\
            " MERGE (c:Label {name: label.name})" +\
            " ON CREATE SET c.date = label.date"+\
            " WITH c, label"+\
            " UNWIND label.rankings as ranking"+\
            " MATCH (b:User) where b.name = ranking.user"+\
            " MERGE (b)-[r:Has{score: ranking.score}]->(c)"

```

Even with a basic request as follow, I got nothing, no errors, nothing at all. it seems like it is running but i got 0 results.

`query = "CALL apoc.periodic.iterate(UNWIND $Labels as label RETURN label, CREATE (:Test {name:label.name}), {batchSize:1000, iterateList:True, parallel:true})"`

Can someone please help me? I'm new to neo4j.

I use neo4j 3.4.7 and the neo4j-driver for python.

Thanks in advance for your help.

---

<div class="post-metadata">

**Author:** ![lyonwj](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/lyonwj/32/27358_2.png) [@lyonwj](https://community.neo4j.com/u/lyonwj)\
**Post date:** [November 21, 2018, 7:58pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/2 "2018-11-21T19:58:44Z")

</div>

Did you try the query in Neo4j Browser? If you do I suspect you'll see some errors. `apoc.periodic.iterate` takes 2 Cypher queries as strings, so you'll need to add quotes. In Python you can use 3 double quotes to do multi-line string I believe, so without having to worry about escaping the quotes:

```
query = """ CALL apoc.periodic.iterate('UNWIND $Labels AS label RETURN label',
            'CREATE (:Test {name: label.name})', 
             {batchSize: 1000, iterateList: True, parallel: True})
 """
```

---

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [November 22, 2018, 9:09am UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/3 "2018-11-22T09:09:41Z")

</div>

Thanks William!

I will try in the browser and add the quotes as suggested and get back to you.

Regards

---

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [November 22, 2018, 2:18pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/4 "2018-11-22T14:18:20Z")

</div>

Hello William,

I tried to make it work from the browse but a couldn't do it.

Here is my sample query

```auto

With [{date: '21/11/2018', r: [{score: '0.009', user: 'F'}, {score: '0.009', user: 'j'}],name: 1}, {date: '21/11/2018', r: [{score: '0.009', user: 'ji'}, {score: '0.006', user: 'Y'}, {score: '0.006', user: 'B'}], name: 2}] as labels 
CALL apoc.periodic.iterate('UNWIND $labels as label RETURN label', 
        'CREATE (:Label {name:label.name})', 
        {batchSize:1, iterateList:True, parallel:true})

```

I got the following error:

```auto
Neo.ClientError.Statement.SyntaxError: Query cannot conclude with CALL (must be RETURN or an update clause) (line 2, column 1 (offset: 352))
"CALL apoc.periodic.iterate('UNWIND $labels as label RETURN label', "
 ^

```

I tried several approaches including putting the "WITH" inside the apoc procedure without success.

Can you tell me where i am wrong?

---

<div class="post-metadata">

**Author:** ![michael.hunger](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/michael.hunger/32/27377_2.png) [@michael.hunger](https://community.neo4j.com/u/michael.hunger)\
**Post date:** [November 23, 2018, 6:46am UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/5 "2018-11-23T06:46:24Z")

</div>

you need to set the params separately in browser:

```auto
:params labels => [{date: '21/11/2018', r: [{score: '0.009', user: 'F'}, {score: '0.009', user: 'j'}],name: 1}, {date: '21/11/2018', r: [{score: '0.009', user: 'ji'}, {score: '0.006', user: 'Y'}, {score: '0.006', user: 'B'}], name: 2}]

```

And then you can pass in the params to `periodic.iterate` like this:

```auto
CALL apoc.periodic.iterate('UNWIND $labels as label RETURN label', 
        'CREATE (:Label {name:label.name})', 
        {batchSize:1, iterateList:True, parallel:true, params: {labels:$labels}})

```

---

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [November 23, 2018, 11:45am UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/6 "2018-11-23T11:45:35Z")

</div>

Great!!! It worked just fine! Thanks a lot!  
Now, i can move on to the whole query :)

---

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [November 23, 2018, 4:43pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/7 "2018-11-23T16:43:56Z")

</div>

Hello,

Just to tell you that i was able to modify and run my global query with apoc. Thanks a lot for your help.

Now i have another question, How can i fetch millions of data quickly.

My basic request is as follow:

`MATCH (n:User)-[r:R1]-(m:User) RETURN n.name,r.weight, m.name`

I have tried with `apoc.cypher.parallel` but i don't understand how it works and i didn't find a lot of documentation.  
Is i possible to use `apoc.periodic.iterate` for this kind of query?

regards

---

<div class="post-metadata">

**Author:** ![andrew\_bowman](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/andrew_bowman/32/73_2.png) [@andrew\_bowman](https://community.neo4j.com/u/andrew_bowman)\
**Post date:** [November 24, 2018, 9:11pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/8 "2018-11-24T21:11:42Z")

</div>

Not presently, `apoc.periodic.iterate()` is designed for batching writes to the graph, not for retrieval.

The client application is likely going to be one of the main bottlenecks for these kinds of queries. You may want to try using [cypher-shell](https://neo4j.com/docs/operations-manual/current/tools/cypher-shell/) when you have to return a lot of text data rather than the browser.

---

<div class="post-metadata">

**Author:** ![michael.hunger](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/michael.hunger/32/27377_2.png) [@michael.hunger](https://community.neo4j.com/u/michael.hunger)\
**Post date:** [November 26, 2018, 1:37am UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/9 "2018-11-26T01:37:08Z")

</div>

You should be able to fetch this data quickly with any Bolt driver after the db is warmed up.  
Make sure you consume the results in a streaming manner so the they don't allocate memory in your client or block processing.

Do you use Community or Enterprise? Enterprise Edition has a faster Cypher Runtime that definitely helps a lot.  
What is your memory config?

Can you share an EXPLAIN plan of your query? Also make sure to add an direction arrow.  
What does your code look like?

---

<div class="post-metadata">

**Author:** ![barrydjidja](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/barrydjidja/32/1565_2.png) [@barrydjidja](https://community.neo4j.com/u/barrydjidja)\
**Post date:** [December 14, 2018, 1:45pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/10 "2018-12-14T13:45:29Z")

</div>

Hello,  
Sorry for the delay. I was finally able to fecth my data quickly.  
I am using the community edition.  
Thanks again all for your help!

---

<div class="post-metadata">

**Author:** ![michael.hunger](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/michael.hunger/32/27377_2.png) [@michael.hunger](https://community.neo4j.com/u/michael.hunger)\
**Post date:** [December 14, 2018, 3:42pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/11 "2018-12-14T15:42:27Z")

</div>

Cool, how did you do it in the end?

---

<div class="post-metadata">

**Author:** ![12kunal34](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/12kunal34/32/2617_2.png) [@12kunal34](https://community.neo4j.com/u/12kunal34)\
**Post date:** [December 31, 2018, 8:02am UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/12 "2018-12-31T08:02:57Z")

</div>

Hi @barrydjidja  
could you please let me know how you done that ??  
i am facing same issue , getting heap memory issue ☹

---

<div class="post-metadata">

**Author:** ![benjamin.squire](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/benjamin.squire/32/1496_2.png) [@benjamin.squire](https://community.neo4j.com/u/benjamin.squire)\
**Post date:** [May 8, 2019, 4:28pm UTC](https://community.neo4j.com/t/struggling-with-apoc-periodic-iterate-in-a-big-query-from-python-code/3165/13 "2019-05-08T16:28:06Z")

</div>

@barrydjidja interested in your steps here too