# How to PERIODIC COMMIT when importing data from large Pandas Dataframe?

**URL:** <https://community.neo4j.com/t/how-to-periodic-commit-when-importing-data-from-large-pandas-dataframe/630>\
**Category:** Python\
**Tags:** import, batching\
**Created:** [August 29, 2018, 2:05am UTC](https://community.neo4j.com/t/how-to-periodic-commit-when-importing-data-from-large-pandas-dataframe/630 "2018-08-29T02:05:56Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![apanza](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/apanza/32/6209_2.png) [@apanza](https://community.neo4j.com/u/apanza)\
**Post date:** [August 31, 2018, 10:14pm UTC](https://community.neo4j.com/t/how-to-periodic-commit-when-importing-data-from-large-pandas-dataframe/630/7 "2018-08-31T22:14:07Z")

</div>

Thanks. I ended up having to also `begin()` a new transaction after the call to `evaluate()` inside the loop. If not, the call to `evaluate()` outside the loop raises a `TransactionFinished` exception.

Full code:

```auto
statement = """
UNWIND $parameters as row
MERGE (iui:ItemUsageInstance {instance_id: trim(row.USAGE_INSTANCE)})
  ON CREATE SET
    iui.usage_type = trim(row.USAGE_TYPE)
MERGE (nhiui:ItemUsageInstance {instance_id: trim(row.NEXT_HIGHER_USAGE_INST)})
  ON CREATE SET
    nhiui.usage_type = trim(row.NEXT_HIGHER_USAGE_TYPE)
"""
tx = graph.begin(autocommit=True)
params = []
# dataframe is indexed with numerical indexes
for index, row in df.iterrows():
    params_dict = {
        'USAGE_INSTANCE': row['USAGE_INSTANCE'], 
        'USAGE_TYPE': row['USAGE_TYPE'],
        'NEXT_HIGHER_USAGE_INST': row['NEXT_HIGHER_USAGE_INST'],
        'NEXT_HIGHER_USAGE_TYPE': row['NEXT_HIGHER_USAGE_TYPE']
    }
    params.append(params_dict)
    if index % 20000 == 0 and index > 0:
        tx.evaluate(statement, parameters = {"parameters" : params})
        tx = graph.begin(autocommit=True)
        params = []
tx.evaluate(statement, parameters = {"parameters" : params})

```

---

_[View the full topic](https://community.neo4j.com/t/how-to-periodic-commit-when-importing-data-from-large-pandas-dataframe/630)._