How to PERIODIC COMMIT when importing data from large Pandas Dataframe?

Drivers & Stacks Python
,
7

Thanks. I ended up having to also begin() a new transaction after the call to evaluate() inside the loop. If not, the call to evaluate() outside the loop raises a TransactionFinished exception.

Full code:

statement = """
UNWIND $parameters as row
MERGE (iui:ItemUsageInstance {instance_id: trim(row.USAGE_INSTANCE)})
  ON CREATE SET
    iui.usage_type = trim(row.USAGE_TYPE)
MERGE (nhiui:ItemUsageInstance {instance_id: trim(row.NEXT_HIGHER_USAGE_INST)})
  ON CREATE SET
    nhiui.usage_type = trim(row.NEXT_HIGHER_USAGE_TYPE)
"""
tx = graph.begin(autocommit=True)
params = []
# dataframe is indexed with numerical indexes
for index, row in df.iterrows():
    params_dict = {
        'USAGE_INSTANCE': row['USAGE_INSTANCE'], 
        'USAGE_TYPE': row['USAGE_TYPE'],
        'NEXT_HIGHER_USAGE_INST': row['NEXT_HIGHER_USAGE_INST'],
        'NEXT_HIGHER_USAGE_TYPE': row['NEXT_HIGHER_USAGE_TYPE']
    }
    params.append(params_dict)
    if index % 20000 == 0 and index > 0:
        tx.evaluate(statement, parameters = {"parameters" : params})
        tx = graph.begin(autocommit=True)
        params = []
tx.evaluate(statement, parameters = {"parameters" : params})