I am trying to join 1.5 million or more records with another couple million records generated by a custom function that creates a Cartesian product of a list. How do I batch it? My server came down when I was using the million records using the Cartesian product. How do I introduce batching and parallel processing?
MATCH (n:TagPolicy {policyId:'6fa31485-c70e-4d21-b45f-662dc6d48bb8'})
MATCH (pe:PolicyExpression)-[:CONSUMER_POLICY_EXPRESSION]-(n)
WITH pe, env
UNWIND pe.values AS tagKey
MATCH (env)-[:TAGGING_ENV {state: "Approved" }]-(tag:Tag {tagKey: tagKey})
WHERE tag.tagType = "CONSUMER"
AND "database" IN tag.levels
WITH pe, tagKey, tag.values AS approvedValues
UNWIND approvedValues AS value
WITH collect({tagKey: tagKey, values: {tagKey: tagKey, tagValue: value}}) AS tagData, pe
UNWIND tagData AS entry
WITH entry.tagKey AS tagKey, entry.values AS value, pe
WITH tagKey, COLLECT(value) AS values1, pe
WITH collect(values1) AS finalList, pe
WITH com.ccb.datalake.cartesianProduct(finalList) AS tagExpressions, pe
UNWIND tagExpressions AS tagExpression
WITH pe, tagExpression, [t IN tagExpression | t.tagKey + '=' + t.tagValue] AS expressionHashString,
apoc.util.md5([t IN tagExpression | t.tagKey + '=' + t.tagValue]) AS expressionHash
MERGE (list:DatabaseTagKeyValueList {expressionHash: expressionHash, expressionHashString: expressionHashString, envName: $auditEnv})
MERGE (pe)-[:HAS_TAG_EXPRESSION]->(list)
WITH pe, list, tagExpression
UNWIND tagExpression AS tag
MERGE (tagNode:TagKeyValue {tagKey: tag.tagKey, tagValue: tag.tagValue})
MERGE (list)-[:HAS_TAG]->(tagNode)