How to insert huge amount of data (i.e, in millions) into the neo4j?
can anyone suggest me any approach?
Thanks.
How to insert huge amount of data (i.e, in millions) into the neo4j?
can anyone suggest me any approach?
Thanks.
Seen this:
val relationshipConfig = Map(
"EASY_ID" -> ("IMSI", "Easy_Id_not_null", "IMSI", "EasyID"),
"AGE_GROUP" -> ("IMSI", "Age_Group", "IMSI", "AgeGroup"),
"GENDER" -> ("IMSI", "gender_not_null", "IMSI", "Gender"),
"DEVICE_OEM" -> ("IMSI", "Device_OEM", "IMSI", "DeviceOEM"),
"DEVICE_MODEL" -> ("IMSI", "Device_Model", "IMSI", "DeviceModel")
)
def writeToNeo4j(
data: DataFrame,
relationship: String,
sourceColumn: String,
targetColumn: String,
sourceLabel: String,
targetLabel: String,
url: String,
username: String,
password: String
) : Unit = {
data.select(sourceColumn, targetColumn).distinct().write
.mode("Append")
.format("org.neo4j.spark.DataSource")
.option("relationship", relationship)
.option("relationship.save.strategy", "keys")
//USE THIS TO PREVENT DUPLICATE RELATIONSHIPS (OPTIONAL FOR NOW)
.option("relationship.merge", "true")
//SOURCE NODE DETAILS
.option("relationship.source.save.mode", "Overwrite")
.option("relationship.source.labels", s":$sourceLabel")
.option("relationship.source.node.properties", s"$sourceColumn:$sourceLabel")
.option("relationship.source.node.keys", s"$sourceColumn:$sourceLabel")
//TARGET NODE DETAILS
.option("relationship.target.save.mode", "Overwrite")
.option("relationship.target.labels", s":$targetLabel")
.option("relationship.target.node.properties", s"$targetColumn:$targetLabel")
.option("relationship.target.node.keys", s"$targetColumn:$targetLabel")
//NEO4J CONNECTION DETAILS
.option("url", url)
.option("authentication.type", "basic")
.option("authentication.basic.username", username)
.option("authentication.basic.password", password)
.save()
}
I am inserting like this, but this approach inserting nodes and relationships one by one. but i have huge data.