How to insert ObjectId's from MongoDB with apoc?

hedefalk · September 14, 2019, 9:50pm

I'm trying to seed a neo4j with data from a mongodb using apoc. The idea is to first create a kindof 1-1 mirror of the mongo db and then replace the key-based mapping with relations. I'm having troubles with ObjectId's which are extensively used as the main means for relations (PK/FK):

{
	"_id" : ObjectId("50db1341c2e638a3ebef2404"),
	"parent" : ObjectId("50db12c2c2e638a3ebef2401"),
	"referralDensityInfo" : ObjectId("50db12c2c2e638a3ebef2401"),
	"unitInfo" : [ ],
	"plural" : "Röda paprikor",
	"singular" : "Röd paprika",
	"referralUnitInfo" : [
		{
			"reference" : ObjectId("50db12c2c2e638a3ebef2401"),
			"unit" : ObjectId("508e86f6c2e60ff9d423e293")
		}
	],
	"dataSourceNutritionInfo" : ObjectId("507aa464c2e631891c0d9046"),
	"referralNutritionInfo" : ObjectId("50db12c2c2e638a3ebef2401"),
	"preferredName" : "Röd paprika",
	"searchWords" : [
		""
	]
}

First try was just using:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {})

Cannot convert org.bson.types.ObjectId to AnyValue

Then I found this little clue:

So now calling:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true)

I get a bunch of data and it seems to work so far. However, when trying to use the values:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true) YIELD value as mongo_i
MERGE (i: Ingredient {_id: mongo_i._id})
SET i += {
  parent: mongo_i.parent,
  dataSourceNutritionInfo: mongo_i.dataSourceNutritionInfo
};

Property values can only be of primitive types or arrays thereof

So I guess there is something I'm missing in the puzzle still to be able to use these ObjectId's.

I have the jar's I'm supposed to have I think:

# ls -l plugins
total 15840
-rwxrwxrwx 1 neo4j neo4j     2217 Aug 22 11:27 README.txt
-rw------- 1 neo4j neo4j 13749813 May 31 11:12 apoc-3.5.0.4-all.jar
-rw-r--r-- 1 root  root    494021 Aug 13 11:06 bson-3.11.0.jar
-rw-r--r-- 1 root  root    374817 Aug 13 11:43 mongodb-driver-3.11.0.jar
-rw-r--r-- 1 root  root   1435219 Aug 13 11:25 mongodb-driver-core-3.11.0.jar
-rw-r--r-- 1 root  root    152721 Aug 13 11:34 mongodb-driver-sync-3.11.0.jar

There seems to be some effort to handle Id's here:

github.com

neo4j-contrib/neo4j-apoc-procedures/blob/9f6cf4fc256eb7505fdfa48c6a8f749d8bf352f1/src/main/java/apoc/mongodb/MongoDBColl.java#L86


      
                  }
              }
          
          
    /**
               * A document in MongoDB has a special field "_id" of type ObjectId
               * This object is not "packable" by Neo4jPacker so it must be converted to a value that Neo4j can deal with
               *
               * If the document is not null we simply override the ObjectId instance with its string representation
               */
          
          
    if (document != null) {
                  Object objectId = document.get("_id");
                  if (objectId != null) {
                      document.put("_id", objectId.toString());
                  }
          
          
    }
          
          
    return document;
          }

but this seems to just handle the object primary key "_id", not any foreign key references to such a primary key which I guess is why I'm having problems.

Can I hook my own serializer in there somewhere? I would like to stick a toString() there on my keys and I would be fine I guess, but now it's just choking…

hedefalk · September 14, 2019, 9:55pm

I noticed that the jar's permissions probably wasn't good. I figured maybe some automagic conversion wasn't flying because of missing bson.jar so I fixed that in my docker build:

COPY --from=sbt-build --chown=neo4j:neo4j /root/target/pack/lib/bson*.jar /var/lib/neo4j/plugins/
COPY --from=sbt-build --chown=neo4j:neo4j /root/target/pack/lib/mongodb-driver*.jar /var/lib/neo4j/plugins/

# ls -l plugins
total 15840
-rwxrwxrwx 1 neo4j neo4j     2217 Aug 22 11:27 README.txt
-rw------- 1 neo4j neo4j 13749813 May 31 11:12 apoc-3.5.0.4-all.jar
-rw-r--r-- 1 neo4j neo4j   494021 Aug 13 11:06 bson-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j   374817 Aug 13 11:43 mongodb-driver-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j  1435219 Aug 13 11:25 mongodb-driver-core-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j   152721 Aug 13 11:34 mongodb-driver-sync-3.11.0.jar

but unfortunately that didn't help.

hedefalk · September 14, 2019, 10:00pm

Ooh, ok, found the apoc.convert.toString

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true) YIELD value as mongo_i
MERGE (i: Ingredient {_id: apoc.convert.toString(mongo_i._id)})
SET i += {
  parent: apoc.convert.toString(mongo_i.parent),
  dataSourceNutritionInfo: apoc.convert.toString(mongo_i.dataSourceNutritionInfo)
};

seems to work :)

hedefalk · September 14, 2019, 10:03pm

Nope, that did not work well. That toString seems to do some object traversal which is definitely not what I want:

Any tips highly appreciated!

mike_r_black · September 15, 2019, 4:00am

I can't offer any help but I hope you're able to figure it out because I too will be doing a similar task soon so I'll be following this post for any wisdom I can glean.

michael.hunger · September 16, 2019, 6:14am

Can you create an APOC GH issue, so that we can handle those types of objects recursively both by themselves but also as part of arrays/subdocuments.

and link this thread.

hedefalk · September 16, 2019, 10:07am

Done. I can't link that issue because:

Probably should allow even newcomers to post links to github at least. But it's #1288.

I went another path so no rush for my sake

@mike_r_black I think it makes sense for anything but the most simple cases to probably make an app in your language of choice for the migration. I made a Scala app now that pulls data from Mongo and sticks them into neo4j, just using the appropriate drivers, in my case mongo-scala-driver and neotypes.

It seems to me that the apoc plugin for mongo is very very basic to say the least. I can't imagine anyone has ever used it for anything real, but maybe I'm missing something.

michael.hunger · November 12, 2019, 12:43am

we fixed the issue in the current apoc release

deep.deepak.3891 · January 2, 2021, 9:13am

Hello Michael,
I am doing a project in which I have to use mongoDB collection with which I want to use mongo ObjectId as neo4j collection's properties. I used apoc.convert.toString, it worked but I only want the exact value part of the objectId. Please suggest me some solution so that I can achieve my goal.

I am attaching screenshot for you to understand my issue.

Thanks.!
Screenshot (88)|690x388

Topic		Replies	Views
Any way to query apoc.mongodb.find by ObjectId Procedures & APOC apoc , mongodb , cypher	2	521	January 25, 2021
Problem using "apoc.mongodb.get" Procedures & APOC apoc	0	252	August 5, 2020
Inserting a date from Neo4j into mongodb as date type Procedures & APOC	6	748	June 22, 2020
Integrate or Migrate data from Mongo DB to Neo4j Integrations apoc , mongodb , import	6	3725	October 8, 2021
Trying to create a recipes graph by importing collection from MongoDB Integrations apoc , mongodb , import	1	564	September 8, 2020

How to insert ObjectId's from MongoDB with apoc?

Related topics