Mongo connector log -v output.. its up .. NEO4J_AUTH exported and set.. then

Neo4j Graph Platform
, , ,
1

its up listening away to mongo log for changes ..I have Neo4j desktop up and the browser and the browser neo4j / password in env variable set.. cool.. BUT I still cant see how the mongodb 'in use' pops up in neo4j ?? do we have th create a local graph db of the same same as the replicaset ? or the actual db in mongo ?

anyway.. as below it listens.. then we use Compass change a mongo collection document.. cool.. it is picked up by mongo connector and the log shows the update.. BUT when it trys to 'update/ show or put in' the mongo changes to neo4j.. it blows.. see below..

ANY help appreciated thanks guys

2020-10-30 09:58:48,171 [DEBUG] mongo_connector.oplog_manager:242 - OplogThread: Cursor is still alive and thread is still running.
2020-10-30 09:58:49,171 [DEBUG] mongo_connector.oplog_manager:338 - OplogThread: updating checkpoint after processing new oplog entries
2020-10-30 09:58:49,171 [DEBUG] mongo_connector.oplog_manager:856 - OplogThread: no checkpoint to update.
2020-10-30 09:58:49,171 [DEBUG] mongo_connector.oplog_manager:242 - OplogThread: Cursor is still alive and thread is still running.
2020-10-30 09:58:49,621 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 0
2020-10-30 09:58:49,621 [DEBUG] mongo_connector.oplog_manager:271 - OplogThread: Operation for this entry is u
2020-10-30 09:58:49,622 [INFO] httpstream:310 - > GET http://localhost:7474/db/data/
2020-10-30 09:58:49,622 [DEBUG] httpstream:316 - > User-Agent: py2neo/2.0.8 HTTPStream/1.5.0 Python/3.6.9-final-0 (linux)
2020-10-30 09:58:49,622 [DEBUG] httpstream:316 - > X-Stream: true
2020-10-30 09:58:49,622 [DEBUG] httpstream:316 - > Authorization: Basic bmVvNGo6cmVkemVwMTA=
2020-10-30 09:58:49,622 [DEBUG] httpstream:316 - > Host: localhost:7474
2020-10-30 09:58:49,657 [INFO] httpstream:511 - < 404 Not Found [0]
2020-10-30 09:58:49,658 [DEBUG] httpstream:514 - < Access-Control-Allow-Origin: *
2020-10-30 09:58:49,658 [DEBUG] httpstream:514 - < Cache-Control: must-revalidate,no-cache,no-store
2020-10-30 09:58:49,658 [DEBUG] httpstream:514 - < Content-Type: text/html;charset=iso-8859-1
2020-10-30 09:58:49,658 [DEBUG] httpstream:514 - < Content-Length: 0
2020-10-30 09:58:49,658 [ERROR] mongo_connector.oplog_manager:310 - Unable to process oplog document {'op': 'u', 'ns': 'modsimv1.dataholons', 'ui': UUID('dd5ead48-d75d-443a-9f57-0e5ec4d81d5c'), 'o': {'_id': '5f77a307e74f033e954fa440', '_t': 'DataHolon', 'name': 'NYC Marshes - Future potential: CRE:', 'description': 'Maps of NY Marshlands, with Ground Penetrating Radar & Salt Ingress timeseries', 'references': '[{"$oid":"5f77a307e74f033e954fa440"}]', 'referencedBy': '[{"$oid":"5f77a307e74f033e954fa440"},{"$oid":"5f875ee96c28101dc2104b15"},{"$oid":"5f89b33a37f31b0752229f8c"}]', 'isaclassificationOf': '[{"$oid":"5f77a307e74f033e954fa441"}]', 'interactsWith': '[{"$oid":"5f77a308e74f033e954fa444"},{"$oid":"5f77a308e74f033e954fa445"},{"$oid":"5f77a308e74f033e954fa446"}]', 'consistsOf': '[{"$oid":"5f77a309e74f033e954fa459"},{"$oid":"5f77a30ae74f033e954fa45a"},{"$oid":"5f77a30ae74f033e954fa45b"},{"$oid":"5f77a30ae74f033e954fa45c"}]', 'usedIn': '[{"$oid":"5f789cf19237286967f29563"}]'}, 'o2': {'_id': '5f77a307e74f033e954fa440'}, 'ts': Timestamp(1604069929, 1), 't': 3, 'wall': datetime.datetime(2020, 10, 30, 14, 58, 49, 620000), 'v': 2}
Traceback (most recent call last):
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 258, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py", line 452, in submit
return Response.wrap(http, uri, self, rs, **response_kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/packages/httpstream/http.py", line 489, in wrap
raise inst
py2neo.packages.httpstream.http.ClientError: 404 Not Found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 94, in update
tx = self.graph.cypher.begin()
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
py2neo.error.GraphError: HTTP GET returned response 404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/oplog_manager.py", line 298, in run
entry["o2"]["_id"], entry["o"], ns, timestamp
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/util.py", line 44, in wrapped
raise new_type(str(exc_value)).with_traceback(exc_tb)
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/home/leveridge/.local/lib/python3.6/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 94, in update
tx = self.graph.cypher.begin()
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/home/leveridge/.local/lib/python3.6/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
mongo_connector.doc_managers.error_handler.Neo4jOperationFailed: HTTP GET returned response 404
2020-10-30 09:58:49,668 [DEBUG] mongo_connector.oplog_manager:322 - OplogThread: Documents removed: 0, inserted: 0, updated: 0 so far
2020-10-30 09:58:49,668 [DEBUG] mongo_connector.oplog_manager:325 - OplogThread: Doc is processed.
2020-10-30 09:58:50,669 [DEBUG] mongo_connector.oplog_manager:338 - OplogThread: updating checkpoint after processing new oplog

2

It seems the problem is the following:

2020-10-30 09:58:49,622 [DEBUG] httpstream:316 - > Host: localhost:7474
2020-10-30 09:58:49,657 [INFO] httpstream:511 - < 404 Not Found [0]
py2neo.packages.httpstream.http.ClientError: 404 Not Found

How are you trying to update the data in Neo4j? Which version of py2neo are you using and which version of Neo4j?

3

Bratanic.. thanks indeed for prompt help :-)

We used mongodb compass that points to localhost and sees our modsimv1 database fine.. I presume and it seems ot bear out.. that when we update a mongo collection in Compass (which is just a front end as we all know) and that updates local mongo.. that inded, yes, the mongo log file is triggered and that in turn is listened to by mongo connector and uses driver py2neo to talk to the doc manager that then "updates" or "produces" and somehow displays the new collection in this case edited collection, automartically in our neo4j browser

5.0b1 is out py2neo version and yep we are on python3
interesting .. neo4j desktop about shows 1.3.8 .. but of course we installed the latest 4.1.1 kernel as per usual wbesite instructions for ubuntu 1804.. said it includes Enterprise 4.1.3 for Devs

if i do a pip list i get py2neo 2020.0.0
as pip3 list returns the same and i get neo4j-doc-manager 1.0.0.dev11 and mongo connector 3.1.1

We are using 4.4.1 as the local mongo and python 3.6.9

mongo-connector thread runs fine :-) finally .. amazed it does so against mongo 4.4.1 as repo sayd only mongo 3.6 supported (but its HELL to get back to 3.6 for mongo LOL) .. but if connector runs then presume its ok.

saw a note that py2neo 5.0b1 is the puppy to use.. as it has the fixes .. since connector or doc manager was initially written for 2.0.7.. 2 yrs ago.

do I have to have 'neo4j' database running ? or create a new database.. same name as my mongo replicaset ? or same name as our mongo modsimv1 db ?

quite a compatibility stack !

best cheers.. hope you can help us.. yep the 404 looks like its some sort of simple connect/ finding issue

4

I solved ensuring our neo4j db was created ! using v 3.5.17 .. no now connection issue from mono-connector.\

BUT Bratanic.. it seems our relationship s in mongo.. objectid lists are not reflected as lines between nodes in graph.. we are diving in.. very sweet ! .. any clues on seeing the relationships ?

cheers!