Error with the Neo4j Python driver : ImportError: sys.meta_path is None, Python is likely shutting down

cobra · June 29, 2023, 6:59am

Hello,

I'm using the Neo4j Python Driver version 5.9 with a Neo4j DBMS version 5.9 and I got the following error at the end of my script. The error happens once my script is finished so everything is fine but I would prefer not to have an error

Exception ignored in: <function Driver.__del__ at 0x000001B9284BAA60>
Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 496, in __del__
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 557, in close
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 483, in close
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 415, in _close_connections
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 906, in close
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 758, in _send_all
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 145, in flush
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\util.py", line 111, in callback
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 837, in _set_defunct_write
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 840, in _set_defunct
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <function Bolt.__del__ at 0x000001B928498310>
Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 169, in __del__
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 906, in close
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 758, in _send_all
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 145, in flush
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\util.py", line 111, in callback
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 837, in _set_defunct_write
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 840, in _set_defunct
ImportError: sys.meta_path is None, Python is likely shutting down

Best regards,
Cobra

rouven_bauer · June 29, 2023, 1:06pm

Interesting

The easiest thing would be to make sure the driver gets closed properly. Something like

import neo4j


with neo4j.GraphDatabase.driver(...) as driver:
    ...  # do the work here

or manually:

import neo4j


driver = neo4j.GraphDatabase.driver(...)
try:
    ...  # do the work here
finally:
    driver.close()

I'm investigating if there's a way to avoid those unclean shut-down errors. As indicated by "Python is likely shutting down", this only happens when the driver is not closed and then garbage collected on interpreter shutdown. This should also emit a deprecation warning (these are supresed by default by Python) because in the future (version 6.0, probably) deleting a driver without closing it will just cause a resource warning without attempting to close the driver like it does right now. So it would then much more behave like Python sockets do. In that version, the reported error would just not occur. So I won't put all too much investigation work into it as 1) there's a workaround: close the driver and 2) in the future it will not happen anymore. But maybe there's a long hanging fruit (i.e., improvement) here.

rouven_bauer · June 29, 2023, 1:10pm

If I interpret that snipped correctly, you only use session as a context manager (with statement). But I don't see what you do with the driver.

cobra · June 29, 2023, 1:11pm

Yeah you right, I saw them instantaneously while re-reading it

Thank you for your help

I just need to close the driver in the right places in my code.

rouven_bauer · June 29, 2023, 1:58pm

Could you kindly try if this PR gets rid of the error (before the change of closing the driver properly)?

github.com/neo4j/neo4j-python-driver

Avoid unclean interpreter shutdown

neo4j:5.0 ← robsdedude:fix-unclean-interpreter-shutdown

opened 01:53PM - 29 Jun 23 UTC

robsdedude

+35 -25

When not closing the driver before shutting down the interpreter, the GC might c…all `__del__` on the driver, which currently (will be changed with 6.0) closes the driver. So any code that's run on driver closure might be execute just before the interpreter shuts down. This can lead to funky errors when trying to import things dynamically. So this should be avoided. This got raised here: https://community.neo4j.com/t/error-with-the-neo4j-python-driver-importerror-sys-meta-path-is-none-python-is-likely-shutting-down/63096/

You can install that PR branch with pip install -U git+https://github.com/robsdedude/neo4j-python-driver.git@fix-unclean-interpreter-shutdown (maybe use a virtualenv for that test to not destroy your dependencies :D). It would be greatly appreciated!

cobra · June 29, 2023, 2:18pm

When I tried, I got multiple errors:

ssl.SSLError: Underlying socket connection gone (_ssl.c:2409)
NameError: name 'open' is not defined
Message: "Failed to write data to connection IPv4Address(('xxxxxxxxxxxxxxxxxxxx', 7687)) (ResolvedIPv4Address(('xxxxxxxxxxxxxxx', 7687)))"

rouven_bauer · June 30, 2023, 9:16am

Thanks a bunch for trying it out! Do you get those errors on shutdown? Or does the driver die right away? Either way, could you provide me with the stack traces? Meanwhile I'll try to reproduce the errors locally myself.

cobra · June 30, 2023, 1:56pm

Hello @rouven_bauer

I guess I got the error on shutdown since I can see the result of the query. Here is the stack trace (I have hidden only the IP and the domain name):

--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 141, in flush
    self.socket.sendall(data)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\network\_bolt_socket.py", line 491, in sendall
    return self._wait_for_io(self._socket.sendall, data)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\network\_bolt_socket.py", line 463, in _wait_for_io
    return func(*args, **kwargs)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\ssl.py", line 1173, in send
    return self._sslobj.write(data)
ssl.SSLError: Underlying socket connection gone (_ssl.c:2409)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\handlers.py", line 69, in emit
    if self.shouldRollover(record):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\handlers.py", line 183, in shouldRollover
    self.stream = self._open()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
NameError: name 'open' is not defined
Call stack:
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 496, in __del__
    self.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 557, in close
    self._pool.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 488, in close
    self._close_connections(connections)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 420, in _close_connections
    connection.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 905, in close
    self._send_all()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 758, in _send_all
    if self.outbox.flush():
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 145, in flush
    Util.callback(self.on_error, error)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\util.py", line 111, in callback
    return cb(*args, **kwargs)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 837, in _set_defunct_write
    self._set_defunct(message, error=error, silent=silent)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 847, in _set_defunct
    log.error(message)
Message: "Failed to write data to connection ResolvedIPv4Address(('***.***.**.***', 7687)) (ResolvedIPv4Address(('***.***.**.***', 7687)))"
Arguments: ()
Failed to write data to connection ResolvedIPv4Address(('***.***.**.***', 7687)) (ResolvedIPv4Address(('***.***.**.***', 7687)))
--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 141, in flush
    self.socket.sendall(data)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\network\_bolt_socket.py", line 491, in sendall
    return self._wait_for_io(self._socket.sendall, data)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\network\_bolt_socket.py", line 463, in _wait_for_io
    return func(*args, **kwargs)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\ssl.py", line 1173, in send
    return self._sslobj.write(data)
ssl.SSLError: Underlying socket connection gone (_ssl.c:2409)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\handlers.py", line 69, in emit
    if self.shouldRollover(record):
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\handlers.py", line 183, in shouldRollover
    self.stream = self._open()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\logging\__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
NameError: name 'open' is not defined
Call stack:
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 496, in __del__
    self.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\driver.py", line 557, in close
    self._pool.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 488, in close
    self._close_connections(connections)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_pool.py", line 420, in _close_connections
    connection.close()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 905, in close
    self._send_all()
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 758, in _send_all
    if self.outbox.flush():
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_common.py", line 145, in flush
    Util.callback(self.on_error, error)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_async_compat\util.py", line 111, in callback
    return cb(*args, **kwargs)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 837, in _set_defunct_write
    self._set_defunct(message, error=error, silent=silent)
  File "C:\Users\Cobra\anaconda3\envs\viziball\lib\site-packages\neo4j\_sync\io\_bolt.py", line 847, in _set_defunct
    log.error(message)
Message: "Failed to write data to connection IPv4Address(('**********', 7687)) (ResolvedIPv4Address(('***.***.**.***', 7687)))"
Arguments: ()
Failed to write data to connection IPv4Address(('**********', 7687)) (ResolvedIPv4Address(('***.***.**.***', 7687)))

Best regards,
Cobra

rouven_bauer · July 3, 2023, 1:45pm

Thanks once more. So now the error moved further down the call stack as there's some logging involved which now fails as the interpreter shuts down. I don't think I can fix that any other way but by stopping to properly shut down the driver in __del__ in 6.0.

I assume you did not run your python script with -X dev (see Python Development Mode — Python 3.11.4 documentation) which among other things enables DeprecationWarnings and ResourceWarnings. Generally that's a good thing to do while developing with Python. In this case it should've given you a clue about the underlying issue (unclosed driver).

cobra · July 3, 2023, 2:06pm

Thank you to you too @rouven_bauer

I did not know the -X dev option, I learned something new

Best regards,
Cobra

Topic		Replies	Views
Where can I find version 1.7 of the python neo4j driver (CANT upgrade beyond neo4j 3.5.x) Python neo4j-driver	9	985	October 2, 2020
Flushing results after closing session in Python neo4j-driver Python	2	1681	September 11, 2018
Trouble connecting to neo4j V4 via python Browser	3	2585	March 11, 2020
Python library issues Neo4j Graph Platform neo4j-python-driver	0	95	February 12, 2025
Need Help with Neo4j and Python Integration: Connection Error and Managing Multiple Databases Neo4j Graph Platform neo4j-driver , pythondriver , neo4j-python-driver	2	215	July 22, 2024

Error with the Neo4j Python driver : ImportError: sys.meta_path is None, Python is likely shutting down

Related topics