Python - generator already executing

Drivers & Stacks Python
1

Hi,
We are using Neo4j driver for python(version 4.2.1).
We have a live system that writes/read from the neo4j cluster all the time, And once in a while we are getting this exception:

Traceback (most recent call last):
  File "main.py", line 85, in on_new_snapshot
    response = customers_list[customer_id]['customer_model_handler_obj']\
  File "/ms_topology_center/../ms_topology_center/customer_model_handler.py", line 130, in on_new_snapshot
    return self.load_last_snapshot_to_graphdb(timestamp=timestamp,
  File "/ms_topology_center/../ms_topology_center/customer_model_handler.py", line 119, in load_last_snapshot_to_graphdb
    self.modeler.create_services('agnostic', region_snapshot_data['created'], account_id, region, timestamp)
  File "/ms_topology_center/../ms_topology_center/modeler.py", line 115, in create_services
    self.graph_db.commit_transaction()
  File "/ms_topology_center/../shared/graph_db.py", line 135, in commit_transaction
    self._transaction.commit()
  File "/usr/local/lib/python3.9/site-packages/neo4j/work/transaction.py", line 134, in commit
    self._connection.fetch_all()
  File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 429, in fetch_all
    detail_delta, summary_delta = self.fetch_message()
  File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 334, in fetch_message
    details, summary_signature, summary_metadata = next(self.inbox)
  File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 79, in __next__
    tag, fields = self.pop()
  File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 73, in pop
    return next(self._messages)
ValueError: generator already executing

After this exception appers we are getting several other very strange exceptions:

  • Failed to read from defunct connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687)))

  • No data

  • Internal server error (Failed to write to closed connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687))))

File "/ms_topology_center/../shared/graph_db.py", line 135, in commit_transaction
    self._transaction.commit()
  File "/usr/local/lib/python3.9/site-packages/neo4j/work/transaction.py", line 133, in commit
    self._connection.send_all()
  File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 296, in send_all
    raise ServiceUnavailable("Failed to write to closed connection {!r} ({!r})".format(
neo4j.exceptions.ServiceUnavailable: Failed to write to closed connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687)))
  • [Errno 9] Bad file descriptor

After I restart the pod everything back to normal.

Any idea why it's happened to us and how can we avoid this kind of errors?

Thanks ,
Dan Yazerski

1 Like
2

Hi @dan3 , Did you find a solution to this problem?

3

I created a GH issue for it:

4

This sounds like a concurrency issue outside the driver. However, without seeing a code sample that is capable of reproducing this bug, I won't be able to say for sure. ValueError: generator already executing is raised when multiple threads try to consume from one generator. Note that the driver object is thread safe, the session objection isn't. You have to spawn (at least) one session per thread.

2 Likes
6

EDIT the now deleted question was about what the following sentence means

You have to spawn (at least) one session per thread.

My phrasing is far from ideal. So better forget about that sentence :sweat_smile:

The key take away is that you can share driver objects across threads, but not sessions, transactions, or result objects.

Here's one example of what not to do.

import concurrent.futures

import neo4j


URL = "neo4j://localhost:7687"
AUTH = ("neo4j", "pass")


def work_shared_driver(driver: neo4j.Driver):
    with driver.session(database="neo4j") as session:
        print("shared driver", session.run("RETURN 1").to_eager_result())


def work_shared_session(session: neo4j.Session):
    print("shared session", session.run("RETURN 1").to_eager_result())


with neo4j.GraphDatabase.driver(URL, auth=AUTH) as driver:
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        # cool, can do!
        futures = [executor.submit(work_shared_driver, driver)
                   for _ in range(5)]
        for future in futures:
            future.result()

        # not cool, can't do!
        with driver.session(database="neo4j") as session:
            futures = [
                # don't share sessions across threads!
                # this can lead to all sorts of hard to decipher errors
                executor.submit(work_shared_session, session)
                for _ in range(5)
            ]
            for future in futures:
                future.result()
1 Like
7

thank you, i started to realize that unique nodes were being added multiple times.