Thoughts on large api execution with FASTAPI getting slow response from Aura

I would be interested in thoughts on my setup. I have been running neo4j on community edition with no speed issues but recently moved to a professional (lowest tier) AuraDB instance and things seem very slow and i am trying to work out why.

I am using fastapi and manage connections with a class which i call from a helper function as below. This has worked well up to now but since moving to Aura I am wondering if there is. a better way to manage the connection pool?

def run_query(query, params=""):

    neo4j_conn = Neo4jConnect(General.BOLT_URL, General.BOLT_DEFAULT_USER, General.BOLT_DATABASE_PASSWORD)

    try:
        result = neo4j_conn.run_query_with_retries(query, params)

    except exceptions.ServiceUnavailable as e:
        result = f"Query failed after multiple retries: {e}"

    neo4j_conn.close()
    return result

c

lass Neo4jConnect:
    def __init__(self, uri, user, password):
        self.driver = GraphDatabase.driver(uri, auth=basic_auth(user, password),
                                           connection_timeout=60.0, connection_acquisition_timeout=120.0)

    def close(self):
        self.driver.close()

    def run_query_with_retries(self, query, parameters, max_retries=5, backoff_factor=0.5):
        for attempt in range(max_retries):
            try:
                with self.driver.session() as session:
                    result = session.run(query, parameters=parameters)
                    return result.data()
            except exceptions.ServiceUnavailable as e:
                if attempt < max_retries - 1:
                    sleep_time = backoff_factor * (2 ** attempt)
                    print(f"Attempt {attempt + 1} failed, retrying in {sleep_time} seconds...")
                    time.sleep(sleep_time)
                else:
                    print("Max retries reached, raising exception")
                    raise e

If possible, you should keep a single driver instance throughout the whole lifetime of the application. The driver already implements a connection pool.

Further, you should know that session.run is not safe to retry on connectivity errors unless your queries are idempotent. That's because this API is like SQL's auto-commit. You get no guarantee whether or whether not the transaction has been committed until you've successfully consumed the whole record stream at which point you know it has been successfully committed.

Also note that there are APIs in the driver that come with retry functionality out of the box. E.g.,

Further, if you want to roll your own retry mechanism (only viable for transaction-style APIs, not for session.run), there's Neo4jError.is_retryable() and DriverError.is_retryable() that you might find helpful.

1 Like

Thank you for this, it has made me realise what I am doing wrong. I am doing a refactor to improve the way i am handling the connections

Hi,

I have update so I now use a single connection pool (hopefully). All seems to work correctly most of the time but I occasionally see this error occurring?

2024-08-12 13:05:51,730 - DEBUG - [#0000]  _: <WORKSPACE> resolve home database
2024-08-12 13:05:51,730 - DEBUG - [#0000]  _: <POOL> attempting to update routing table from IPv4Address(('317a0615.databases.neo4j.io', 7687))
2024-08-12 13:05:51,730 - DEBUG - [#0000]  _: <RESOLVE> in: 317a0615.databases.neo4j.io:7687
2024-08-12 13:05:51,744 - DEBUG - [#0000]  _: <RESOLVE> dns resolver out: 34.105.131.232:7687
2024-08-12 13:05:51,744 - DEBUG - [#0000]  _: <POOL> _acquire router connection, database=None, address=ResolvedIPv4Address(('34.105.131.232', 7687))
2024-08-12 13:05:51,745 - DEBUG - [#BE8E]  _: <POOL> picked existing connection bolt-2151
2024-08-12 13:05:51,745 - DEBUG - [#BE8E]  _: <POOL> checked re_auth auth=None updated=False force=False
2024-08-12 13:05:51,745 - DEBUG - [#BE8E]  _: <POOL> handing out existing connection
2024-08-12 13:05:51,745 - DEBUG - [#BE8E]  C: ROUTE {'address': '317a0615.databases.neo4j.io:7687'} () {}
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
2024-08-12 13:05:51,746 - DEBUG - [#BE8E]  _: <CONNECTION> error: ConnectionResetError(104, 'Connection reset by peer')
2024-08-12 13:05:51,746 - ERROR - Failed to read from defunct connection ResolvedIPv4Address(('34.105.131.232', 7687)) (ResolvedIPv4Address(('34.105.131.232', 7687)))
2024-08-12 13:05:51,746 - DEBUG - [#BE8E]  C: <CLOSE>
2024-08-12 13:05:51,746 - DEBUG - [#0000]  _: <POOL> deactivating address ResolvedIPv4Address(('34.105.131.232', 7687))
2024-08-12 13:05:51,747 - DEBUG - [#0000]  _: <POOL> table={'neo4j': RoutingTable(database='neo4j' routers={IPv4Address(('p-317a0615-56a7-0001.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0002.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0003.production-orch-0010.neo4j.io', 7687))}, readers={IPv4Address(('p-317a0615-56a7-0002.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0001.production-orch-0010.neo4j.io', 7687))}, writers={IPv4Address(('p-317a0615-56a7-0003.production-orch-0010.neo4j.io', 7687))}, last_updated_time=9926518.324638003, ttl=10), None: RoutingTable(database=None routers={IPv4Address(('317a0615.databases.neo4j.io', 7687))}, readers={}, writers={}, last_updated_time=9926516.595586564, ttl=0)}
2024-08-12 13:05:51,747 - DEBUG - [#BE8E]  _: <POOL> released bolt-2151
2024-08-12 13:05:51,747 - DEBUG - [#0000]  _: <POOL> failed to fetch routing info from ResolvedIPv4Address(('34.105.131.232', 7687))
2024-08-12 13:05:51,747 - DEBUG - [#0000]  _: <POOL> deactivating address IPv4Address(('317a0615.databases.neo4j.io', 7687))
2024-08-12 13:05:51,747 - DEBUG - [#0000]  _: <POOL> table={'neo4j': RoutingTable(database='neo4j' routers={IPv4Address(('p-317a0615-56a7-0001.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0002.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0003.production-orch-0010.neo4j.io', 7687))}, readers={IPv4Address(('p-317a0615-56a7-0002.production-orch-0010.neo4j.io', 7687)), IPv4Address(('p-317a0615-56a7-0001.production-orch-0010.neo4j.io', 7687))}, writers={IPv4Address(('p-317a0615-56a7-0003.production-orch-0010.neo4j.io', 7687))}, last_updated_time=9926518.324638003, ttl=10), None: RoutingTable(database=None routers={}, readers={}, writers={}, last_updated_time=9926516.595586564, ttl=0)}
2024-08-12 13:05:51,747 - ERROR - Unable to retrieve routing information

Hi :thinking:

Are you solely using APIs with built-in retries or wrapping all driver calls in your own retry mechanism? Because this error is fine to retry on.

To reduce the frequency with which the error occurs, you could try to play around with different timeout configs in the driver:

1 Like