Neo4J - Client and DB Update Giving Issues

Drivers & Stacks .NET
1

Hey all!

So our application (.NET/C#) was running with a Neo4J 4.4 database.
Right now we are starting to update to one of the newest versions. So far this is what was done:

  • Neo4J DB updated from version 4.4.4 to 5.15.0.
  • All existing DB's migrated to 5 version.
  • .NET Drivers updated to 5.15.0 version.

I've had to fix a few things on the code, specially related with APOC plugin, but besides that I was able to have quite a stable version of our app running with Neo4J 5.15.

The problem I'm having now is when sending a specific request to bolt. I get an exception that does not inform a lot:

Exception Code: Neo.ClientError.Request.InvalidFormat
Message: Neo4j.Driver.ProtocolException: No such statement: 1

In our case, I've noticed this happening when sending in a query that has a list of strings as parameters. This is the code where we build the query:

var query = new Query(queryContext.Statement, queryContext.Parameters);

Where:

  • queryContext.Statement is the following string:

MATCH (Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) WHERE Person.Id IN $startNodeId
OPTIONAL MATCH (Person)-[Person_Person_Has_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]->(Person_Person_Has_Home_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Home)
OPTIONAL MATCH (Person)-[Person_Person_Likes_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Likes]->(Person_Person_Likes_Mate_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Mate)
RETURN
Person {Id: Person.Id,personId: Person.personId,Name: Person.Name} AS N6d6799e474f84dc08bef8e5bb6f792c2 , Person_Person_Has_Home AS R9dcb168c897d42b6bff7556f96cda1af, Person_Person_Has_Home_Home {Id: Person_Person_Has_Home_Home.Id,address: Person_Person_Has_Home_Home.address,Name: Person_Person_Has_Home_Home.Name} AS N261cdfd36d784e0cb13fcec22adbed12 , Person_Person_Likes_Mate AS R63cf8008fab34a34987f3d3d1570015b, Person_Person_Likes_Mate_Mate {Id: Person_Person_Likes_Mate_Mate.Id,Name: Person_Person_Likes_Mate_Mate.Name} AS N5ae0d3b35db14e67844aebf9d74a0209
ORDER BY
toLower(Person.Name), toLower(Person.personId), Person.Id, toLower(Person_Person_Has_Home_Home.Name), toLower(Person_Person_Has_Home_Home.address), Person_Person_Has_Home_Home.Id, toLower(Person_Person_Likes_Mate_Mate.Name), Person_Person_Likes_Mate_Mate.Id

  • queryContext.Parameters is a Dictionary<string,object> in this case Dictionary<string, List> where it only contains a value -> the key is startNodeId and the value is a list of strings.

We then run this as following:

var resultStream = await unitOfWork.Transaction.RunAsync(query);

So I'm trying to understand why do I get this error. With latest 4.4 driver and latest Neo4J version we had no issues at all. Could it be because of the WHERE IN clause and parameters related to it?

If I manually run this query on Neo4J browser I have no issues. I just change the parameter $startNodeId with ("id1", "id2") and all works.

Would really appreciate some help.
Thank you!
Regards

2

Hi @ricardomota

  • Do you have any logs beyond the exception?
  • Can I see the code on how this query is run, my first thought is this looks like a concurrency issue.
3

Hey @grant_lodge !

No logs besides what I've gave. This is basically it:

warn: SemantX.Neo4JDriver.Neo4JLogger[0]
[bolt-367] A bolt protocol error has occurred with server bolt://localhost:7687/, connection will be terminated.
Neo4j.Driver.ProtocolException: No such statement: 1
at Neo4j.Driver.Internal.MessageHandling.ResponsePipelineError.EnsureThrownIfT
at Neo4j.Driver.Internal.MessageHandling.ResponsePipeline.AssertNoProtocolViolation()
at Neo4j.Driver.Internal.Connector.SocketClient.ReceiveOneAsync(IResponsePipeline responsePipeline)
warn: SemantX.Neo4JDriver.Neo4JLogger[0]
[bolt-367] Unable to send message to server bolt://localhost:7687/, connection will be terminated.
System.ObjectDisposedException: Cannot access a closed Stream.
Object name: 'NetworkStream'.

I've created a replica of the code that is failing. Yes, you might be right on being a concurrency issue. This is the code we run:

      var startNodeIds = new List<string>()
        {
            "5d8707d3-89ad-4975-9dd0-8b48d7f858c7",
            "4d8707d3-89ad-4975-9dd0-8b48d7f858c3"
        };

        await using var unitOfWork = await _unitOfWorkFactory.CreateOpenRead(
            context, 
            getTemplate.Result.Id,
            projectId,
            $"{nameof(DataHub)}_{nameof(DoStuff)}");
        
        await Parallel.ForEachAsync(startNodeIds, async (startNodeId, _) =>
        {
            var queryStatement = "MATCH (Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) " +
                                 "WHERE Person.Id IN $startNodeId " +
                                 "OPTIONAL MATCH (Person)-[Person_Person_Has_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]->(Person_Person_Has_Home_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Home) " +
                                 "OPTIONAL MATCH (Person)-[Person_Person_Likes_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Likes]->(Person_Person_Likes_Mate_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Mate) " +
                                 "RETURN " +
                                 "Person {Id: Person.Id,personId: Person.personId,Name: Person.Name} AS N6d6799e474f84dc08bef8e5bb6f792c2, Person_Person_Has_Home AS R9dcb168c897d42b6bff7556f96cda1af, Person_Person_Has_Home_Home {Id: Person_Person_Has_Home_Home.Id,address: Person_Person_Has_Home_Home.address,Name: Person_Person_Has_Home_Home.Name} AS N261cdfd36d784e0cb13fcec22adbed12, Person_Person_Likes_Mate AS R63cf8008fab34a34987f3d3d1570015b, Person_Person_Likes_Mate_Mate {Id: Person_Person_Likes_Mate_Mate.Id,Name: Person_Person_Likes_Mate_Mate.Name} AS N5ae0d3b35db14e67844aebf9d74a0209 " +
                                 "ORDER BY toLower(Person.Name), toLower(Person.personId), Person.Id, toLower(Person_Person_Has_Home_Home.Name), toLower(Person_Person_Has_Home_Home.address), Person_Person_Has_Home_Home.Id, toLower(Person_Person_Likes_Mate_Mate.Name), Person_Person_Likes_Mate_Mate.Id";

            var queryParameters = new Dictionary<string, object>()
            {
                { "startNodeId", new List<string>() { startNodeId } }
            };
            
            var resultStream = await unitOfWork.Transaction.RunAsync(
                new Query(queryStatement, queryParameters));
            
            await resultStream.ToListAsync(async (record) =>
            {
                // Do stuff with Data
                return record;
            });
        });

        unitOfWork.NoCommit();

So we are executing two Neo4J transactions on a Parallel.ForEachAsync. This seems to cause some issues on the Neo4J driver side. I was checking the driver code on Github and I've noticed some volatile variables (called _thrown) that could be causing some concurrency/race condition issues.

Keep in mind that if I use Neo4J 4.4 driver I'm having no issues. This is the first time I'm having this issue - as soon as I've updated to Neo4J 5.15.0 driver. On another note, it is strange that sometimes this code runs without issues and sometimes it just works.

Not sure if it helps but here is also my configuration for Neo4J 5 Docker:

docker run --name "$NEO4J_CONTAINER_NAME"
-p 7474:7474 -p 7687:7687
-d
-v C:/Docker/Neo4J5/data:/data:z
-v C:/Docker/Neo4J5/logs:/logs:z
-v C:/Docker/Neo4J5/plugins:/plugins:z
--env NEO4J_AUTH=neo4j/12345678
--env NEO4J_server_memory_pagecache_size=6G
--env NEO4J_server_memory_heap_initial__size=6G
--env NEO4J_server_memory_heap_max__size=8G
--env NEO4J_dbms_ssl_policy_bolt_client__auth=NONE
--env NEO4J_dbms_ssl_policy_https_client__auth=NONE
--env NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
--env NEO4J_apoc_export_file_enabled=true
--env NEO4J_apoc_import_file_enabled=true
--env NEO4J_apoc_import_file_use__neo4j__config=true
--env NEO4J_PLUGINS='["apoc", "apoc-extended"]'
neo4j:5.15.0

4

what does _unitOfWorkFactory.CreateOpenRead do?

5

@ricardomota
could you run your test with this logger and send me back the output, it will give me a really good hint at what is happening.

public class MessageLog : ILogger
{
    public void Error(Exception cause, string message, params object[] args)
    {
        Console.WriteLine(message, args);
    }

    public void Warn(Exception cause, string message, params object[] args)
    {
        Console.WriteLine(message, args);
    }

    public void Info(string message, params object[] args)
    {
        Console.WriteLine(message, args);
    }

    public void Debug(string message, params object[] args)
    {
        var text = string.Format(message, args);
        if (!text.StartsWith("[bolt-") || text.Contains("RECORD"))
            return;
        Console.WriteLine(text);
    }

    public void Trace(string message, params object[] args)
    {
    }

    public bool IsTraceEnabled()
    {
        return false;
    }

    public bool IsDebugEnabled()
    {
        return true;
    }
}

this is added to the driver like so:

await using var driver = GraphDatabase.Driver("bolt://localhost:7687", AuthTokens.Basic("neo4j", "password"),
    x => x.WithLogger(new MessageLog()));

Obviously make sure that you check nothing sensitive is included.

6

Hey @grant_lodge

Here are the logs as requested (they do not inform that much when compared to the ones I've sent before):

bolt-9] C: BEGIN [{tx_metadata, [{Database, MyDatabase}, {AuthenticationType, UserToService}, {TemplateId, c19143d7-9465-43eb-8692-37165c7b9d18}, {ProjectId, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}, {OurOwnTransactionId, f0a601f4-a7e7-41f7-898e-6bc64a4d30b4}, {Intent, DataNodeDeleted_DatabaseUpdater_1}, {TenantUser, admin}, {Mode, Unknown}]}, {db, MyDatabase}]
[bolt-9] S: SUCCESS
[bolt-10] S: SUCCESS
[bolt-10] C: BEGIN [{tx_metadata, [{Database, MyDatabase}, {AuthenticationType, UserToService}, {TemplateId, 00000000-0000-0000-0000-000000000000}, {ProjectId, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}, {OurOwnTransactionId, 3fee256b-86d4-4977-b78a-92421d2de0fa}, {Intent, ProjectProvider_Get}, {TenantUser, admin}, {Mode, Unknown}]}, {db, MyDatabase}]
[bolt-10] S: SUCCESS
[bolt-10] C: RUN MATCH (node:Project { Id : $node_id }) RETURN node {.*} , [{node_id, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}]
[bolt-10] C: PULL [{n, 1000}]
[bolt-10] S: SUCCESS [{t_first, 2}, {fields, [node]}, {qid, 0}]
[bolt-10] S: SUCCESS [{type, r}, {t_last, 1}, {db, MyDatabase}]
[bolt-10] C: ROLLBACK
[bolt-10] S: SUCCESS
[bolt-10] C: RESET
[bolt-10] S: SUCCESS
[bolt-9] C: RUN MATCH (Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Home) WHERE Home.Id = $p_b7e6d2280b404f3caca6dc76fda4fa49 OPTIONAL MATCH (Home)<-[Home_Home_Has_Person_reverse:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]-(Home_Home_Has_Person_reverse_Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) RETURN Home {Id: Home.Id,Name: Home.Name} AS Nbf4910d08b724c2a924f7565b8c473d7 , Home_Home_Has_Person_reverse AS R77ad095b6fde4605874c1c5bf01310f4_reverse, Home_Home_Has_Person_reverse_Person {Id: Home_Home_Has_Person_reverse_Person.Id,Name: Home_Home_Has_Person_reverse_Person.Name} AS N83081b180e844c3ebddd981f9f4a2fa4 ORDER BY toLower(Home.Name), Home.Id, toLower(Home_Home_Has_Person_reverse_Person.Name), Home_Home_Has_Person_reverse_Person.Id , [{p_b7e6d2280b404f3caca6dc76fda4fa49, f753323b-f80f-4a0f-a19b-b23b85d2b97a}]
[bolt-9] C: PULL [{n, 1000}]
[bolt-9] S: SUCCESS [{t_first, 98}, {fields, [Nbf4910d08b724c2a924f7565b8c473d7, R77ad095b6fde4605874c1c5bf01310f4_reverse, N83081b180e844c3ebddd981f9f4a2fa4]}, {qid, 0}]
[bolt-9] S: SUCCESS [{type, r}, {t_last, 3}, {db, MyDatabase}]
[bolt-9] C: ROLLBACK
[bolt-9] S: SUCCESS
[bolt-9] C: RESET
[bolt-9] S: SUCCESS
[bolt-10] C: BEGIN [{tx_metadata, [{Database, MyDatabase}, {AuthenticationType, UserToService}, {TemplateId, 00000000-0000-0000-0000-000000000000}, {ProjectId, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}, {OurOwnTransactionId, 40f22258-2761-4a48-9d63-736a1f8d0b9f}, {Intent, ProjectProvider_Get}, {TenantUser, admin}, {Mode, Unknown}]}, {db, MyDatabase}]
[bolt-10] S: SUCCESS
[bolt-10] C: RUN MATCH (node:Project { Id : $node_id }) RETURN node {.*} , [{node_id, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}]
[bolt-10] C: PULL [{n, 1000}]
[bolt-10] S: SUCCESS [{t_first, 2}, {fields, [node]}, {qid, 0}]
[bolt-10] S: SUCCESS [{type, r}, {t_last, 3}, {db, MyDatabase}]
[bolt-10] C: ROLLBACK
[bolt-10] S: SUCCESS
[bolt-10] C: RESET
[bolt-10] S: SUCCESS
[bolt-9] C: BEGIN [{tx_metadata, [{Database, MyDatabase}, {AuthenticationType, UserToService}, {TemplateId, c19143d7-9465-43eb-8692-37165c7b9d18}, {ProjectId, ce425b71-4aed-4f5c-ad91-30dc9f1ebd3d}, {OurOwnTransactionId, 530d89b3-b203-44be-bbf4-905de9b1ba41}, {Intent, DataHub_RequestRootRows}, {TenantUser, admin}, {Mode, Unknown}]}, {mode, r}, {db, MyDatabase}]
[bolt-9] S: SUCCESS
[bolt-9] C: RUN MATCH (Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) WHERE Person.Id IN $startNodeId OPTIONAL MATCH (Person)-[Person_Person_Has_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]->(Person_Person_Has_Home_Home:Pce425b714aed4f5cad91 30dc9f1ebd3d_Home) OPTIONAL MATCH (Person)-[Person_Person_Likes_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Likes]->(Person_Person_Likes_Mate_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Mate) RETURN Person {Id: Person.Id,personId: Person.personId,Name: Person.Name} AS N6d6799e474f84dc08bef8e5bb6f792c2, Person_Person_Has_Home AS R9dcb168c897d42b6bff7556f96cda1af, Person_Person_Has_Home_Home {Id: Person_Person_Has_Home_Home.Id,address: Person_Person_Has_Home_Home.address,Name: Person_Person_Ha s_Home_Home.Name} AS N261cdfd36d784e0cb13fcec22adbed12, Person_Person_Likes_Mate AS R63cf8008fab34a34987f3d3d1570015b, Person_Person_Likes_Mate_Mate {Id: Person_Person_Likes_Mate_Mate.Id,Name: Person_Person_Likes_Mate_Mate.Name} AS N5ae0d3b35d b14e67844aebf9d74a0209 ORDER BY toLower(Person.Name), toLower(Person.personId), Person.Id, toLower(Person_Person_Has_Home_Home.Name), toLower(Person_Person_Has_Home_Home.address), Person_Person_Has_Home_Home.Id, toLower(Person_Person_Likes_Mate_Mate.Name), Person_Person_Likes_Mate_Mate.Id, [{startNodeId, [5d8707d3-89ad-4975-9dd0-8b48d7f858c7]}]
[bolt-9] C: RUN MATCH (Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) WHERE Person.Id IN $startNodeId OPTIONAL MATCH (Person)-[Person_Person_Has_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]->(Person_Person_Has_Home_Home:Pce425b714aed4f5cad91 30dc9f1ebd3d_Home) OPTIONAL MATCH (Person)-[Person_Person_Likes_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Likes]->(Person_Person_Likes_Mate_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Mate) RETURN Person {Id: Person.Id,personId: Person.personId,Name: Person.Name} AS N6d6799e474f84dc08bef8e5bb6f792c2, Person_Person_Has_Home AS R9dcb168c897d42b6bff7556f96cda1af, Person_Person_Has_Home_Home {Id: Person_Person_Has_Home_Home.Id,address: Person_Person_Has_Home_Home.address,Name: Person_Person_Ha s_Home_Home.Name} AS N261cdfd36d784e0cb13fcec22adbed12, Person_Person_Likes_Mate AS R63cf8008fab34a34987f3d3d1570015b, Person_Person_Likes_Mate_Mate {Id: Person_Person_Likes_Mate_Mate.Id,Name: Person_Person_Likes_Mate_Mate.Name} AS N5ae0d3b35d b14e67844aebf9d74a0209 ORDER BY toLower(Person.Name), toLower(Person.personId), Person.Id, toLower(Person_Person_Has_Home_Home.Name), toLower(Person_Person_Has_Home_Home.address), Person_Person_Has_Home_Home.Id, toLower(Person_Person_Likes_Mate_Mate.Name), Person_Person_Likes_Mate_Mate.Id, [{startNodeId, [5d8707d3-89ad-4975-9dd0-8b48d7f858c7]}]
[bolt-9] C: PULL [{n, 1000}]
[bolt-9] C: PULL [{n, 1000}]
[bolt-9] S: SUCCESS [{t_first, 333}, {fields, [N6d6799e474f84dc08bef8e5bb6f792c2, R9dcb168c897d42b6bff7556f96cda1af, N261cdfd36d784e0cb13fcec22adbed12, R63cf8008fab34a34987f3d3d1570015b, N5ae0d3b35db14e67844aebf9d74a0209]}, {qid, 0}]
[bolt-9] S: SUCCESS [{t_first, 2}, {fields, [N6d6799e474f84dc08bef8e5bb6f792c2, R9dcb168c897d42b6bff7556f96cda1af, N261cdfd36d784e0cb13fcec22adbed12, R63cf8008fab34a34987f3d3d1570015b, N5ae0d3b35db14e67844aebf9d74a0209]}, {qid, 1}]
[bolt-9] S: SUCCESS [{type, r}, {t_last, 5}, {db, MyDatabase}]
[bolt-9] S: FAILURE code=Neo.ClientError.Request.InvalidFormat, message=No such statement: 1
[bolt-9] A bolt protocol error has occurred with server bolt://localhost:7687/, connection will be terminated.
[bolt-9] C: GOODBYE
[bolt-9] Unable to send message to server bolt://localhost:7687/, connection will be terminated.

Also I have removed the unit of work and just used what we use on it and we have the same issue (once again, keep in mind that with driver 4.4 all of this was working). Here is the code:

var startNodeIds = new List()
{
"5d8707d3-89ad-4975-9dd0-8b48d7f858c7",
"3d8707d3-89ad-4975-9dd0-8b48d7f858c4"
};
var driver = GraphDatabase.Driver("bolt://localhost:7687",
AuthTokens.Basic("neo4j", "12345678"), configBuilder =>
{
configBuilder.WithLogger(new MessageLog());
configBuilder.WithMaxConnectionPoolSize(150);
configBuilder.WithMaxIdleConnectionPoolSize(150);
configBuilder.WithConnectionIdleTimeout(new TimeSpan(0, 0, 10));
});
var session = driver.AsyncSession(c =>
{
c.WithDatabase("MyDatabase");
});
var transaction = await session.BeginTransactionAsync();
await Parallel.ForEachAsync(startNodeIds, async (startNodeId, _) =>
{
var queryStatement = "MATCH (Person:Pce425b714aed4f5cad9130dc9f1ebd3d_Person) " +
"WHERE Person.Id IN $startNodeId " +
"OPTIONAL MATCH (Person)-[Person_Person_Has_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Has]->(Person_Person_Has_Home_Home:Pce425b714aed4f5cad9130dc9f1ebd3d_Home) " +
"OPTIONAL MATCH (Person)-[Person_Person_Likes_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Likes]->(Person_Person_Likes_Mate_Mate:Pce425b714aed4f5cad9130dc9f1ebd3d_Mate) " +
"RETURN " +
"Person {Id: Person.Id,personId: Person.personId,Name: Person.Name} AS N6d6799e474f84dc08bef8e5bb6f792c2, Person_Person_Has_Home AS R9dcb168c897d42b6bff7556f96cda1af, Person_Person_Has_Home_Home {Id: Person_Person_Has_Home_Home.Id,address: Person_Person_Has_Home_Home.address,Name: Person_Person_Has_Home_Home.Name} AS N261cdfd36d784e0cb13fcec22adbed12, Person_Person_Likes_Mate AS R63cf8008fab34a34987f3d3d1570015b, Person_Person_Likes_Mate_Mate {Id: Person_Person_Likes_Mate_Mate.Id,Name: Person_Person_Likes_Mate_Mate.Name} AS N5ae0d3b35db14e67844aebf9d74a0209 " +
"ORDER BY toLower(Person.Name), toLower(Person.personId), Person.Id, toLower(Person_Person_Has_Home_Home.Name), toLower(Person_Person_Has_Home_Home.address), Person_Person_Has_Home_Home.Id, toLower(Person_Person_Likes_Mate_Mate.Name), Person_Person_Likes_Mate_Mate.Id";
var queryParameters = new Dictionary<string, object>()
{
{ "startNodeId", new List() { startNodeId } }
};
var resultStream = await transaction.RunAsync(
new Query(queryStatement, queryParameters));
await resultStream.ToListAsync(async (record) =>
{
// Do stuff with Data
return record;
});
});
// No Commit
await transaction.RollbackAsync();

7

Here are the logs as requested (they do not inform that much when compared to the ones I've sent before):

For me they show me exactly what is breaking.

[bolt-9] C: PULL [{n, 1000}]
[bolt-9] C: PULL [{n, 1000}]

is the issue, I will look into this as this is causing the breakage.

If you run the same logging for 4.4, I would expect that you get

RUN 
PULL
RUN
PULL

correctly sequenced.

This will be fixed in 5.18 which will be released this week.

8

@ricardomota I hope you saw, 5.18.0 of the driver was released yesterday, it includes a fix for this issue.

9

Hey @grant_lodge !

Thank you very much for this.
I will have a look asap

Regards

10

Hey @grant_lodge !

I've done a few checks and can confirm that so far this issue seems to have been fixed.

Many thanks

11

I'm glad to hear that!