When using the Neo4j-OGM dependency for Java, we run into a problem where the amount of threads keeps increasing until the server doesn't have enough CPU power left to sustain the threads. We think we've narrowed down the issue to the amount of connections to the database. On every request (Servlet), a new connection is opened and never closed. Even with the config specifying the connectionLivenessCheckTimeout and connectionPoolSize. Currently this is how theNeo4jService is setup as a singleton. Below is a screenshot attached showing the amount of connections made to the database.
package com.caraer.Caraer.config;
import com.caraer.Caraer.model.User;
import lombok.Getter;
import org.neo4j.driver.AuthTokens;
import org.neo4j.driver.Config;
import org.neo4j.driver.Driver;
import org.neo4j.driver.GraphDatabase;
import org.neo4j.ogm.config.Configuration;
import org.neo4j.ogm.config.DatabaseSelection;
import org.neo4j.ogm.config.DatabaseSelectionProvider;
import org.neo4j.ogm.drivers.bolt.driver.BoltDriver;
import org.neo4j.ogm.session.Session;
import org.neo4j.ogm.session.SessionFactory;
import org.springframework.security.core.context.SecurityContextHolder;
import java.util.concurrent.TimeUnit;
public class Neo4jService {
private final Session mainDatabaseSession;
private final Session userDatabaseSession;
private final SessionFactory mainFactory;
private final SessionFactory userFactory;
GenerateUuid generateUuid = new GenerateUuid();
public Neo4jService() {
Configuration mainConfiguration = new Configuration.Builder()
.uri("bolt://localhost:7687")
.connectionLivenessCheckTimeout(10)
.connectionPoolSize(50)
.database("neo4j")
.credentials("neo4j", "****")
.build();
Configuration userConfiguration = new Configuration.Builder()
.uri("bolt://localhost:7687")
.connectionLivenessCheckTimeout(10)
.connectionPoolSize(50)
.databaseSelectionProvider(new UserDatabase())
.credentials("neo4j", "****")
.build();
this.mainFactory = new SessionFactory(mainConfiguration, "<packages>");
this.userFactory = new SessionFactory(userConfiguration, "<packages>");
this.mainFactory.register(generateUuid);
this.userFactory.register(generateUuid);
this.mainDatabaseSession = mainFactory.openSession();
this.userDatabaseSession = userFactory.openSession();
}
private static final class InstanceHolder {
private static final Neo4jService instance = new Neo4jService();
}
public static Neo4jService getInstance() {
return InstanceHolder.instance;
}
public Session getMainSession() {
return InstanceHolder.instance.mainDatabaseSession;
}
public Session getUserSession() {
return InstanceHolder.instance.userDatabaseSession;
}
}
class UserDatabase implements DatabaseSelectionProvider {
@Override
public DatabaseSelection getDatabaseSelection() {
User user = (User) SecurityContextHolder.getContext().getAuthentication().getPrincipal();
if(user == null) {
return null;
} else {
return DatabaseSelection.select(user.getDatabase());
}
}
}
On a side note: It is not expected that the Java driver closes connections (as in physical close) but it puts them back into the connection pool to be used again.
Thanks for getting back at me. You're right, I included the browser connections. I ran the same test and after a couple of requests this is what happens.
The connections go above 100 and don't decrease when idle.
When putting breakpoints at the exception that gets thrown when creating a new session I get either two of the following:
org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.extractColumnValue(ExecuteQueriesDelegate.java:236) -> java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
Netty: failed to create a child event loop -> failed to open a new selector
It seems like all these issues are caused by the increase in thread activity and the number of open files. When upgrading the server, it takes longer before these exceptions kick in, but infinitely upgrading is impossible.
This would be awesome. We tried already to come up with something that might reproduce your observations but for the threading part we couldn't see any problem on our side.
Is Neo4j-OGM really necessary for the reproducer or could it get boiled down to the Java driver? Trying to reduce the surface where we need to inspect the problem.
I discovered however that the mistake was somewhere deeper in our own source-code. We've managed to solve it and are now getting response times as expected. Thanks a lot for your help.
