Why does a neo4j database have to rebuild some of its indexes when instantiated in a Docker container running on Linux, but not when the container is running in MacOs?
I have a deployment that involves:
neo4j 5.11.0 Community Edition database
external bind mount to database
Docker
The neo4j database has a number of indexes, including relationship indexes. I make sure that all of the indexes are created completely, using SHOW INDEXES, before exporting the database to the external bind mount. As far as I can tell, no indexes are complete. I even stop the neo4j server before taking a copy of the database.
The Dockerfile is complicated; however, it essentially creates a container that houses the neo4j server that points to the external bind mount; stops the server; does a few tweaks to the server configuration, including setting it to read-only; and then restarts the server.
All of this works without a problem in MacOs. However, whenever I deploy the same setup in Linux, the neo4j server always needs to rebuild at least one of the indexes (usually, a large relationship index), even though I know that all of the indexes were created when I built the database. The need to rebuild an index winds up breaking the deployment in Linux when the Dockerfile script tries to set the database as read-only.
To get around this problem, I added to the Dockerfile scripting a wait loop that repeatedly executes a SHOW INDEXES query and checks for anything still populating. Once all indexes are rebuilt again, the script proceeds with the next steps. Although I'm happy that I solved the problem, I don't understand why the rebuilding was necessary, and why this only happens in the Linux deployment. It's as though neo4j on Linux considers an index that I thought was complete to be in an incomplete state at startup. Maybe a log file indicates something. Wouldn't this be the case for MacOs, too?
I would use the latest release available - you are missing out on lots of new stuff with 5.11
But on to your question. I think this has more to do with how Mac vs. Linux handles the underlying file system. Or you are missing files, or... If you stop the Neo4j service, the copy should work. That being said, doing file operations is fraught at this point. We don't document it. I would recommend doing the work, getting the indexes the way you want them, then stopping the database and then do a database dump and use that instead of copying files.
The solution that I developed has been working consistently, and I no longer experience the problem that I was before.
First, I shut down the source neo4j database before doing anything. Then, I run a script that starts the Docker container.
The Dockerfile calls a shell script.
The shell script has the following function. (NEO4J, NEO4J_USER, and NEO4J_PASSWORD are passed to the script.)
function test_index_population {
# The query will return a string in format 'p x', where x is the number of indexes that are still populating.
# A return of "0" indicates that all indexes have been built.
local test=$(echo 'SHOW INDEXES YIELD state, populationPercent WHERE populationPercent <100 RETURN COUNT(populationPercent) AS p;' | ${NEO4J}/bin/cypher-shell -u ${NEO4J_USER} -p ${NEO4J_PASSWORD})
# Trim the leading "p ".
test=$(echo $test | tr -d 'p ')
echo $test
}
After the shell script starts the neo4j server , it executes the following code:
echo 'Verifying that all indexes have been populated...'
# Initialize the count of indexes that are still populating.
indexpopulating=$(test_index_population)
# The return from test_index_population is interpreted as an integer instead of a string.
until [[ indexpopulating -eq 0 ]]; do
echo $indexpopulating' indexes still populating...'
sleep 5
indexpopulating=$(test_index_population)
done
echo "All indexes populated."
Once the loop terminates, the server is running and all indexes are ready.
I agree that the issue is in how Mac vs. Linux implementations of Docker differ. However, it might be beneficial if the neo4j start command checked that all indexes had finished populating.
We’re running into a similar problem with our vector indexes. Even if we let Neo4j write for once to try and fix them, it still won't start properly when we switch to read-only mode. It keeps trying to rebuild the indexes every time.
