Dockerized Neo4j instance won't find database after import

Integrations Docker
,
1

Hi everyone!

I've been trying for a whole day to set up a docker container with a database created via neo4j-admin import. I do this exactly as the documentation states and found it to behave very inconsistently: The container is created just fine, and the imports seem to work as well, but then the database created may or may not be available (throwing a Database "<DB>" is unavailable, its status is "offline" most of the times, but not every time. I found a myriad of issue reports on the matter, though most of them were not answered. I leave details of my scripts and logs in case anyone could help me.

OS

I'm using Windows 10 with WSL+Docker integration.

Container creation

docker run --interactive --tty -d \
    --name neo4j -p7474:7474 -p7687:7687 \
    --mount type=bind,source="$(pwd)"/data,target=/var/lib/neo4j/data \
    --mount type=bind,source="$(pwd)"/logs,target=/var/lib/neo4j/logs \
    --mount type=bind,source="$(pwd)"/import,target=/var/lib/neo4j/import \
    --mount type=bind,source="$(pwd)"/plugins,target=/var/lib/neo4j/plugins \
    --mount type=bind,source="$(pwd)"/conf,target=/var/lib/neo4j/conf \
    --env NEO4J_AUTH=neo4j/test \
    --env NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
neo4j:4.2.7-enterprise

neo4j.conf

I customized the neo4j.conf to set my DB as default:

...
dbms.default_database=tdr
...

DB Creation and data Import

DB_NAME="tdr"
DB_PATH="data/databases/$DB_NAME"
IMPORT="import/tdr"

# Since neo4j-improt needs a clean DB, I remove any existing data for that pat
rm -fr $DB_PATH

docker exec --interactive --tty neo4j neo4j-admin import \
    --database=$DB_NAME \
    --nodes=Target=$IMPORT/nodes/targets.csv \
    --nodes=Species=$IMPORT/nodes/species.csv \
    --nodes=PFAM=$IMPORT/nodes/pfam.csv \
    --nodes=OMCL=$IMPORT/nodes/orthologs.csv \
    --relationships=HAS_PFAM_DOMAIN=$IMPORT/relationships/has_pfam.csv \
    --skip-bad-relationships

Import seems to be working fine :slight_smile:

IMPORT DONE in 31s 158ms.
Imported:
  644679 nodes
  0 relationships
  2299876 properties
Peak memory usage: 1.002GiB

Restart Container

After data import, I noticed data/databases/tdr had the wrong file ownership, so I tried changing going inside the container and using chown

host > docker exec -it neo4j bash
container > chown -R neo4j:neo4j data/databases/tdr

Then I noticed, by restarting the container, that the Dockerfile is instructed to detect and fix this issue, prompting Warning: Some files inside "/data" are not writable from inside container. Changing folder owner to neo4j.. I figured the expected behavior is to avoid changing ownerships by hand, so I just restart after import.

DatabaseNotFoundError

All seems good, but when I go to the browser and login, the tdr database is not available!

image
image893×353 18.6 KB

It exists and has the right owner and permissions (I guess?):

root@87f75bafdb9b:/var/lib/neo4j# ls -l data/databases/tdr
total 76232
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.counts.db
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.id
-rw-r--r-- 1 neo4j neo4j   409600 Jun 12 03:00 neostore.labelscanstore.db
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.labeltokenstore.db
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.labeltokenstore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.labeltokenstore.db.names
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.labeltokenstore.db.names.id
-rw-r--r-- 1 neo4j neo4j  9707520 Jun 12 03:00 neostore.nodestore.db
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.nodestore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.nodestore.db.labels
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.nodestore.db.labels.id
-rw-r--r-- 1 neo4j neo4j 67108864 Jun 12 03:00 neostore.propertystore.db
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.propertystore.db.arrays
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.propertystore.db.arrays.id
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.propertystore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.propertystore.db.index
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.propertystore.db.index.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.propertystore.db.index.keys
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.propertystore.db.index.keys.id
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.propertystore.db.strings
-rw-r--r-- 1 neo4j neo4j    49152 Jun 12 03:00 neostore.propertystore.db.strings.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.relationshipgroupstore.db
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.relationshipgroupstore.db.id
-rw-r--r-- 1 neo4j neo4j        0 Jun 12 03:00 neostore.relationshipstore.db
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.relationshipstore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.relationshiptypestore.db
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.relationshiptypestore.db.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.relationshiptypestore.db.names
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.relationshiptypestore.db.names.id
-rw-r--r-- 1 neo4j neo4j     8192 Jun 12 03:00 neostore.schemastore.db
-rw-r--r-- 1 neo4j neo4j    40960 Jun 12 03:00 neostore.schemastore.db.id

Can anyone give me a hint of what I'm doing wrong? I spent the whole day and still couldn't figure it out. :confused:

2

Can you try with as simple setup first, one that doesn't use custom mounts etc.

don't delete database folders manually, use --force instead.
or drop the database from the systemdb

after you imported the database physcially you still need to create it logically in the system db
create database tdr

there are bits about the ownership thing in the docs,
but best keep it at the user neo4j

as neo4j uses multiple folders for a database and you're deleting only one of them.

3

Thank you for your response Michael.

I'm gonna try a simpler approach as you suggested, but I wonder: how can I make the import files available without binding those custom volumes? Should I scp into the container? Sounds tricky. Any further help is greatly appreciated :slight_smile:

4

Only by putting them on public URLs.