Hi, I am spinning up a docker container to run neo4j. I need to spin up the container and after it has started I would like to be able to run some cypher code to load a json file and run a couple of other queries in order to initialise the database so that when the browser is opened the initial graph is presented, is this possible?
I have read somewhere that there is an environment tag called EXTENSION_SCRIPT that can be used to execute scripts but I dont know if this the answer as the little test that I did with a simple 'echo' statement script appeared to be executed before the container had started as I see my message echoed to the the output in console window before I see statements about starting and then started.
This is still all very new to me, both neo4j and docker, so any pointers would really help.
Thank you for your assistance, this is very helpful but I dont think it quite addresses my requirement as I wont have the option of being able to build the image.
A bit more detail may help. I have an application that will generate a json file as the output of a task. This generated output will change every time that it is generated. Once this json has been generated I then need to spin up an instance of neo4j using docker and once the container is running then load this json into the default neo4j database. I have also just had the requirement added to be able to add any number of specified query files to the 'Favourites' list available to neo4j browser at the same time (that seems like a long shot to me but that remains to be seen).
Hi @andy_mcshane , mine is an example of how to use a csv file on the disk and have it load into a docker. you can change the csv into json file of your choice.
i would recommend you to use Apache Airflow or other orchestration software's and devops to build the workflow.
Apache airflow can trigger jobs and whenever Jenkins or something like Travis CI CD outputs a file.
For favorites - you have 2 options - 1. Sync Neo4j with github 2. Create a custom autostart play when opening the browser.
Thank you replying so quickly. I have followed your blog and It now makes more sense to me and I cannot really see that much of a downside to building the container each time as it doesn't take long so I will have to argue that one out with the powers that be. I do still have a problem though as even after following your guide I cannot get my container to run, (I suspect it is at the point of attempting to run the cypher script?), as I get this error:
Unable to connect to localhost:7687, ensure the database is running and that there is a working network connection to it.
maybe a permissions error or is it because of trying to run cypher-shell in localhost? I have no idea!
are you connecting from outside or inside the container ? if outside, I haven't exposed the ports.
You can access the ports either 3 ways, (1) use [EXPOSE] in the Dockerfile (Dockerfile reference | Docker Documentation), (2) start the docker with -p in the command line -> (neo4js is my built image name) (3) create a docker-compose file and add ports
docker run -it -d -p7474:7474 -p7687:7687 -p7473:7473 neo4j:seed
Hi, I am wondering if it is because I am trying to use a docker-compose file and dockerfile together?
I have copied your dockerfile and only changed the import file name so currently have this
FROM neo4j
ENV NEO4J_HOME="/var/lib/neo4j" \
NEO4J_PASSWD=neo4j_seed
COPY Dependencies_Tree_Full_UnSorted.json ${NEO4J_HOME}/import/
COPY data_loader.cypher ${NEO4J_HOME}/import/
# set initial-password to start loading the data
# sleep for 10 secs for neo4j to start without any overlapping
CMD bin/neo4j-admin set-initial-password ${NEO4J_PASSWD} && \
bin/neo4j start && sleep 30 && \
if [ -f "${NEO4J_HOME}/import/data_loader.cypher" ]; then \
cat ${NEO4J_HOME}/import/data_loader.cypher | NEO4J_USERNAME=neo4j NEO4J_PASSWORD=${NEO4J_PASSWD} bin/cypher-shell --fail-fast && rm ${NEO4J_HOME}/import/*; \
fi && /bin/bash
I am then trying to execute this via a docker-compose file that I have been previously using but added the 'build: .' command. This is my docker-compose file that was working:
version: '3.4'
services:
neo4j:
container_name: neo4j_custom
image: neo4j:4.2.2
restart: unless-stopped
build: .
ports:
- 7474:7474
- 7687:7687
volumes:
# Set volumes and paths as required
#- ./conf:/conf
#- ./data:/data
#- ./import:/import
#- ./logs:/logs
#- ./plugins:/plugins
- D:\Neo4j\conf:/conf
- D:\Neo4j\data:/data
- D:\Neo4j\import:/import
- D:\Neo4j\logs:/logs
- D:\Neo4j\plugins:/plugins
environment:
# Authentication - temporsry default used will need changing for deployment
- NEO4J_AUTH=neo4j/neo4j_seed
# Raise memory limits
- NEO4J_dbms_memory_pagecache_size=1G
- NEO4J_dbms.memory.heap.initial_size=1G
- NEO4J_dbms_memory_heap_max__size=1G
# Install plugins as an array of items
- NEO4JLABS_PLUGINS=["apoc"]
# Enable APOC to import files
- NEO4J_apoc_import_file_enabled=true
I am guessing that I have conflicts between the docker-compose file and dockerfile?
Again, thanks for your patience, as frustrating as this learning curve is I am quite enjoying it, I am getting there slowly!
Yes they are. I have tried running the dockerfile directly on its own using
docker run -it -d -p7474:7474 -p7687:7687 neo4j:seed
but then at startup I am back to the error:
`Changed password for user 'neo4j'.`
`Unable to connect to localhost:7687, ensure the database is running and that there is a working network connection to it.`
Yes the file is just called 'dockerfile', no extension.
Building neo4j
Step 1/5 : FROM neo4j:4.2.2
---> 9edee9e153ab
Step 2/5 : ENV NEO4J_HOME="/var/lib/neo4j" NEO4J_PASSWD=neo4j_dependency
---> Using cache
---> 4a726ca36728
Step 3/5 : COPY Dependencies_Tree_Full_UnSorted.json ${NEO4J_HOME}/import/
---> Using cache
---> c4bf9966c2e5
Step 4/5 : COPY data_loader.cypher ${NEO4J_HOME}/import/
---> Using cache
---> 8e4b5613b8f9
Step 5/5 : CMD bin/neo4j start && sleep 10 && if [ -f "${NEO4J_HOME}/import/data_loader.cypher" ]; then cat ${NEO4J_HOME}/import/data_loader.cypher | NEO4J_USERNAME=neo4j NEO4J_PASSWORD=${NEO4J_PASSWD} bin/cypher-shell --fail-fast && rm ${NEO4J_HOME}/import/*; fi && /bin/bash
---> Running in b3985159a9d3
Removing intermediate container b3985159a9d3
---> 26e860ffa4cf
Successfully built 26e860ffa4cf
Successfully tagged deltafsdatamigrationdependency_neo4j:latest
Recreating neo4j_custom ... done
Attaching to neo4j_custom
neo4j_custom | grep: /var/lib/neo4j/conf/neo4j.conf: No such file or directory
neo4j_custom | Fetching versions.json for Plugin 'apoc' from https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
neo4j_custom | Installing Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.1/apoc-4.2.0.1-all.jar to /plugins/apoc.jar
neo4j_custom | Applying default values for plugin apoc to neo4j.conf
neo4j_custom | Directories in use:
neo4j_custom | home: /var/lib/neo4j
neo4j_custom | config: /var/lib/neo4j/conf
neo4j_custom | logs: /logs
neo4j_custom | plugins: /plugins
neo4j_custom | import: /import
neo4j_custom | data: /var/lib/neo4j/data
neo4j_custom | certificates: /var/lib/neo4j/certificates
neo4j_custom | run: /var/lib/neo4j/run
neo4j_custom | Starting Neo4j.
neo4j_custom | Started neo4j (pid 329). It is available at http://localhost:7474/
neo4j_custom | There may be a short delay until the server is ready.
neo4j_custom | See /logs/neo4j.log for current status.
neo4j_custom | Unable to connect to localhost:7687, ensure the database is running and that there is a working network connection to it.
It seems I am so close but for the 'unable to connect' issue!
Hi @andy.mcshane , gotcha ... modified the dockerfile
FROM neo4j
ENV NEO4J_HOME="/var/lib/neo4j" \
NEO4J_PASSWD=neo4j_seed
COPY Dependencies_Tree_Full_UnSorted.json ${NEO4J_HOME}/import/
COPY data_loader.cypher ${NEO4J_HOME}/import/
# set initial-password to start loading the data
# sleep for 10 secs for neo4j to start without any overlapping
CMD bin/neo4j-admin set-initial-password ${NEO4J_PASSWD} && \
bin/neo4j start && sleep 30 && \
if [ -f "${NEO4J_HOME}/import/data_loader.cypher" ]; then \
cat ${NEO4J_HOME}/import/data_loader.cypher | NEO4J_USERNAME=neo4j NEO4J_PASSWORD=${NEO4J_PASSWD} bin/cypher-shell --fail-fast && rm ${NEO4J_HOME}/import/*; \
fi && tail -f /logs/neo4j.log
added tail -f /logs/neo4j.log at the end to keep it running
Log
Creating network "everyday_default" with the default driver
Building neo4j
Step 1/5 : FROM neo4j
latest: Pulling from library/neo4j
a076a628af6f: Pull complete
943d8acaac04: Pull complete
b9998d19c116: Pull complete
eba5b958e041: Pull complete
b8d0884b547f: Pull complete
4b3572cb5079: Pull complete
e743ea4f2800: Pull complete
020ba241c011: Pull complete
Digest: sha256:c7f24de1dc1d2020ab24a884b8a39538937c1b14bc0ca1da3ddb2573b6fc412f
Status: Downloaded newer image for neo4j:latest
---> 9edee9e153ab
Step 2/5 : ENV NEO4J_HOME="/var/lib/neo4j" NEO4J_PASSWD=neo4j_seed
---> Running in 0ac728b0e379
Removing intermediate container 0ac728b0e379
---> 8403c4c5b83a
Step 3/5 : COPY Dependencies_Tree_Full_UnSorted.json ${NEO4J_HOME}/import/
---> 173896eaffd9
Step 4/5 : COPY data_loader.cypher ${NEO4J_HOME}/import/
---> 254b3704b87f
Step 5/5 : CMD bin/neo4j-admin set-initial-password ${NEO4J_PASSWD} && bin/neo4j start && sleep 30 && if [ -f "${NEO4J_HOME}/import/data_loader.cypher" ]; then
cat ${NEO4J_HOME}/import/data_loader.cypher | NEO4J_USERNAME=neo4j NEO4J_PASSWORD=${NEO4J_PASSWD} bin/cypher-shell --fail-fast && rm ${NEO4J_HOME}/import/*; fi && tail -f
/logs/neo4j.log
---> Running in f2a1ecb14a24
Removing intermediate container f2a1ecb14a24
---> 6ec65332f690
Successfully built 6ec65332f690
Successfully tagged everyday_neo4j:latest
Creating neo4j_custom ... done
Attaching to neo4j_custom
neo4j_custom | grep: /var/lib/neo4j/conf/neo4j.conf: No such file or directory
neo4j_custom | Changed password for user 'neo4j'.
neo4j_custom | Directories in use:
neo4j_custom | home: /var/lib/neo4j
neo4j_custom | config: /var/lib/neo4j/conf
neo4j_custom | logs: /logs
neo4j_custom | plugins: /plugins
neo4j_custom | import: /import
neo4j_custom | data: /var/lib/neo4j/data
neo4j_custom | certificates: /var/lib/neo4j/certificates
neo4j_custom | run: /var/lib/neo4j/run
neo4j_custom | Starting Neo4j.
neo4j_custom | Started neo4j (pid 292). It is available at http://localhost:7474/
neo4j_custom | There may be a short delay until the server is ready.
neo4j_custom | See /logs/neo4j.log for current status.
neo4j_custom | n
neo4j_custom | (:person {name: "test"})
neo4j_custom | 2021-01-25 19:20:09.489+0000 INFO Started.
neo4j_custom | 2021-01-25 19:20:49.736+0000 WARN The client is unauthorized due to authentication failure.
neo4j_custom | 2021-01-25 19:26:43.119+0000 WARN Unrecognized setting. No declared setting with name: PASSWD
neo4j_custom | 2021-01-25 19:26:43.136+0000 INFO Starting...
neo4j_custom | 2021-01-25 19:26:46.126+0000 INFO ======== Neo4j 4.2.2 ========
neo4j_custom | 2021-01-25 19:26:48.614+0000 INFO Performing postInitialization step for component 'security-users' with version 2 and status CURRENT
neo4j_custom | 2021-01-25 19:26:48.615+0000 INFO Updating the initial password in component 'security-users'
neo4j_custom | 2021-01-25 19:26:53.358+0000 INFO Bolt enabled on 0.0.0.0:7687.
neo4j_custom | 2021-01-25 19:26:55.173+0000 INFO Remote interface available at http://localhost:7474/
neo4j_custom | 2021-01-25 19:26:55.175+0000 INFO Started.
my cypher query
merge (a:person{name:"test"});
verification
PS C:\scratch> docker exec -it neo4j_custom bash
root@cb3cf7d20ca6:/var/lib/neo4j# cypher-shell
username: neo4j
password: **********
Connected to Neo4j 4.2.0 at neo4j://localhost:7687 as user neo4j.
Type :help for a list of available commands or :exit to exit the shell.
Note that Cypher queries must end with a semicolon.
neo4j@neo4j> match (n) return (n);
+--------------------------+
| n |
+--------------------------+
| (:person {name: "test"}) |
+--------------------------+
1 row available after 5 ms, consumed after another 1 ms
neo4j@neo4j>
hank you so much for your help. What you suggest does indeed get past the connection issue if I run the dockerfile in islation. If I run the docker-compose file using build to run the dockerfile the connection still exists. I seem to be opening a can of worms here. I am going to try to continue just with the dockerfile and see how much further I can get, I need to figure out how to add the apoc plugin and then verify the COPY commands now though.
If you are adding apoc and gds in docker-compose, you can use
- NEO4JLABS_PLUGINS=["apoc","graph-data-science"]
If you are adding in Dockerfile - gds is a zip file, and neo4j image doesnt have unzip, so you have to trick something like this -
FROM alpine:latest
WORKDIR /tmp
ADD https://s3-eu-west-1.amazonaws.com/com.neo4j.graphalgorithms.dist/graph-data-science/neo4j-graph-data-science-1.4.1-standalone.zip /tmp
RUN unzip neo4j-graph-data-science-1.1.6-standalone.zip
FROM neo4j
ENV NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
NEO4J_HOME="/var/lib/neo4j"
ADD https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.1/apoc-4.2.0.1-all.jar "${NEO4J_HOME}"/plugins
COPY --from=0 /tmp/*.jar "${NEO4J_HOME}"/plugins
or if you have using a docker-compose volume mounts, then you can just copy and plugins to your soft link mount volume.
Network - even if you run docker-compose and a separate Dockerfile, you can specify a Network name in both of them, and they will connect with each other.
You are being extremely helpful and I very much appreciate it and I feel that I must be missing the vital lightbulb moment for it all to click together. As it stands I have 1 issue outstanding.
It does not look like my cypher query is actually being executed? I cannot see the expected data in my default neo4j database and I do not see any errors on screen? I can confirm that the apoc library has been copied to the 'plugins' directory and also the json & cypher file have been copied to the 'import' directory.
Is there anything wrong with my cypher? This is what my 'data_loader.cypher' file contains.
// load data from json file (without full path default location is /import)
call apoc.load.json("file:///Dependencies_Tree_Full_UnSorted.json") yield value
// merge loaded data into DB as nodes
merge (n:Node {nodeId: value.id}) on create set n.name = value.name
// using current data
with n, value
// unwind all of the node depends on nodes
unwind value.dependsOn as depOn
// merge loaded dependencies into DB as nodes
merge (e:Node {nodeId: depOn.id}) on create set e.name = depOn.name
// specify depends on relationship
merge (n)-[:Depends_On]-(e)
Oh boy, do I feel stupid, all that was missing was the semi0colon at the end of my cypher query and everything works as expected now, doh! Thank you very much for all your help and patience, it is very much appreciated.
