Permission denied when restarting core servers in a casual cluster

Integrations & Ecosystem Docker
1

Hello,

I am learning on the casual cluster configuration.
My current configuration is as follows -the neo4j instance of my vm has been stopped:

uname -r
4.14.35-1844.4.5.el7uek.x86_64

docker version
Client:
Version: 18.09.1-ol
API version: 1.39
Go version: go1.10.8
Git commit: c3ab8a8
Built: Mon Feb 11 23:20:32 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.1-ol
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: c3ab8a8
Built: Mon Feb 11 23:09:34 2019
OS/Arch: linux/amd64
Experimental: false
Default Registry: docker.io

neo4j version
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
neo4j 3.5.5

I have created 3 core servers with a shell script as follows:

docker run --name=core1
--volume=pwd/core1/conf:/conf --volume=pwd/core1/data:/data --volume=pwd/core1/logs:/logs
--publish=11474:7474 --publish=11687:7687
--env=NEO4J_dbms_connector_bolt_advertised__address=localhost:11687
--network=training-cluster
--env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
--env=NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
--env=NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
--env=NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000,core4:5000,core5:5000
--env=NEO4J_dbms_mode=CORE
--detach
$1
The last line being: docker stop core1 core2 core3

When I tried to restart the core servers, one by one, I met the following problem:

docker start -a core1
find: /conf/original-neo4j.conf: Permission denied
find: /conf/neo4j.conf: Permission denied

If my user is neo4j and has been added to the docker group -even to root, what am I missing?

id neo4j
uid=990(neo4j) gid=988(neo4j) groups=988(neo4j),0(root),986(docker)

The docker log shows this:

May 01 18:36:54 ol7-neo4j-db.localdomain dockerd[11291]: time="2019-05-01T18:36:54.570991376+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

Thanks in advance for your help.
Blai

2

This bit right here creates a mapping from the HOST OS drive to the GUEST OS drive. On the HOST OS, you should check that the user executing docker is permitted to read those files, since if not, not even root on the GUEST OS will be able to see the files.

It's like docker inception.

3

On the machine that runs docker (not inside of the docker container) you can follow up post the owner/group/permissions for your /core1/conf files, and also show what user is executing the docker command with whoami

4

Hello David,

First of all, thanks for replying.

My main line of investigation has been this bit you mention

--volume= pwd /core1/conf:/conf

I run all commands as neo4j which has it home directory set to

/var/lib/neo4j

In order to avoid access issues I have created a working directory just for this exercise and given full acces to neo4j. Still, I get:

-bash-4.2$ docker start -a core1
find: /conf: Permission denied

This is the working directory:

drwxrwxrwx. 3 neo4j neo4j 48 May 3 18:29 work

-bash-4.2$ ls -Rlart work
work:
total 2664
-rw-rw-r--. 1 neo4j neo4j 2719271 Apr 27 18:49 neo4j-docker.zip
dr-xr-xr-x. 18 root root 4096 May 3 18:28 ..
drwxrwxrwx. 3 neo4j neo4j 48 May 3 18:29 .
drwxrwxrwx. 12 neo4j neo4j 4096 May 3 19:14 neo4j-docker

work/neo4j-docker:
total 28
drwxrwxrwx. 5 neo4j neo4j 39 Jan 17 21:17 core1
drwxrwxrwx. 5 neo4j neo4j 39 Jan 17 21:18 core2
drwxrwxrwx. 5 neo4j neo4j 39 Jan 17 21:19 core3

work/neo4j-docker/core1:
total 4
drwxrwxrwx. 2 neo4j neo4j 6 Jan 17 21:17 logs
drwxrwxrwx. 5 neo4j neo4j 39 Jan 17 21:17 .
drwxrwxrwx. 2 neo4j neo4j 27 Jan 17 21:17 data
drwxrwxrwx. 2 neo4j neo4j 49 Jan 17 21:27 conf
drwxrwxrwx. 12 neo4j neo4j 4096 May 3 19:14 ..

work/neo4j-docker/core1/conf:
total 80
drwxrwxrwx. 5 neo4j neo4j 39 Jan 17 21:17 ..
-rwxrwxrwx. 1 neo4j neo4j 39368 Jan 17 21:26 neo4j.conf
-rwxrwxrwx. 1 neo4j neo4j 39368 Jan 17 21:26 original-neo4j.conf
drwxrwxrwx. 2 neo4j neo4j 49 Jan 17 21:27 .

For that, I am still in the dark... I don't see what it is that I am missing.

If you have any further advise...

Thanks.

5

I am not 100% sure, but I think you may be running into this problem. Please have a look here:

6

Thanks again, David, for your reply.

I am clueless on what is the issue. I have tried the solution explained in the shared link with the same result.

I am even more clueless as I have recovered an "older" virtual machine and it went through just fine:

-bash-4.2$ docker version
Client:
Version: 17.06.2-ol
API version: 1.30
Go version: go1.8.3
Git commit: d02b7ab
Built: Fri Oct 6 00:02:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.06.2-ol
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: d02b7ab
Built: Fri Oct 6 00:03:48 2017
OS/Arch: linux/amd64
Experimental: false
-bash-4.2$ uname -r
4.1.12-124.15.2.el7uek.x86_64
-bash-4.2$ neo4j version
neo4j 3.5.3

-bash-4.2$ id neo4j
uid=988(neo4j) gid=981(neo4j) groups=981(neo4j),982(docker)

Not sure if it has anything to do with the versions.

Regards.