cancel
Showing results for 
Search instead for 
Did you mean: 

Remote Neo4j Backups from a Docker Container

mike_blum_neo4j
Graph Buddy

Neo4j Version: 3.4.3

I'm working on this question here:

for our own backup solution and as a part of that effort, I want to standup Neo4j inside a Docker container for our CI testing. While my backup script works well against a bare neo4j instance running as a plain java process, remote backups don't seem to work so well when the database is inside a container.

Here is my Docker command:

# start neo4j instance
sudo mkdir -p /tmp/neo4j/data
sudo mkdir -p /tmp/neo4j/logs
sudo chown mblum:staff -R /tmp/neo4j
docker run --name neo4j-ci --detach \
  --publish=7474:7474 --publish=7687:7687 --publish=6362:6362 \
  --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
  --env=NEO4J_dbms_memory_pagecache_size=4G \
  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 \
  --volume=/tmp/neo4j/data:/data \
  --volume=/tmp/neo4j/logs:/logs \
  neo4j:3.4-enterprise

When i test it with the neo4j-admin tool I get some buffer writing issues:

neo4j-admin backup --backup-dir=/tmp/backups --name=neo4j-ci
2018-11-06 18:27:10.288+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
org.neo4j.commandline.admin.CommandFailed: Failed to run a backup using the available strategies.
	at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:99)
	at org.neo4j.backup.impl.OnlineBackupCommand.execute(OnlineBackupCommand.java:74)
	at org.neo4j.commandline.admin.AdminTool.execute(AdminTool.java:127)
	at org.neo4j.commandline.admin.AdminTool.main(AdminTool.java:51)
	Suppressed: org.neo4j.causalclustering.catchup.storecopy.StoreIdDownloadFailedException: org.neo4j.causalclustering.catchup.CatchUpClientException: Completed exceptionally when executing operation org.neo4j.causalclustering.catchup.storecopy.GetStoreIdRequest@f1da57d on localhost:6362
		at org.neo4j.causalclustering.catchup.storecopy.StoreCopyClient.fetchStoreId(StoreCopyClient.java:203)
		at org.neo4j.backup.impl.BackupDelegator.fetchStoreId(BackupDelegator.java:87)
		at org.neo4j.backup.impl.CausalClusteringBackupStrategy.performFullBackup(CausalClusteringBackupStrategy.java:68)
		at org.neo4j.backup.impl.BackupStrategyWrapper.fullBackupWithTemporaryFolderResolutions(BackupStrategyWrapper.java:146)
		at org.neo4j.backup.impl.BackupStrategyWrapper.performBackupWithoutLifecycle(BackupStrategyWrapper.java:111)
		at org.neo4j.backup.impl.BackupStrategyWrapper.doBackup(BackupStrategyWrapper.java:72)
		at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:86)
		... 3 more
	Caused by: org.neo4j.causalclustering.catchup.CatchUpClientException: Completed exceptionally when executing operation org.neo4j.causalclustering.catchup.storecopy.GetStoreIdRequest@f1da57d on localhost:6362
		at org.neo4j.causalclustering.catchup.TimeoutLoop.exception(TimeoutLoop.java:87)
		at org.neo4j.causalclustering.catchup.TimeoutLoop.waitForCompletion(TimeoutLoop.java:57)
		at org.neo4j.causalclustering.catchup.CatchUpClient.makeBlockingRequest(CatchUpClient.java:94)
		at org.neo4j.causalclustering.catchup.storecopy.StoreCopyClient.fetchStoreId(StoreCopyClient.java:199)
		... 9 more
	Caused by: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
		at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
		at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
		at org.neo4j.causalclustering.catchup.TimeoutLoop.waitForCompletion(TimeoutLoop.java:48)
		... 11 more
	Caused by: java.nio.channels.ClosedChannelException
		at org.neo4j.causalclustering.catchup.TrackingResponseHandler.onClose(TrackingResponseHandler.java:146)
		at org.neo4j.causalclustering.catchup.CatchUpClient$CatchUpChannel.lambda$connect$1(CatchUpClient.java:168)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
		at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
		at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
		at java.lang.Thread.run(Thread.java:748)
		at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:110)
	Suppressed: org.neo4j.com.ComException: Channel has been closed, so no need to try to write to it anymore. Client closed it?
		at org.neo4j.com.ChunkingChannelBuffer.writeCurrentChunk(ChunkingChannelBuffer.java:640)
		at org.neo4j.com.ChunkingChannelBuffer.done(ChunkingChannelBuffer.java:698)
		at org.neo4j.com.Protocol.serializeRequest(Protocol.java:190)
		at org.neo4j.com.Client.sendRequest(Client.java:305)
		at org.neo4j.com.Client.sendRequest(Client.java:290)
		at org.neo4j.backup.impl.BackupClient.fullBackup(BackupClient.java:68)
		at org.neo4j.backup.impl.BackupProtocolService$FullBackupStoreCopyRequester.copyStore(BackupProtocolService.java:493)
		at org.neo4j.com.storecopy.StoreCopyClient.copyStore(StoreCopyClient.java:134)
		at org.neo4j.backup.impl.BackupProtocolService.fullBackup(BackupProtocolService.java:165)
		at org.neo4j.backup.impl.BackupProtocolService.doFullBackup(BackupProtocolService.java:138)
		at org.neo4j.backup.impl.HaBackupStrategy.performFullBackup(HaBackupStrategy.java:86)
		at org.neo4j.backup.impl.BackupStrategyWrapper.fullBackupWithTemporaryFolderResolutions(BackupStrategyWrapper.java:146)
		at org.neo4j.backup.impl.BackupStrategyWrapper.performBackupWithoutLifecycle(BackupStrategyWrapper.java:111)
		at org.neo4j.backup.impl.BackupStrategyWrapper.doBackup(BackupStrategyWrapper.java:72)
		at org.neo4j.backup.impl.BackupStrategyCoordinator.performBackup(BackupStrategyCoordinator.java:86)
		... 3 more
command failed: Failed to run a backup using the available strategies.

I've noted that on the enterprise Docker images 3.4.3 has been struck from the repository: Neo4j on Docker. I've had good luck using the 3.4.9 backup driver targeting a 3.4.3 instance but I assume this won't always be the case and will break at some point.

Are there any caveats to doing remote backups from a Docker container?

Thanks!
Mike

5 REPLIES 5

mike_blum_neo4j
Graph Buddy

I downloaded the latest 3.4.9 and got the same error:

neo4j version
neo4j 3.4.9

sudo mkdir -p /tmp/neo4j/data
sudo mkdir -p /tmp/neo4j/logs
sudo chown mblum:staff -R /tmp/neo4j
docker run --name neo4j-ci --detach \
  --publish=7474:7474 --publish=7687:7687 --publish=6362:6362 \
  --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
  --env=NEO4J_dbms_memory_pagecache_size=4G \
  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 \
  --volume=/tmp/neo4j/data:/data \
  --volume=/tmp/neo4j/logs:/logs \
  neo4j:3.4.9-enterprise

neo4j-admin backup --backup-dir=/tmp/backups --name=neo4j-ci
2018-11-06 18:59:02.928+0000 INFO [o.n.b.i.BackupOutputMonitor] Start receiving store files
command failed: Failed to run a backup using the available strategies.

Figured out my mistake:

  --env=NEO4J_dbms.backup.enabled=true \
  --env=NEO4J_dbms.backup.address=0.0.0.0:6362 

needs to be:

--env=NEO4J_dbms_backup_enabled=true \
--env=NEO4J_dbms_backup_address=0.0.0.0:6362

as specified in the docs here: https://neo4j.com/docs/operations-manual/current/installation/docker/#docker-environment-variables

Thank you for the info.
But do you run the neo4j-admin back outside the container ? Because I can't see you doing docker exec or though ?

I ran it outside the container so that the backup ends up in a durable location. I wonder if a docker volume for the backup would be performant?

mike_blum_neo4j
Graph Buddy

@bakk if you're on here.

Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.