I have a deployment of neo4j which runs together with Azure (Azure-Files) and it works great - find below the yaml used to create the service and deployment.
I next fill my graph with nodes Question, Author, Translation and the TimeTree from GraphAware is attached (this all happens via neomodel in Django). The data is successfully mounted and I see the data stored in Azure, great! In the neo4j browser (port 7474 or 7473) I can view all nodes and play around with them, no problem.
However, the moment I shut down the pod, for example using:
kubectl scale deployment mygraph-deployment replicas=0 # stop the pod
kubectl scale deployment mygraph-deployment replicas=1 # start the pod
Something strange happens. If I now go to the browser and click on Question I see a limit of 25 Question nodes, great! However, if I click on Translation I get an error regarding permissions, see below.
As a note:
- I have experimented with both setting the environmental variables for authorisation, e.g.
Authentication
- name: NEO4J_AUTH
value: neo4j/mypass
- I have also tried removing the
store_lockfile in/data/databases/graph.db, but that made no impact either. - When adding different nodes, the same behaviour is visible. Some can be viewed in the browser, whereas other cannot. For no clear reason.
- The files
auth,auth.iniandrolesare successfully created and mounted within the/data/dbmsfolder.
Any help is very welcome, thanks!
debug.log
2019-09-10 15:59:31.243+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [UnknownError]: java.io.IOException: Permission denied, reference 0f51f4b8-097a-491a-9648-8ee6eec75c86.
2019-09-10 15:59:31.243+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [UnknownError]: java.io.IOException: Permission denied, reference 0f51f4b8-097a-491a-9648-8ee6eec75c86. java.io.IOException: Permission denied
org.neo4j.kernel.impl.store.UnderlyingStorageException: java.io.IOException: Permission denied
at org.neo4j.kernel.impl.store.StoreRecordCursor.next(StoreRecordCursor.java:87)
at org.neo4j.kernel.impl.store.StoreRecordCursor.next(StoreRecordCursor.java:50)
at org.neo4j.kernel.impl.api.store.StorePropertyPayloadCursor.readFromStore(StorePropertyPayloadCursor.java:291)
at org.neo4j.kernel.impl.api.store.StorePropertyPayloadCursor.stringValue(StorePropertyPayloadCursor.java:210)
at org.neo4j.kernel.impl.api.store.StorePropertyPayloadCursor.value(StorePropertyPayloadCursor.java:253)
at org.neo4j.kernel.impl.api.store.StorePropertyCursor.value(StorePropertyCursor.java:106)
at org.neo4j.kernel.impl.core.NodeProxy.getAllProperties(NodeProxy.java:423)
at org.neo4j.kernel.impl.util.NodeProxyWrappingNodeValue.properties(NodeProxyWrappingNodeValue.java:111)
at org.neo4j.kernel.impl.util.NodeProxyWrappingNodeValue.writeTo(NodeProxyWrappingNodeValue.java:59)
at org.neo4j.bolt.v1.messaging.Neo4jPack$Packer.pack(Neo4jPack.java:86)
at org.neo4j.bolt.v1.messaging.BoltResponseMessageWriter.onRecord(BoltResponseMessageWriter.java:75)
at org.neo4j.bolt.v1.messaging.BoltMessageRouter$ResultHandler$1.visit(BoltMessageRouter.java:153)
at org.neo4j.bolt.v1.runtime.CypherAdapterStream.lambda$accept$0(CypherAdapterStream.java:78)
at org.neo4j.cypher.internal.compiler.v3_3.generated.GeneratedExecutionPlan626997086318577Execution.accept(Unknown Source)
at org.neo4j.cypher.internal.compatibility.v3_3.runtime.compiled.CompiledExecutionResult.accept(CompiledExecutionResult.scala:53)
at org.neo4j.cypher.internal.compatibility.ClosingExecutionResult$$anonfun$accept$2.apply$mcV$sp(ClosingExecutionResult.scala:175)
at org.neo4j.cypher.internal.compatibility.ClosingExecutionResult$$anonfun$accept$2.apply(ClosingExecutionResult.scala:174)
at org.neo4j.cypher.internal.compatibility.ClosingExecutionResult$$anonfun$accept$2.apply(ClosingExecutionResult.scala:174)
at org.neo4j.cypher.internal.compatibility.v3_3.exceptionHandler$runSafely$.apply(exceptionHandler.scala:90)
at org.neo4j.cypher.internal.compatibility.ClosingExecutionResult.accept(ClosingExecutionResult.scala:174)
at org.neo4j.bolt.v1.runtime.CypherAdapterStream.accept(CypherAdapterStream.java:76)
at org.neo4j.bolt.v1.messaging.BoltMessageRouter$ResultHandler.onRecords(BoltMessageRouter.java:146)
at org.neo4j.bolt.v1.runtime.BoltStateMachine$State$3.lambda$pullAll$0(BoltStateMachine.java:462)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$1.streamResult(TransactionStateMachine.java:271)
at org.neo4j.bolt.v1.runtime.TransactionStateMachine.streamResult(TransactionStateMachine.java:105)
at org.neo4j.bolt.v1.runtime.BoltStateMachine$State$3.pullAll(BoltStateMachine.java:461)
at org.neo4j.bolt.v1.runtime.BoltStateMachine.pullAll(BoltStateMachine.java:236)
at org.neo4j.bolt.v1.messaging.BoltMessageRouter.lambda$onPullAll$6(BoltMessageRouter.java:114)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.execute(RunnableBoltWorker.java:152)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.executeBatch(RunnableBoltWorker.java:145)
at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
Caused by: java.io.IOException: Permission denied
at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:735)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:721)
at org.neo4j.io.fs.StoreFileChannel.read(StoreFileChannel.java:105)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.swapIn(SingleFilePageSwapper.java:237)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.read(SingleFilePageSwapper.java:292)
at org.neo4j.io.pagecache.impl.muninn.PageList.fault(PageList.java:446)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pageFault(MuninnPageCursor.java:372)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.uncommonPin(MuninnPageCursor.java:321)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.pin(MuninnPageCursor.java:295)
at org.neo4j.io.pagecache.impl.muninn.MuninnReadPageCursor.next(MuninnReadPageCursor.java:61)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCursor.next(MuninnPageCursor.java:127)
at org.neo4j.kernel.impl.store.CommonAbstractStore.readIntoRecord(CommonAbstractStore.java:1069)
at org.neo4j.kernel.impl.store.StoreRecordCursor.next(StoreRecordCursor.java:82)
... 36 more
security.log
2019-09-10 15:59:21.913+0000 ERROR []: failed to log in: invalid principal or credentials
2019-09-10 15:59:22.337+0000 INFO [neo4j]: logged in
2019-09-10 15:59:23.729+0000 INFO [neo4j]: logged in
2019-09-10 15:59:23.846+0000 INFO [neo4j]: logged in
2019-09-10 15:59:23.892+0000 INFO [neo4j]: logged in
2019-09-10 15:59:23.997+0000 INFO [neo4j]: logged in
2019-09-10 15:59:25.510+0000 INFO [neo4j]: logged in
2019-09-10 15:59:27.669+0000 INFO [neo4j]: logged in
2019-09-10 15:59:28.685+0000 INFO [neo4j]: logged in
2019-09-10 15:59:28.723+0000 INFO [neo4j]: logged in
2019-09-10 15:59:29.334+0000 INFO [neo4j]: logged in
2019-09-10 15:59:31.026+0000 INFO [neo4j]: logged in
2019-09-10 15:59:43.480+0000 INFO [neo4j]: logged in
2019-09-10 15:59:43.527+0000 INFO [neo4j]: logged in
2019-09-10 15:59:44.041+0000 INFO [neo4j]: logged in
2019-09-10 16:00:03.784+0000 INFO [neo4j]: logged in
2019-09-10 16:00:03.822+0000 INFO [neo4j]: logged in
2019-09-10 16:00:04.410+0000 INFO [neo4j]: logged in
Graph yaml
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: mygraph
ports:
- name: bolt
protocol: TCP
port: 7687
targetPort: 7687
nodePort: 30076
- name: http
protocol: TCP
port: 7474
targetPort: 7474
nodePort: 30074
- name: https
protocol: TCP
port: 7473
targetPort: 7473
nodePort: 30073
type: LoadBalancer
loadBalancerIP: 40.118.172.46
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mygraph-deployment
labels:
app: mygraph
spec:
replicas: 1
selector:
matchLabels:
app: mygraph
template:
metadata:
labels:
app: mygraph
spec:
containers:
- name: mygraph
image: mycontainer.azurecr.io/mygraph:dev
securityContext:
privileged: true
ports:
- containerPort: 7687
- containerPort: 7474
- containerPort: 7473
env:
- name: NEO4J_dbms_jvm_additional
value: "-Xms512m -Xmx6g"
# When adding the below lines the deployment failes. Without it, it works great!
volumeMounts:
- name: neo4jdata
mountPath: /var/lib/neo4j/data/
volumes:
- name: neo4jdata
persistentVolumeClaim:
claimName: azurefile
