Neo4j HA Pods in Statefulset can't connect to the other cluster members

tech-mint · March 13, 2019, 2:10pm

Hi,
My environment is Openshift v3.9 (~Kubernetes 1.9)
I deployed based on the templates in this helm package

I only skipped readreplicas-deployment.yaml and poddisruptionbudget.yaml so I have Statefulset of Core Neo4j pods and headless service + secret.

The issue is after I scale up cluster to 3 replicas every pod got stuck in the state blow:

++ hostname -f
+ export NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local
+ NEO4J_dbms_connectors_default__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local
++ hostname -f
+ export NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000
+ NEO4J_causal__clustering_discovery__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000
++ hostname -f
+ export NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000
+ NEO4J_causal__clustering_transaction__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000
++ hostname -f
+ export NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000
+ NEO4J_causal__clustering_raft__advertised__address=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000
+ exec /docker-entrypoint.sh neo4j
Active database: graph.db
Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /var/lib/neo4j/logs
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/lib/neo4j/run
Starting Neo4j.
2019-03-13 14:03:47.313+0000 INFO  ======== Neo4j 3.4.10 ========
2019-03-13 14:03:47.357+0000 INFO  Starting...
2019-03-13 14:03:49.413+0000 INFO  Initiating metrics...
2019-03-13 14:03:49.580+0000 INFO  Resolved initial host 'neo4j.rag-uat.svc.cluster.local:5000' to [10.130.1.54:5000, 10.128.0.45:5000]
2019-03-13 14:03:49.611+0000 INFO  My connection info: [
	Discovery:   listen=0.0.0.0:5000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:5000,
	Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, 
	Raft:        listen=0.0.0.0:7000, advertised=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, 
	Client Connector Addresses: bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473 
]
2019-03-13 14:03:49.611+0000 INFO  Discovering cluster with initial members: [neo4j.rag-uat.svc.cluster.local:5000]
2019-03-13 14:03:49.611+0000 INFO  Attempting to connect to the other cluster members before continuing...

As I googled around the history of the issue have an unresolved story
in here , here ,and here

As it is obvious from the logs I am using Bolt protocol instead of the original.

I did a DNS experiment from inside of one pod, here is the result:

/var/lib/neo4j # nslookup neo4j.rag-uat.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve

Name:      neo4j.rag-uat.svc.cluster.local
Address 1: 10.128.0.45 neo4j-core-1.neo4j.rag-uat.svc.cluster.local
Address 2: 10.129.0.104 neo4j-core-2.neo4j.rag-uat.svc.cluster.local
Address 3: 10.130.1.54 neo4j-core-0.neo4j.rag-uat.svc.cluster.local

david_allen · March 13, 2019, 2:27pm

When you start new nodes, they need to hook into the cluster by locating their buddies. This is controlled by the setting "initial discovery members" which in your yaml should look something like:

          - name: NEO4J_causal__clustering_initial__discovery__members
            value: "{{ template "neo4j.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local:5000"

First confirm that. Second thing to know is that the discovery member set is specifying only a single member by a single DNS address. As you've found, that DNS address looks up to 3 different A records (for the individual nodes). A key piece of magic is this:

          - name: NEO4J_causal__clustering_discovery__type
            value: DNS

Can you verify this too? This tells neo4j that the DNS address isn't literally a single node, but a multi-lookup to find all of the other cluster members.

If you're attempting to connect to other cluster members and getting stuck, this seems to be the problem. The nslookup verified that you have good DNS setup (via the service in the helm charts)

The missing bit is likely to be the discovery_type = DNS above. Please check.

Relevant docs here to consult, in case I've missed what the problem is -- something else here should help:

tech-mint · March 13, 2019, 2:46pm

I don't know if it's a good news or bad one

I confirm that both specifications are set as you expressed in previous reply.
here is my yaml

            - name: NEO4J_causal__clustering_discovery__type
              value: DNS
            - name: NEO4J_causal__clustering_initial__discovery__members
              value: 'neo4j.rag-uat.svc.cluster.local:5000'

NEO4J_causal__clustering_initial__discovery__members is pointing to the address of kubernetes Service which supports the Pods.

What is next?

tech-mint · March 13, 2019, 3:18pm

here is the logs from inside one of the Pods: part1

2019-03-13 15:06:11.293+0000 INFO [o.n.k.AvailabilityGuard] Requirement makes database unavailable: Database is stopped
2019-03-13 15:06:11.870+0000 INFO [o.n.k.i.f.CommunityEditionModule] No locking implementation specified, defaulting to 'forseti'
2019-03-13 15:06:11.916+0000 INFO [o.n.c.c.TransactionBackupServiceProvider] Binding backup service on address 127.0.0.1:6362
2019-03-13 15:06:11.920+0000 INFO [o.n.c.u.s.TypicallyConnectToRandomReadReplicaStrategy] Using upstream selection strategy typically-connect-to-random-read-replica
2019-03-13 15:06:11.944+0000 INFO [o.n.k.i.f.StatementLocksFactorySelector] No services implementing StatementLocksFactory found. Using SimpleStatementLocksFactory
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics START ---
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] Neo4j Kernel properties:
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.http.enabled=true
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_listen_address=0.0.0.0:5000
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.mode=CORE
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.security.auth_enabled=false
2019-03-13 15:06:12.169+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.http.listen_address=0.0.0.0:7474
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.directories.neo4j_home=/var/lib/neo4j
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.raft_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.directories.import=import
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.heap.initial_size=512M
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.raft_listen_address=0.0.0.0:7000
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connectors.default_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ACCEPT.LICENSE.AGREEMENT=yes
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ha.pull_interval=10
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_type=DNS
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] ha.host.coordination=neo4j-core-1:5001
2019-03-13 15:06:12.170+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.https.enabled=true
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.jvm.additional=-Dunsupported.dbms.udc.source=tarball
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.tx_log.rotation.retention_policy=100M size
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.heap.max_size=512M
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connectors.default_listen_address=0.0.0.0
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] EDITION=enterprise
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.windows_service_name=neo4j
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.https.listen_address=0.0.0.0:7473
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.listen_address=0.0.0.0:7687
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.transaction_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000
2019-03-13 15:06:12.171+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.discovery_advertised_address=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:5000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.memory.pagecache.size=512M
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] ha.host.data=neo4j-core-1:6001
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.initial_discovery_members=neo4j.rag-uat.svc.cluster.local:5000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.edition=enterprise
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] dbms.connector.bolt.enabled=true
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] causal_clustering.transaction_listen_address=0.0.0.0:6000
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] wrapper.java.additional=-Dneo4j.ext.udc.source=docker
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] Diagnostics providers:
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.configuration.Config
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] org.neo4j.kernel.info.DiagnosticsManager
2019-03-13 15:06:12.172+0000 INFO [o.n.k.i.DiagnosticsManager] SYSTEM_MEMORY
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] JAVA_MEMORY
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] OPERATING_SYSTEM
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] JAVA_VIRTUAL_MACHINE
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] CLASSPATH
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] LIBRARY_PATH
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] SYSTEM_PROPERTIES
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] TIMEZONE_DATABASE
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] LINUX_SCHEDULERS
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] NETWORK
2019-03-13 15:06:12.173+0000 INFO [o.n.k.i.DiagnosticsManager] System memory information:
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Total Physical memory: 7.64 GB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Free Physical memory: 335.31 MB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Committed virtual memory: 1.97 GB
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Total swap space: 0.00 B
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] Free swap space: 0.00 B
2019-03-13 15:06:12.194+0000 INFO [o.n.k.i.DiagnosticsManager] JVM memory information:
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Free  memory: 437.00 MB
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Total memory: 512.00 MB
2019-03-13 15:06:12.195+0000 INFO [o.n.k.i.DiagnosticsManager] Max   memory: 512.00 MB
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Young Generation: [G1 Eden Space, G1 Survivor Space]
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Garbage Collector: G1 Old Generation: [G1 Eden Space, G1 Survivor Space, G1 Old Gen]
2019-03-13 15:06:12.199+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Code Cache (Non-heap memory): committed=6.06 MB, used=6.02 MB, max=240.00 MB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Metaspace (Non-heap memory): committed=20.25 MB, used=19.18 MB, max=-1.00 B, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: Compressed Class Space (Non-heap memory): committed=2.75 MB, used=2.48 MB, max=1.00 GB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Eden Space (Heap memory): committed=322.00 MB, used=66.00 MB, max=-1.00 B, threshold=?
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Survivor Space (Heap memory): committed=1.00 MB, used=1.00 MB, max=-1.00 B, threshold=?
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Memory Pool: G1 Old Gen (Heap memory): committed=189.00 MB, used=8.00 MB, max=512.00 MB, threshold=0.00 B
2019-03-13 15:06:12.200+0000 INFO [o.n.k.i.DiagnosticsManager] Operating system information:
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Operating System: Linux; version: 3.10.0-514.el7.x86_64; arch: amd64; cpus: 1
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Max number of file descriptors: 1048576
2019-03-13 15:06:12.201+0000 INFO [o.n.k.i.DiagnosticsManager] Number of open file descriptors: 161
2019-03-13 15:06:12.202+0000 INFO [o.n.k.i.DiagnosticsManager] Process id: 1@neo4j-core-1
2019-03-13 15:06:12.202+0000 INFO [o.n.k.i.DiagnosticsManager] Byte order: LITTLE_ENDIAN
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] Local timezone: GMT
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] JVM information:
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Name: OpenJDK 64-Bit Server VM
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Vendor: Oracle Corporation
2019-03-13 15:06:12.207+0000 INFO [o.n.k.i.DiagnosticsManager] VM Version: 25.181-b13
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] JIT compiler: HotSpot 64-Bit Tiered Compilers
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] VM Arguments: [-Xms512M, -Xmx512M, -XX:+UseG1GC, -XX:-OmitStackTraceInFastThrow, -XX:+AlwaysPreTouch, -XX:+UnlockExperimentalVMOptions, -XX:+TrustFinalNonStaticFields, -XX:+DisableExplicitGC, -Djdk.tls.ephemeralDHKeySize=2048, -Djdk.tls.rejectClientInitiatedRenegotiation=true, -Dunsupported.dbms.udc.source=tarball, -Dfile.encoding=UTF-8]
2019-03-13 15:06:12.208+0000 INFO [o.n.k.i.DiagnosticsManager] Java classpath:
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-com-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-core-asl-1.9.13.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/jaccess.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-core-1.4.0.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/asm-analysis-6.2.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-planner-3.4-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/server-api-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/concurrentlinkedhashmap-lru-1.4.2.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-command-line-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jPowerShell-1.9.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-lucene-upgrade-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/netty-3.9.9.Final.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-udc-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-bolt-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-server-1.19.3.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-shell-3.4.10.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jce.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-compress-1.16.1.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient-0.1.0.jar
2019-03-13 15:06:12.210+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-csv-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-planner-spi-3.4-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-commercial-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-io-2.6.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-util-9.4.11.v20180605.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-core-1.19.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-jmx-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-runtime-util-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/asm-6.2.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-configuration-1.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-lang3-3.7.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-slotted-runtime-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-config-core-1.4.0.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunec.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jersey-servlet-1.19.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-util-3.4-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-security-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-text-1.3.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-values-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-management-3.4.10.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/scala-library-2.11.12.jar
2019-03-13 15:06:12.211+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/classes
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-event-1.4.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-security-9.4.11.v20180605.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-interpreted-runtime-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-hash-1.4.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-servlet-9.4.11.v20180605.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-morsel-runtime-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-ssl-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/javax.servlet-api-3.1.0.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/rt.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/localedata.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-security-enterprise-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jProcesses-1.6.4.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-2.3-2.3.12.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graph-algo-3.4.10.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/scala-reflect-2.11.12.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/plugins/
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/asm-util-6.2.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunjce_provider.jar
2019-03-13 15:06:12.212+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/slf4j-nop-1.7.25.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-lang-2.6.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-core-5.5.5.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-enterprise-kernel-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graphdb-api-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-codecs-5.5.5.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-codegen-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-metrics-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/rhino-1.7.9.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-3.3-3.3.8.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-core-1.4.0.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-ir-3.4-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/nashorn.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-compiler-3.1-3.1.9.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-beanutils-1.9.3.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-unsafe-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-api-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/dnsns.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jfr.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jline-2.12.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-logging-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-enterprise-3.4.10.jar
2019-03-13 15:06:12.213+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-logical-plans-3.4-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/conf/
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-logical-plans-3.3-3.3.8.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/sunpkcs11.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/asm-tree-6.2.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-expression-3.4-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-lucene-index-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/scallop_2.11-2.1.1.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/hazelcast-all-3.7.8.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-webapp-9.4.11.v20180605.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-consistency-check-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cluster-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/openCypher-frontend-1-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/opencsv-2.3.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/parboiled-scala_2.11-1.1.7.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_common-0.1.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-kernel-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-index-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-procedure-api-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-cache-1.4.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/kiama_2.11-2.1.0.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-backup-3.4.10.jar
2019-03-13 15:06:12.214+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-server-9.4.11.v20180605.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-enterprise-cypher-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-primitive-collections-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-collections-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/resources.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/lz4-1.3.0.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/metrics-core-4.0.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/charsets.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/slf4j-api-1.7.25.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-http-9.4.11.v20180605.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/metrics-graphite-4.0.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-mapper-asl-1.9.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-causal-clustering-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/caffeine-2.6.2.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-queryparser-5.5.5.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/bcprov-jdk15on-1.60.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-browser-3.2.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jackson-jaxrs-1.9.13.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/zipfs.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-io-3.4.10.jar

tech-mint · March 13, 2019, 3:19pm

logs part two:

2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-causal-clustering-commercial-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-configuration-3.4.10.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-3.1-3.1.9.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/WMI4Java-1.6.1.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-crypto-cipher-1.4.0.jar
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/plugins/*
2019-03-13 15:06:12.215+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-3.3-3.3.8.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_dropwizard-0.1.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [loader.1] file:/usr/lib/jvm/java-1.8-openjdk/jre/lib/ext/cldrdata.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jsr311-api-1.1.2.r612.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/simpleclient_httpserver-0.1.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-xml-9.4.11.v20180605.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-frontend-2.3-2.3.12.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-config-ogdl-1.4.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-server-commercial-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/lucene-analyzers-common-5.5.5.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-resource-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-auth-plugin-api-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-ha-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/jsse.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [bootstrap] /usr/lib/jvm/java-1.8-openjdk/jre/lib/sunrsasign.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/bcpkix-jdk15on-1.60.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-dbms-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-ir-3.3-3.3.8.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-logging-1.1.1.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/shiro-lang-1.4.0.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-query-logging-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/jetty-io-9.4.11.v20180605.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/commons-collections-3.2.2.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-graph-matching-3.1.9.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/parboiled-core-1.1.7.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-common-3.4.10.jar
2019-03-13 15:06:12.216+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/netty-all-4.1.24.Final.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-cypher-physical-planning-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-import-tool-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager]  [classpath + loader.0] file:/var/lib/neo4j/lib/neo4j-spatial-index-3.4.10.jar
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] Library path:
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib/jvm/java-1.8-openjdk/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/java/packages/lib/amd64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /lib64
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /lib
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] /usr/lib
2019-03-13 15:06:12.217+0000 INFO [o.n.k.i.DiagnosticsManager] System.properties:
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.shutdownhook.enabled = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.boot.library.path = /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] jdk.tls.rejectClientInitiatedRenegotiation = true
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] path.separator = :
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] file.encoding.pkg = sun.io
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.country = US
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.java.launcher = SUN_STANDARD
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.os.patch.level = unknown
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.dir = /var/lib/neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.jnu.encoding = UTF-8
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.phone.home.enabled = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] unsupported.dbms.udc.source = tarball
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.management.compiler = HotSpot 64-Bit Tiered Compilers
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.logging.type = none
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.home = /var/lib/neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.timezone = GMT
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] file.encoding = UTF-8
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] hazelcast.socket.bind.any = false
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] user.name = neo4j
2019-03-13 15:06:12.218+0000 INFO [o.n.k.i.DiagnosticsManager] sun.java.command = com.neo4j.server.enterprise.CommercialEntryPoint --home-dir=/var/lib/neo4j --config-dir=/var/lib/neo4j/conf
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.arch.data.model = 64
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] user.language = en
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] awt.toolkit = sun.awt.X11.XToolkit
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] jdk.tls.ephemeralDHKeySize = 2048
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] file.separator = /
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.io.unicode.encoding = UnicodeLittle
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.cpu.endian = little
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] sun.cpu.isalist = 
2019-03-13 15:06:12.219+0000 INFO [o.n.k.i.DiagnosticsManager] (IANA) TimeZone Database Version:
2019-03-13 15:06:12.250+0000 INFO [o.n.k.i.DiagnosticsManager]   TimeZone version: 2018e (available for 599 zone identifiers)
2019-03-13 15:06:12.250+0000 INFO [o.n.k.i.DiagnosticsManager] Linux scheduler information:
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] noop deadline [cfq] 
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] noop deadline [cfq] 
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.256+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.257+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.258+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.259+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.260+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.261+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] none
2019-03-13 15:06:12.262+0000 INFO [o.n.k.i.DiagnosticsManager] Network information:
2019-03-13 15:06:12.263+0000 INFO [o.n.k.i.DiagnosticsManager] Interface eth0:
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager]     address: fe80:0:0:0:f481:e2ff:fe42:5a4d%eth0
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager]     address: 10.128.0.45
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] Interface lo:
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager]     address: 0:0:0:0:0:0:0:1%lo
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager]     address: 127.0.0.1
2019-03-13 15:06:12.264+0000 INFO [o.n.k.i.DiagnosticsManager] --- INITIALIZED diagnostics END ---
2019-03-13 15:06:12.271+0000 INFO [o.n.c.c.s.s.DurableStateStorage] term state restored, up to ordinal 19701
2019-03-13 15:06:12.275+0000 INFO [o.n.c.c.s.s.DurableStateStorage] vote state restored, up to ordinal 19701
2019-03-13 15:06:12.279+0000 INFO [o.n.c.c.s.s.DurableStateStorage] membership state restored, up to ordinal 0
2019-03-13 15:06:12.283+0000 INFO [o.n.c.c.s.s.DurableStateStorage] session-tracker state restored, up to ordinal -1
2019-03-13 15:06:12.288+0000 INFO [o.n.c.c.s.s.DurableStateStorage] lock-token state restored, up to ordinal -1
2019-03-13 15:06:12.292+0000 INFO [o.n.c.c.s.s.DurableStateStorage] id-allocation state restored, up to ordinal 0
2019-03-13 15:06:12.295+0000 INFO [o.n.c.c.s.s.DurableStateStorage] last-flushed state restored, up to ordinal 0
2019-03-13 15:06:12.845+0000 INFO [o.n.b.BoltKernelExtension] Bolt Server extension loaded.
2019-03-13 15:06:12.906+0000 INFO [o.n.m.MetricsExtension] Initiating metrics...
2019-03-13 15:06:13.012+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Cluster discovery service starting
2019-03-13 15:06:13.039+0000 INFO [o.n.c.d.DnsHostnameResolver] Resolved initial host 'neo4j.rag-uat.svc.cluster.local:5000' to [10.130.1.54:5000, 10.129.0.104:5000, 10.128.0.45:5000]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] My connection info: [
	Discovery:   listen=0.0.0.0:5000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:5000,
	Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000, 
	Raft:        listen=0.0.0.0:7000, advertised=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000, 
	Client Connector Addresses: bolt://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7473
]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Discovering cluster with initial members: [neo4j.rag-uat.svc.cluster.local:5000]
2019-03-13 15:06:13.066+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Attempting to connect to the other cluster members before continuing...
2019-03-13 15:06:13.082+0000 INFO [o.n.c.c.c.l.s.RecoveryProtocol] Skipping from index -1 to 1.
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.l.s.SegmentedRaftLog] log started with recovered state State{prevIndex=0, prevTerm=-1, appendIndex=0}
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state before recovery: RaftMembershipState{committed=MembershipEntry{logIndex=0, members=[MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]}, appended=null, ordinal=0}
2019-03-13 15:06:13.089+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Recovering from: 0 to: 0
2019-03-13 15:06:13.091+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Membership state after recovery: RaftMembershipState{committed=MembershipEntry{logIndex=0, members=[MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]}, appended=null, ordinal=0}
2019-03-13 15:06:13.091+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: []
2019-03-13 15:06:13.092+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove members [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:13.175+0000 INFO [o.n.c.n.Server] raft-server: bound to 0.0.0.0:7000
2019-03-13 15:06:14.021+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58088
2019-03-13 15:06:14.251+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58090
2019-03-13 15:06:21.790+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Cluster discovery service started
2019-03-13 15:06:21.803+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}, {memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}, {memberId=MemberId{bdf87a06}, info=CoreServerInfo{raftServer=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-1.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-1.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:06:21.803+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:21.814+0000 INFO [o.n.c.i.ClusterBinder] Published: ClusterId{uuid=1144130e-fdbe-4157-988a-436e58d235b7}
2019-03-13 15:06:21.815+0000 INFO [o.n.c.i.ClusterBinder] Already bound to cluster: ClusterId{uuid=1144130e-fdbe-4157-988a-436e58d235b7}
2019-03-13 15:06:21.828+0000 INFO [o.n.c.c.s.LocalDatabase] Starting with storeId: Store{creationTime:1544642757823, randomId:6597478259518778525, upgradeTime:1544642757823, upgradeId:1}
2019-03-13 15:06:21.876+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /var/lib/neo4j/data/databases/graph.db
2019-03-13 15:06:21.909+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /var/lib/neo4j/data/databases/graph.db
2019-03-13 15:06:21.909+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Format not configured. Selected format from the store: RecordFormat:StandardV3_4[v0.A.9]
2019-03-13 15:06:22.170+0000 INFO [o.n.k.NeoStoreDataSource] No commits found after last check point (which is at LogPosition{logVersion=0, byteOffset=90})
2019-03-13 15:06:22.195+0000 INFO [o.n.k.i.DatabaseHealth] Database health set to OK
2019-03-13 15:06:22.504+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last transaction id in metadata store 2
2019-03-13 15:06:22.510+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Start id of last committed transaction in transaction log 1
2019-03-13 15:06:22.510+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last committed transaction id in transaction log 2
2019-03-13 15:06:22.511+0000 INFO [o.n.c.c.s.m.t.LastCommittedIndexFinder] Last committed consensus log index committed into tx log -1
2019-03-13 15:06:22.511+0000 INFO [o.n.c.c.s.m.t.ReplicatedTransactionStateMachine] Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (Label) Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (RelationshipType) Updated lastCommittedIndex to -1
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.m.t.ReplicatedTokenStateMachine] (PropertyKey) Updated lastCommittedIndex to -1

Forum did not let me to attach logs file, sorry if logs are long

tech-mint · March 13, 2019, 3:19pm

logs - final part

2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.CommandApplicationProcess] Restoring last applied index to 0
2019-03-13 15:06:22.512+0000 INFO [o.n.c.c.s.CommandApplicationProcess] Resuming after startup (count = 0)
2019-03-13 15:06:22.520+0000 INFO [o.n.c.n.Server] catchup-server: bound to 0.0.0.0:6000
2019-03-13 15:06:22.525+0000 INFO [o.n.c.c.c.s.RaftState] First leader elected: MemberId{531e9f4a}
2019-03-13 15:06:22.528+0000 INFO [o.n.c.n.Server] backup-server: bound to 127.0.0.1:6362
2019-03-13 15:06:22.531+0000 INFO [o.n.c.m.SenderService] Creating channel to: [neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000] 
2019-03-13 15:06:22.534+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19701
2019-03-13 15:06:22.536+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:22.551+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0x4e27fe80, L:/10.128.0.45:55262 - R:neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000]
2019-03-13 15:06:22.553+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:55262 remote neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000
2019-03-13 15:06:22.561+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:27.895+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000)
2019-03-13 15:06:27.903+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:27.904+0000 WARN [o.n.c.m.SenderService] Failed to connect to: neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000. Retrying in 100 ms
2019-03-13 15:06:28.006+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:28.208+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:28.611+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:29.413+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:30.523+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member removed MembershipEvent {member=Member [10.130.1.54]:5000 - b78d7d4f-1ebd-477a-9a39-4c13eee6ea0e,type=removed}
2019-03-13 15:06:30.525+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[], removed=[{memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}]}
2019-03-13 15:06:30.525+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:30.525+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove member [MemberId{531e9f4a}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:31.017+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:32.621+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:34.224+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:35.828+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:36.347+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.129.0.104:59194
2019-03-13 15:06:36.356+0000 INFO [o.n.c.m.SenderService] Creating channel to: [neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000] 
2019-03-13 15:06:36.356+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:36.357+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0xc771bf7f, L:/10.128.0.45:45486 - R:neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000]
2019-03-13 15:06:36.357+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:45486 remote neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000
2019-03-13 15:06:36.360+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:36.369+0000 INFO [o.n.c.c.c.s.RaftState] Leader changed from MemberId{531e9f4a} to MemberId{495e65e3}
2019-03-13 15:06:36.369+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19702
2019-03-13 15:06:37.430+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:39.032+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:40.635+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.238+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.861+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000)
2019-03-13 15:06:42.862+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:42.863+0000 WARN [o.n.c.m.SenderService] Failed to connect to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000. Retrying in 100 ms
2019-03-13 15:06:42.965+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.167+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.570+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.840+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:43.841+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0x573c8293, L:/10.128.0.45:55462 - R:neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000]
2019-03-13 15:06:43.841+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:55462 remote neo4j-core-0.neo4j.rag-uat.svc.cluster.local/10.130.1.54:7000
2019-03-13 15:06:43.983+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:06:44.219+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member added MembershipEvent {member=Member [10.130.1.54]:5000 - ecbfcc66-9b7b-4d01-9483-8deedb3b817f,type=added}
2019-03-13 15:06:44.220+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{531e9f4a}, info=CoreServerInfo{raftServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-0.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-0.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:06:44.220+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:06:44.371+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:45.621+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member removed MembershipEvent {member=Member [10.129.0.104]:5000 - 890c1610-3d35-45f6-9cde-c5fe27a4804c,type=removed}
2019-03-13 15:06:45.625+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[], removed=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}]}
2019-03-13 15:06:45.625+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{bdf87a06}, MemberId{531e9f4a}]
2019-03-13 15:06:45.625+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Not safe to remove member [MemberId{495e65e3}] because it would reduce the number of voting members below the expected cluster size of 3. Voting members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:45.974+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:47.577+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:48.885+0000 INFO [o.n.c.c.c.RaftMachine] Election timeout triggered
2019-03-13 15:06:48.888+0000 INFO [o.n.c.c.c.RaftMachine] Election started with vote request: Vote.Request from MemberId{bdf87a06} {term=19703, candidate=MemberId{bdf87a06}, lastAppended=0, lastLogTerm=-1} and members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:48.888+0000 INFO [o.n.c.c.c.RaftMachine] Moving to CANDIDATE state after successfully starting election
2019-03-13 15:06:48.891+0000 INFO [o.n.c.m.RaftOutbound] No address found for MemberId{495e65e3}, probably because the member has been shut down.
2019-03-13 15:06:49.179+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:50.534+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:06:50.783+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:52.387+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:53.990+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:55.594+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:56.712+0000 INFO [o.n.c.c.c.RaftMachine] Failed to get elected. Got votes from: []
2019-03-13 15:06:56.712+0000 INFO [o.n.c.c.c.RaftMachine] Election started with vote request: Vote.Request from MemberId{bdf87a06} {term=19704, candidate=MemberId{bdf87a06}, lastAppended=0, lastLogTerm=-1} and members: [MemberId{bdf87a06}, MemberId{531e9f4a}, MemberId{495e65e3}]
2019-03-13 15:06:56.733+0000 INFO [o.n.c.p.h.HandshakeServerInitializer] Installing handshake server local /10.128.0.45:7000 remote /10.130.1.54:58408
2019-03-13 15:06:57.196+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:06:57.648+0000 INFO [o.n.c.c.c.RaftMachine] Moving to FOLLOWER state after receiving vote request from MemberId{531e9f4a} at term 19705 (I am at 19704)
2019-03-13 15:06:57.660+0000 INFO [o.n.c.c.c.s.RaftState] Leader changed from MemberId{495e65e3} to MemberId{531e9f4a}
2019-03-13 15:06:57.660+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Leader MemberId{bdf87a06} updating leader info for database default and term 19705
2019-03-13 15:06:58.800+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:00.402+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:00.760+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core member added MembershipEvent {member=Member [10.129.0.104]:5000 - 02077349-7bf3-4f41-afde-f86e2b348c35,type=added}
2019-03-13 15:07:00.762+0000 INFO [c.n.c.d.SslHazelcastCoreTopologyService] Core topology changed {added=[{memberId=MemberId{495e65e3}, info=CoreServerInfo{raftServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000, catchupServer=neo4j-core-2.neo4j.rag-uat.svc.cluster.local:6000, clientConnectorAddresses=bolt://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7687,http://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7474,https://neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7473, groups=[]}}], removed=[]}
2019-03-13 15:07:00.762+0000 INFO [o.n.c.c.c.m.RaftMembershipManager] Target membership: [MemberId{531e9f4a}, MemberId{bdf87a06}, MemberId{495e65e3}]
2019-03-13 15:07:02.005+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Scheduling handshake (and timeout) local null remote null
2019-03-13 15:07:02.006+0000 INFO [o.n.c.m.SenderService] Connected: [id: 0xdc46d48e, L:/10.128.0.45:45720 - R:neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000]
2019-03-13 15:07:02.006+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Initiating handshake local /10.128.0.45:45720 remote neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000
2019-03-13 15:07:02.010+0000 INFO [o.n.c.p.h.HandshakeClientInitializer] Installing: ProtocolStack{applicationProtocol=RAFT_1, modifierProtocols=[]}
2019-03-13 15:07:18.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:07:46.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:08:14.535+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:08:42.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:09:10.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:09:38.536+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:10:06.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:10:34.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:02.537+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:30.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:11:58.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown
2019-03-13 15:12:26.538+0000 INFO [o.n.c.c.c.m.MembershipWaiter] Leader commit unknown

DNS is completely unrelated. Since as far it is obvious from the log it tries to address each of them and other nodes were available to ping but did not reply back

david_allen · March 13, 2019, 7:43pm

The things I'm seeing which are most relevant are:

2019-03-13 15:06:42.861+0000 WARN [o.n.c.m.SenderService] Lost connection to: neo4j-core-2.neo4j.rag-uat.svc.cluster.local:7000 (neo4j-core-2.neo4j.rag-uat.svc.cluster.local/10.129.0.104:7000)

2019-03-13 15:06:48.891+0000 INFO [o.n.c.m.RaftOutbound] No address found for MemberId{495e65e3}, probably because the member has been shut down.

These logs indicate DNS isn't the problem, you are successfully connecting to other members of the cluster, but you're not getting a stable cluster because members are leaving/entering the cluster.

The thing to look for next is evidence that the docker containers you're launching are crashing in a loop, or something like that. This would explain this behavior. Can you verify that when you launch the pods they're staying up, or do you see events in the k8s log that indicate the pods are shutting down and getting auto-restarted by kubernetes?

Also, are you using any additional plugins or non-standard configuration which might interfere with any pods ability to start normally?

tech-mint · March 14, 2019, 5:51am

Not at all. All pods are healthy and don't CrashLoopBackOff. They were up and running otherwise I could not get into them via ssh and gather neo4j logs.

About the plugins: I only deployed a standard Openshift 3.9 nothing more or less.

If you look into the history of issue you'll see a kind of pattern happening to people deploying without helm via yamls. Helm is not supported directly in Openshift. I configured our statefulset based on its helm definitions ,that's it.

It's not likable for me to have to turn into single instance architecture

david_allen · March 15, 2019, 8:20pm

Hi @tech-mint - I'm not sure what other reports you're referring to, could you please link them? I have not seen this issue before with folks deploying neo4j via helm and YAML files. Where are you seeing the history of this issue?

I understand this is frustrating and will help out if I can. It appears the logs you've provided are probably from one of the nodes. The best next step is to get debug.log from all three cluster nodes in the core set, and then maybe upload them somewhere where we can take a look. The complete files would be helpful, so we can look at full context. Pasted snippets of this much data will be hard to look at with these forums.

I can tell you what we'll be looking for: it's going to be those raft cluster connection messages. We need a log of all 3 talking to each other, and how each of the 3 sees the progression of events. This will make clear whether a node is restarting, whether there's some intermediate network problem, whether you have bad cluster state on some node, etc. The other thing you have that's odd in your logs are these lines:

Scheduling handshake (and timeout) local null remote null

Those nulls are concerning, but I don't know enough to say why that might be happening.

tech-mint · March 17, 2019, 1:10pm

I installed Helm on Openshift and redeploy neo4j. Issue is resolved,
thanks for your follow up

david_allen · March 17, 2019, 1:43pm

Whoa! Congrats! But I have to ask -- what was different here?

Because I think you had a lot of frustrating trouble here, make sure to say what you did differently so the next person who finds this thread can do the same.

tech-mint · March 18, 2019, 6:25am

I previously was deploying with the files in the helm chart without using helm directly. maybe I was missing something or so. I was wrong that helm is not possible in Openshift. after I installed helm in openshift and deployed neo4j via helm everything got smooth and also it saves me time too.

thanks for your support

Topic		Replies	Views
Cannot connect to cluster using k8s Ingress Neo4j Graph Platform	13	3131	April 20, 2022
Helm charts to create Neo4j cluster Neo4j Graph Platform kubernetes , helm	5	705	June 9, 2021
Causal Cluster not forming Neo4j Graph Platform	5	5704	October 18, 2018
New4j Cluster Stuck in discovery Cluster	17	1795	May 15, 2021
Neo4j Causal Cluster fails to form despite service showing ok Cluster cluster	0	102	May 20, 2024

Neo4j HA Pods in Statefulset can't connect to the other cluster members

Related topics