Troubleshooting Connection Issues to Neo4j

Hi @david_allen I'm still having a problem with this. I have tried to override these values with environment variables in the deployment / pod spec, but it seems like the docker sh script that is embedded in the container overrides my environment values.

Here's a look at the deployed pod spec with a few things redacted -- note the fqdn-here represents a real DNS fully qualified domain name that I've redacted here.

It is receiving some of my environment variables (e.g. I enable prometheus monitoring and those stick), but it just always seems to overwrite the advertised address to be 0.0.0.0 no matter what I do.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: eks.privileged
    prometheus.io/port: "2004"
    prometheus.io/scrape: "true"
  creationTimestamp: "2020-03-31T22:08:59Z"
  generateName: neo4j-6d6585bcbf-
  labels:
    app: neo4j
    pod-template-hash: 6d6585bcbf
  name: neo4j-6d6585bcbf-fl8pw
  namespace: alma
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: neo4j-6d6585bcbf
    uid: 38aefaa9-739c-11ea-8fd4-0aa6c32e78f9
  resourceVersion: "43523590"
  selfLink: /api/v1/namespaces/alma/pods/neo4j-6d6585bcbf-fl8pw
  uid: 38afec27-739c-11ea-8fd4-0aa6c32e78f9
spec:
  containers:
  - env:
    - name: NEO4J_ACCEPT_LICENSE_AGREEMENT
      value: "yes"
    - name: NEO4J_AUTH
      value: neo4j/Salido4u-2.78
    - name: NEO4J_dbms_mode
      value: single
    - name: NEO4J_metrics_prometheus_enabled
      value: "true"
    - name: NEO4J_metrics_prometheus_endpoint
      value: 0.0.0.0:2004
    - name: NEO4J_dbms_connectors_default_listen_address
      value: 0.0.0.0
    - name: NEO4J_dbms_logs_query_threshold
      value: 2s
    - name: NEO4J_dbms_logs_query_rotation_size
      value: 20m
    - name: NEO4J_dbms_logs_query_rotation_keep_number
      value: "7"
    - name: NEO4J_dbms_logs_query_time_logging_enabled
      value: "true"
    - name: NEO4J_dbms_logs_query_page_logging_enabled
      value: "true"
    - name: NEO4J_dbms_connector_bolt_address
      value: :7687
    - name: NEO4J_dbms_connector_https_advertised_address
      value: fqdn-here:7473
    - name: NEO4J_dbms_connector_http_advertised_address
      value: fqdn-here:7474
    - name: NEO4J_dbms_connector_bolt_advertised_address
      value: fqdn-here:7687
    image: neo4j:4.0.2-enterprise
    imagePullPolicy: IfNotPresent
    name: neo4j
    ports:
    - containerPort: 7474
      name: browser
      protocol: TCP
    - containerPort: 7687
      name: bolt
      protocol: TCP
    - containerPort: 2004
      name: metrics
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/neo4j/data/
      name: neo4jdata
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-2t5rf
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-192-168-174-9.ec2.internal
  nodeSelector:
    beta.kubernetes.io/instance-type: m4.large
  priority: 0
  restartPolicy: Always
  schedulerName: stork
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: neo4jdata
    persistentVolumeClaim:
      claimName: px-neo4j-pvc
  - name: default-token-2t5rf
    secret:
      defaultMode: 420
      secretName: default-token-2t5rf
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:08:59Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:09:00Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:09:00Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:08:59Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://1223f6779066aa0dab7c8c1f482d9f04584ceb623700594ca0095ef8e4a197fa
    image: neo4j:4.0.2-enterprise
    imageID: docker-pullable://neo4j@sha256:a090c2ed169a68bdbf7dd2f1e5b0c47891530d489dc7f5a5f43c8d719b5a32e4
    lastState: {}
    name: neo4j
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2020-03-31T22:09:00Z"
  hostIP: 192.168.174.9
  phase: Running
  podIP: 192.168.174.77
  qosClass: BestEffort
  startTime: "2020-03-31T22:08:59Z"

When I shell into the pod itself and cat logs/debug.log I can see it resets these addresses back to 0.0.0.0, and indeed when I try the WebSocket it again responds with 0.0.0.0 address.

Here's a snipped from that log note the bolt advertised address is now reset again to 0.0.0.0 -- what am I missing here?

2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager]                                 [ DBMS config ]
2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
2020-03-31 22:09:08.200+0000 INFO [o.n.i.d.DiagnosticsManager] DBMS provided settings:
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.discovery_advertised_address=neo4j-6d6585bcbf-fl8pw:5000
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.discovery_listen_address=0.0.0.0:5000
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.raft_advertised_address=neo4j-6d6585bcbf-fl8pw:7000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.raft_listen_address=0.0.0.0:7000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.transaction_advertised_address=neo4j-6d6585bcbf-fl8pw:6000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.transaction_listen_address=0.0.0.0:6000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.bolt.advertised_address=0.0.0.0:7687
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.bolt.enabled=true
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.http.advertised_address=0.0.0.0:7474
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.http.enabled=true
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.https.advertised_address=0.0.0.0:7473
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.https.enabled=false
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.default_listen_address=0.0.0.0
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.import=/var/lib/neo4j/import
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.logs=/logs
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.neo4j_home=/var/lib/neo4j
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.jvm.additional=-Djdk.tls.rejectClientInitiatedRenegotiation=true
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.logs.query.rotation.size=20971520
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.logs.query.threshold=2s
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.memory.pagecache.size=512M
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.mode=SINGLE
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.tx_log.rotation.retention_policy=100M size
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.windows_service_name=neo4j
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] metrics.prometheus.enabled=true
2020-03-31 22:09:08.214+0000 INFO [o.n.i.d.DiagnosticsManager] metrics.prometheus.endpoint=0.0.0.0:2004
2020-03-31 22:09:08.214+0000 INFO [o.n.i.d.DiagnosticsManager]