Neo4j server 5.5 failed to start

In Kubernetes environment am trying to install neo4j and start the server. But it failed to start. and gives the below exception

To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Defaulted container "mahesh-deployement-mdm" out of: mahesh-deployement-mdm, databackup-init (init)
NEO4JLABS_PLUGINS has been renamed to NEO4J_PLUGINS since Neo4j 5.0.0.
The old name will still work, but is likely to be deprecated in future releases.
Installing Plugin 'apoc' from /var/lib/neo4j/labs/apoc-*-core.jar to /var/lib/neo4j/plugins/apoc.jar
Applying default values for plugin apoc to neo4j.conf
/docker-entrypoint.sh: line 361: expand_commands_optionally: command not found
==== RUNNING ENTRY POINT CUSTOMIZATIONS =====
==== NAMESPACE: sandbox-cwb-mahesh-external
==== graphileonSSOEnabled parameter is not set, fallback to default values
2023-03-31 10:11:19.290+0000 WARN Setting 'dbms.allow_upgrade' is removed. It no longer has any effect.
2023-03-31 10:11:19.294+0000 WARN Use of deprecated setting 'dbms.jvm.additional'. It is replaced by 'server.jvm.additional'.
2023-03-31 10:11:19.294+0000 WARN Use of deprecated setting 'dbms.memory.pagecache.size'. It is replaced by 'server.memory.pagecache.size'.
2023-03-31 10:11:19.294+0000 WARN Use of deprecated setting 'dbms.memory.heap.max_size'. It is replaced by 'server.memory.heap.max_size'.
2023-03-31 10:11:19.294+0000 WARN Use of deprecated setting 'dbms.memory.heap.initial_size'. It is replaced by 'server.memory.heap.initial_size'.
2023-03-31 10:11:19.294+0000 WARN Use of deprecated setting 'dbms.unmanaged_extension_classes'. It is replaced by 'server.unmanaged_extension_classes'.
2023-03-31 10:11:19.295+0000 WARN Unrecognized setting. No declared setting with name: gitlab.username.
2023-03-31 10:11:19.295+0000 WARN Unrecognized setting. No declared setting with name: google.maps.api.key.
2023-03-31 10:11:19.295+0000 WARN Unrecognized setting. No declared setting with name: usps.api.endpoint.
2023-03-31 10:11:19.295+0000 WARN Unrecognized setting. No declared setting with name: usps.api.userid.
2023-03-31 10:11:19.296+0000 WARN Unrecognized setting. No declared setting with name: gitlab.access.token.
2023-03-31 10:11:19.296+0000 WARN Unrecognized setting. No declared setting with name: usps.api.target.
2023-03-31 10:11:19.306+0000 INFO Starting...
2023-03-31 10:11:19.835+0000 INFO This instance is ServerId{6e1bd683} (6e1bd683-6535-4776-a881-d00219434625)
2023-03-31 10:11:20.748+0000 INFO ======== Neo4j 5.5.0 ========
2023-03-31 10:11:31.273+0000 ERROR Failed to start Neo4j on 0.0.0.0:7474.
java.lang.RuntimeException: Error starting Neo4j database server at /data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:256) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:194) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:170) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:86) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:30) ~[neo4j-5.5.0.jar:5.5.0]
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@1682c08c' was successfully initialized, but failed to start. Please see the attached cause exception "Transaction logs are missing and recovery is not possible.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:370) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:92) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:247) ~[neo4j-5.5.0.jar:5.5.0]
... 5 more
Suppressed: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DatabaseLifecycles$AllDatabaseStopper@260d5273' failed to stop. Please see the attached cause exception "System database is not (yet) available".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:384) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.stopInstances(LifeSupport.java:212) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.stop(LifeSupport.java:120) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:97) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:247) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:194) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:170) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:86) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:30) ~[neo4j-5.5.0.jar:5.5.0]
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
at org.neo4j.kernel.database.SystemGraphDatabaseIdRepository.execute(SystemGraphDatabaseIdRepository.java:52) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.SystemGraphDatabaseIdRepository.getByName(SystemGraphDatabaseIdRepository.java:40) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.DatabaseIdRepository.getByName(DatabaseIdRepository.java:55) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.MapCachingDatabaseIdRepository.lambda$getByName$0(MapCachingDatabaseIdRepository.java:53) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
at org.neo4j.kernel.database.MapCachingDatabaseIdRepository.getByName(MapCachingDatabaseIdRepository.java:52) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.DatabaseIdRepository.getByName(DatabaseIdRepository.java:55) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseRepository.getDatabaseContext(DatabaseRepository.java:54) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles.defaultContext(DatabaseLifecycles.java:76) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles$AllDatabaseStopper.stop(DatabaseLifecycles.java:150) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:380) ~[neo4j-common-5.5.0.jar:5.5.0]
... 9 more
Caused by: org.neo4j.dbms.api.DatabaseManagementException: System database is not (yet) available
at org.neo4j.kernel.database.SystemGraphDatabaseIdRepository.execute(SystemGraphDatabaseIdRepository.java:52) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.SystemGraphDatabaseIdRepository.getByName(SystemGraphDatabaseIdRepository.java:40) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.DatabaseIdRepository.getByName(DatabaseIdRepository.java:55) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.MapCachingDatabaseIdRepository.lambda$getByName$0(MapCachingDatabaseIdRepository.java:53) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
at org.neo4j.kernel.database.MapCachingDatabaseIdRepository.getByName(MapCachingDatabaseIdRepository.java:52) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.DatabaseIdRepository.getByName(DatabaseIdRepository.java:55) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseRepository.getDatabaseContext(DatabaseRepository.java:54) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles.defaultContext(DatabaseLifecycles.java:76) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles$AllDatabaseStopper.stop(DatabaseLifecycles.java:150) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.stop(LifeSupport.java:380) ~[neo4j-common-5.5.0.jar:5.5.0]
... 9 more
Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:154) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:166) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:135) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:92) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseTransactions.beginTransaction(GraphDatabaseTransactions.java:113) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseTransactions.beginTransaction(GraphDatabaseTransactions.java:107) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseTransactions.beginTransaction(GraphDatabaseTransactions.java:102) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.impl.factory.GraphDatabaseTransactions.beginTx(GraphDatabaseTransactions.java:93) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.SystemGraphComponents.initializeSystemGraph(SystemGraphComponents.java:97) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:38) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:27) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:353) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:92) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:247) ~[neo4j-5.5.0.jar:5.5.0]
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@7d12429' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:340) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:57) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:86) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:649) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:377) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.Database.specificStart(Database.java:397) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.AbstractDatabase.start(AbstractDatabase.java:156) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles.startDatabase(DatabaseLifecycles.java:120) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles$SystemDatabaseStarter.start(DatabaseLifecycles.java:143) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:353) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:92) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:247) ~[neo4j-5.5.0.jar:5.5.0]
... 5 more
Caused by: java.lang.RuntimeException: Transaction logs are missing and recovery is not possible.
at org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck.checkForMissingLogFiles(Recovery.java:901) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck.init(Recovery.java:887) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:323) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:57) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:86) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:649) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:377) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.Database.specificStart(Database.java:397) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.kernel.database.AbstractDatabase.start(AbstractDatabase.java:156) ~[neo4j-kernel-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles.startDatabase(DatabaseLifecycles.java:120) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.dbms.database.DatabaseLifecycles$SystemDatabaseStarter.start(DatabaseLifecycles.java:143) ~[neo4j-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:353) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:92) ~[neo4j-common-5.5.0.jar:5.5.0]
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:247) ~[neo4j-5.5.0.jar:5.5.0]
... 5 more
2023-03-31 10:11:31.276+0000 INFO Neo4j Server shutdown initiated by request
2023-03-31 10:11:31.276+0000 INFO Stopped.

Can you explain a little more how you installed / set-up your k8s cluster.

The errors look a bit like a mixed install where some libraries / config are still lying around from an old installation?