Unable to Upgrade to v4.0.0 from v3.0.1 (Community Edition)

Hi,

I'm trying to upgrade the Neo4j Community edition Server v3.0.1 to latest v4.0.0 (Single Instance)

I tried several ways but in all the approaches, the v4.0.0 didn't work. The logs that got generated were:

2020-01-24 12:21:19.331+0000 INFO [o.n.k.i.a.i.IndexingService] [system] Constraint Index( 3, 'constraint_952bbd70', UNIQUE BTREE, :label[1](property[0]), native-btree-1.0 ) is ONLINE.
2020-01-24 12:21:19.374+0000 INFO [o.n.s.s.a.CommunitySecurityModule] Completed migration of 1 user into system graph.
2020-01-24 12:21:19.411+0000 INFO [o.n.d.d.DefaultDatabaseManager] Creating 'graph.db' database.
2020-01-24 12:21:19.412+0000 INFO [o.n.d.d.DefaultDatabaseManager] Starting 'graph.db' database.
2020-01-24 12:21:19.418+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [graph.db] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /opt/neo4j/neo4j-4.0.0/neo4j-community-4.0.0/data/databases/graph.db
2020-01-24 12:21:19.419+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [graph.db] Selected format 'RecordFormat:StandardV4_0[SF4.0.0]' for existing store /opt/neo4j/neo4j-4.0.0/neo4j-community-4.0.0/data/databases/graph.db with format 'RecordFormat:StandardV3_4[v0.A.9]'
2020-01-24 12:21:19.426+0000 INFO [o.n.k.i.s.DatabaseMigrator] [graph.db] Starting upgrade of database
2020-01-24 12:21:19.436+0000 WARN [o.n.k.d.Database] [graph.db] Exception occurred while starting the database. Trying to stop already started components. The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
..
..
Suppressed: java.lang.RuntimeException: Error reading transaction logs, recovery not possible. To force the database to start anyway, you can specify 'unsupported.dbms.tx_log.fail_on_corrupted_log_files=false'. This will try to recover as much as possible and then truncate the corrupt part of the transaction log. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.
..
..
Caused by: org.neo4j.kernel.impl.transaction.log.entry.UnsupportedLogVersionException: Transaction logs contains entries with prefix -6, and the lowest supported prefix is -10. This indicates that the log files originates from an older version of neo4j, which we don't support migrations from.

Approach 1:
The recommended upgrade path as per the documentation that initially I chose:
3.0.1 --> 3.2.14 --> 3.4.17 --> 3.5.14 --> 4.0.0

The following steps were executed to perform the upgrade:
Upgrade to v3.2.14
(i) Stopped the Neo4j Server
(ii) Download and extracted the Neo4j v3.2.14 Community version
(iii) Replaced the data folder copied from v3.0.1
(iv) Enabled the config: dbms.allow_format_migration=true
(v) Started the v3.2.14 server.

The logs suggested that the upgrade was successful:

2020-01-24 10:31:01.031+0000 INFO  Starting upgrade of database
2020-01-24 10:31:01.376+0000 INFO  Migrating Indexes (1/3):
2020-01-24 10:31:01.378+0000 INFO    10% completed
..
2020-01-24 10:31:01.379+0000 INFO    100% completed
2020-01-24 10:31:01.379+0000 INFO  Migrating Legacy indexes (2/3):
2020-01-24 10:31:01.379+0000 INFO    10% completed
..
2020-01-24 10:31:01.380+0000 INFO    100% completed
2020-01-24 10:31:01.380+0000 INFO  Migrating Store files (3/3):
2020-01-24 10:31:01.386+0000 INFO    10% completed
..
2020-01-24 10:31:01.387+0000 INFO    100% completed
2020-01-24 10:31:02.518+0000 INFO  Successfully finished upgrade of database

Then similar steps were taken to upgrade to v3.4.17 (config: dbms.allow_upgrade=true) and the logs suggested that the upgrade was successful:

2020-01-24 10:35:17.088+0000 INFO  Starting upgrade of database
2020-01-24 10:35:17.719+0000 INFO  Migrating Explicit indexes (1/4):
2020-01-24 10:35:17.727+0000 INFO    10% completed
..
2020-01-24 10:35:17.732+0000 INFO    100% completed
2020-01-24 10:35:17.733+0000 INFO  Migrating Store files (2/4):
2020-01-24 10:35:17.897+0000 INFO    10% completed
..
2020-01-24 10:35:17.898+0000 INFO    100% completed
2020-01-24 10:35:17.898+0000 INFO  Migrating Native label scan index (3/4):
2020-01-24 10:35:17.921+0000 INFO    10% completed
..
2020-01-24 10:35:17.922+0000 INFO    100% completed
2020-01-24 10:35:17.923+0000 INFO  Migrating Counts store (4/4):
2020-01-24 10:35:17.923+0000 INFO    10% completed
..
2020-01-24 10:35:17.924+0000 INFO    100% completed
2020-01-24 10:35:17.930+0000 INFO  Successfully finished upgrade of database

Similar steps were taken to upgrade to v3.5.14 (config: dbms.allow_upgrade=true) but here the logs that suggests upgrade of DB didn't get generated. However, the neo4j server started and was getting connected both by the browser and the application code. All I could see the following in neo4j.log:

`2020-01-24 11:47:55.686+0000 INFO  Starting...
2020-01-24 11:47:58.844+0000 INFO  Deprecated index providers in use: lucene-1.0 (32 indexes). Use procedure 'db.indexes()' to see what indexes use which index provider.
2020-01-24 11:48:03.813+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2020-01-24 11:48:05.679+0000 INFO  Started.
2020-01-24 11:48:06.627+0000 INFO  Remote interface available at http://localhost:7474/

debug.log also didn't have any errors (except for the warnings for procedures that failed to load from apoc JAR):

2020-01-24 12:17:26.901+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2020-01-24 12:17:27.008+0000 INFO [o.n.b.BoltServer] Bolt enabled on 0.0.0.0:7687.
2020-01-24 12:17:28.488+0000 INFO [o.n.i.d.DiagnosticsManager] --- SERVER STARTED START ---
2020-01-24 12:17:29.317+0000 INFO [o.n.i.d.DiagnosticsManager] --- SERVER STARTED END ---
2020-01-24 12:17:42.566+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database graph.db is unavailable.
2020-01-24 12:17:42.579+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2020-01-24 12:17:42.590+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutting down 'graph.db' database.
2020-01-24 12:17:42.590+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2020-01-24 12:17:42.593+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" @ txId: 1008565 checkpoint started...
2020-01-24 12:17:44.546+0000 INFO [o.n.k.i.s.c.CountsTracker] Rotated counts store at transaction 1008565 to [/opt/neo4j/neo4j-3.5.14/neo4j-community-3.5.14/data/databases/graph.db/neostore.counts.db.a], from [/opt/neo4j/neo4j-3.5.14/neo4j-community-3.5.14/data/databases/graph.db/neostore.counts.db.b].
2020-01-24 12:17:44.615+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" @ txId: 1008565 checkpoint completed in 2s 20ms
2020-01-24 12:17:44.616+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in version 46
2020-01-24 12:17:44.750+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics START ---
2020-01-24 12:17:44.751+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics END ---

But when upgrading from v3.5.14 to v4.0.0 (config: dbms.allow_upgrade=true), got the errors as mentioned above.

Approach 2:
v3.0.1 -> v3.5.14 (Direct)

When upgrading from v3.0.1 directly to v3.5.14, the neo4j.log suggested that all went well:

2020-01-24 13:13:17.960+0000 WARN  You are using an unsupported version of the Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 8, OpenJDK(TM) 8 or IBM J9.
2020-01-24 13:13:17.991+0000 INFO  ======== Neo4j 3.5.14 ========
2020-01-24 13:13:18.007+0000 INFO  Starting...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.eclipse.collections.impl.utility.ArrayListIterate (file:/opt/neo4j/neo4j-3.5.14/direct-from-3.0.1/neo4j-community-3.5.14/lib/eclipse-collections-9.2.0.jar) to field java.util.ArrayList.elementData
WARNING: Please consider reporting this to the maintainers of org.eclipse.collections.impl.utility.ArrayListIterate
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-01-24 13:13:19.702+0000 INFO  Starting upgrade of database
2020-01-24 13:13:20.048+0000 INFO  Migrating Indexes (1/5):
2020-01-24 13:13:20.080+0000 INFO    10% completed
..
2020-01-24 13:13:20.082+0000 INFO    100% completed
2020-01-24 13:13:20.082+0000 INFO  Migrating Explicit indexes (2/5):
2020-01-24 13:13:20.083+0000 INFO    10% completed
..
2020-01-24 13:13:20.084+0000 INFO    100% completed
2020-01-24 13:13:20.084+0000 INFO  Migrating Store files (3/5):
2020-01-24 13:13:20.253+0000 INFO    10% completed
..
2020-01-24 13:13:20.256+0000 INFO    100% completed
2020-01-24 13:13:20.256+0000 INFO  Migrating Native label scan index (4/5):
2020-01-24 13:13:20.611+0000 INFO    10% completed
..
2020-01-24 13:13:20.941+0000 INFO    100% completed
2020-01-24 13:13:20.941+0000 INFO  Migrating Counts store (5/5):
2020-01-24 13:13:21.083+0000 INFO    10% completed
..
2020-01-24 13:13:21.509+0000 INFO    100% completed
2020-01-24 13:13:21.595+0000 INFO  Successfully finished upgrade of database, took 1s 892ms
2020-01-24 13:13:23.149+0000 INFO  Deprecated index providers in use: lucene-1.0 (32 indexes). Use procedure 'db.indexes()' to see what indexes use which index provider.
2020-01-24 13:13:29.214+0000 INFO  Bolt enabled on 127.0.0.1:7687.
2020-01-24 13:13:30.908+0000 INFO  Started.
2020-01-24 13:13:32.099+0000 INFO  Remote interface available at http://localhost:7474/

Upgrade from this (3.5) version to v4.0.0 gave the same error logs.

So for now, I have upgraded the database directly from v3.0.1 to v3.5.14 (Though this is not the recommended path yet I chose because of the confirmation from the logs generated)

Any reasons why its is not getting upgraded to v4.0.0? (In each step, the server was shutdown before copying the data folder.)

Without knowing exactly what you're doing for 3.5.14 to 4.0.0, one item suggested to me was that 4.0 does not access the 'dot' db (i.e. graph.db) as a naming conversion seen in previous migrations. I was told to get rid of the '.db' portion of the database name. I'm essentially asking the same question here...

If you are able to successfully migrate from 3.5. to 4.0 please share your steps.