Unable to Upgrade to v4.0.0 from v3.0.1 (Community Edition)

[puneet]

Hi,

I'm trying to upgrade the Neo4j Community edition Server v3.0.1 to latest v4.0.0 (Single Instance)

I tried several ways but in all the approaches, the v4.0.0 didn't work. The logs that got generated were:

2020-01-24 12:21:19.331+0000 INFO [o.n.k.i.a.i.IndexingService] [system] Constraint Index( 3, 'constraint_952bbd70', UNIQUE BTREE, :label[1](property[0]), native-btree-1.0 ) is ONLINE.
2020-01-24 12:21:19.374+0000 INFO [o.n.s.s.a.CommunitySecurityModule] Completed migration of 1 user into system graph.
2020-01-24 12:21:19.411+0000 INFO [o.n.d.d.DefaultDatabaseManager] Creating 'graph.db' database.
2020-01-24 12:21:19.412+0000 INFO [o.n.d.d.DefaultDatabaseManager] Starting 'graph.db' database.
2020-01-24 12:21:19.418+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [graph.db] Selected RecordFormat:StandardV3_4[v0.A.9] record format from store /opt/neo4j/neo4j-4.0.0/neo4j-community-4.0.0/data/databases/graph.db
2020-01-24 12:21:19.419+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [graph.db] Selected format 'RecordFormat:StandardV4_0[SF4.0.0]' for existing store /opt/neo4j/neo4j-4.0.0/neo4j-community-4.0.0/data/databases/graph.db with format 'RecordFormat:StandardV3_4[v0.A.9]'
2020-01-24 12:21:19.426+0000 INFO [o.n.k.i.s.DatabaseMigrator] [graph.db] Starting upgrade of database
2020-01-24 12:21:19.436+0000 WARN [o.n.k.d.Database] [graph.db] Exception occurred while starting the database. Trying to stop already started components. The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
org.neo4j.kernel.impl.storemigration.StoreUpgrader$DatabaseNotCleanlyShutDownException: The database is not cleanly shutdown. The database needs recovery, in order to recover the database, please run the old version of the database on this store.
..
..
Suppressed: java.lang.RuntimeException: Error reading transaction logs, recovery not possible. To force the database to start anyway, you can specify 'unsupported.dbms.tx_log.fail_on_corrupted_log_files=false'. This will try to recover as much as possible and then truncate the corrupt part of the transaction log. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.
..
..
Caused by: org.neo4j.kernel.impl.transaction.log.entry.UnsupportedLogVersionException: Transaction logs contains entries with prefix -6, and the lowest supported prefix is -10. This indicates that the log files originates from an older version of neo4j, which we don't support migrations from.

Approach 1:
The recommended upgrade path as per the documentation that initially I chose:
3.0.1 --> 3.2.14 --> 3.4.17 --> 3.5.14 --> 4.0.0

The following steps were executed to perform the upgrade:
Upgrade to v3.2.14
(i) Stopped the Neo4j Server
(ii) Download and extracted the Neo4j v3.2.14 Community version
(iii) Replaced the data folder copied from v3.0.1
(iv) Enabled the config: dbms.allow_format_migration=true
(v) Started the v3.2.14 server.

The logs suggested that the upgrade was successful:

2020-01-24 10:31:01.031+0000 INFO  Starting upgrade of database
2020-01-24 10:31:01.376+0000 INFO  Migrating Indexes (1/3):
2020-01-24 10:31:01.378+0000 INFO    10% completed
..
2020-01-24 10:31:01.379+0000 INFO    100% completed
2020-01-24 10:31:01.379+0000 INFO  Migrating Legacy indexes (2/3):
2020-01-24 10:31:01.379+0000 INFO    10% completed
..
2020-01-24 10:31:01.380+0000 INFO    100% completed
2020-01-24 10:31:01.380+0000 INFO  Migrating Store files (3/3):
2020-01-24 10:31:01.386+0000 INFO    10% completed
..
2020-01-24 10:31:01.387+0000 INFO    100% completed
2020-01-24 10:31:02.518+0000 INFO  Successfully finished upgrade of database

Then similar steps were taken to upgrade to v3.4.17 (config: dbms.allow_upgrade=true) and the logs suggested that the upgrade was successful:

2020-01-24 10:35:17.088+0000 INFO  Starting upgrade of database
2020-01-24 10:35:17.719+0000 INFO  Migrating Explicit indexes (1/4):
2020-01-24 10:35:17.727+0000 INFO    10% completed
..
2020-01-24 10:35:17.732+0000 INFO    100% completed
2020-01-24 10:35:17.733+0000 INFO  Migrating Store files (2/4):
2020-01-24 10:35:17.897+0000 INFO    10% completed
..
2020-01-24 10:35:17.898+0000 INFO    100% completed
2020-01-24 10:35:17.898+0000 INFO  Migrating Native label scan index (3/4):
2020-01-24 10:35:17.921+0000 INFO    10% completed
..
2020-01-24 10:35:17.922+0000 INFO    100% completed
2020-01-24 10:35:17.923+0000 INFO  Migrating Counts store (4/4):
2020-01-24 10:35:17.923+0000 INFO    10% completed
..
2020-01-24 10:35:17.924+0000 INFO    100% completed
2020-01-24 10:35:17.930+0000 INFO  Successfully finished upgrade of database

Similar steps were taken to upgrade to v3.5.14 (config: dbms.allow_upgrade=true) but here the logs that suggests upgrade of DB didn't get generated. However, the neo4j server started and was getting connected both by the browser and the application code. All I could see the following in neo4j.log:

`2020-01-24 11:47:55.686+0000 INFO  Starting...
2020-01-24 11:47:58.844+0000 INFO  Deprecated index providers in use: lucene-1.0 (32 indexes). Use procedure 'db.indexes()' to see what indexes use which index provider.
2020-01-24 11:48:03.813+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2020-01-24 11:48:05.679+0000 INFO  Started.
2020-01-24 11:48:06.627+0000 INFO  Remote interface available at http://localhost:7474/

debug.log also didn't have any errors (except for the warnings for procedures that failed to load from apoc JAR):

2020-01-24 12:17:26.901+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2020-01-24 12:17:27.008+0000 INFO [o.n.b.BoltServer] Bolt enabled on 0.0.0.0:7687.
2020-01-24 12:17:28.488+0000 INFO [o.n.i.d.DiagnosticsManager] --- SERVER STARTED START ---
2020-01-24 12:17:29.317+0000 INFO [o.n.i.d.DiagnosticsManager] --- SERVER STARTED END ---
2020-01-24 12:17:42.566+0000 INFO [o.n.k.a.DatabaseAvailabilityGuard] Database graph.db is unavailable.
2020-01-24 12:17:42.579+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2020-01-24 12:17:42.590+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutting down 'graph.db' database.
2020-01-24 12:17:42.590+0000 INFO [o.n.g.f.GraphDatabaseFacadeFactory] Shutdown started
2020-01-24 12:17:42.593+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" @ txId: 1008565 checkpoint started...
2020-01-24 12:17:44.546+0000 INFO [o.n.k.i.s.c.CountsTracker] Rotated counts store at transaction 1008565 to [/opt/neo4j/neo4j-3.5.14/neo4j-community-3.5.14/data/databases/graph.db/neostore.counts.db.a], from [/opt/neo4j/neo4j-3.5.14/neo4j-community-3.5.14/data/databases/graph.db/neostore.counts.db.b].
2020-01-24 12:17:44.615+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] Checkpoint triggered by "Database shutdown" @ txId: 1008565 checkpoint completed in 2s 20ms
2020-01-24 12:17:44.616+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] No log version pruned, last checkpoint was made in version 46
2020-01-24 12:17:44.750+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics START ---
2020-01-24 12:17:44.751+0000 INFO [o.n.i.d.DiagnosticsManager] --- STOPPING diagnostics END ---

But when upgrading from v3.5.14 to v4.0.0 (config: dbms.allow_upgrade=true), got the errors as mentioned above.

Approach 2:
v3.0.1 -> v3.5.14 (Direct)

When upgrading from v3.0.1 directly to v3.5.14, the neo4j.log suggested that all went well:

2020-01-24 13:13:17.960+0000 WARN  You are using an unsupported version of the Java runtime. Please use Oracle(R) Java(TM) Runtime Environment 8, OpenJDK(TM) 8 or IBM J9.
2020-01-24 13:13:17.991+0000 INFO  ======== Neo4j 3.5.14 ========
2020-01-24 13:13:18.007+0000 INFO  Starting...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.eclipse.collections.impl.utility.ArrayListIterate (file:/opt/neo4j/neo4j-3.5.14/direct-from-3.0.1/neo4j-community-3.5.14/lib/eclipse-collections-9.2.0.jar) to field java.util.ArrayList.elementData
WARNING: Please consider reporting this to the maintainers of org.eclipse.collections.impl.utility.ArrayListIterate
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-01-24 13:13:19.702+0000 INFO  Starting upgrade of database
2020-01-24 13:13:20.048+0000 INFO  Migrating Indexes (1/5):
2020-01-24 13:13:20.080+0000 INFO    10% completed
..
2020-01-24 13:13:20.082+0000 INFO    100% completed
2020-01-24 13:13:20.082+0000 INFO  Migrating Explicit indexes (2/5):
2020-01-24 13:13:20.083+0000 INFO    10% completed
..
2020-01-24 13:13:20.084+0000 INFO    100% completed
2020-01-24 13:13:20.084+0000 INFO  Migrating Store files (3/5):
2020-01-24 13:13:20.253+0000 INFO    10% completed
..
2020-01-24 13:13:20.256+0000 INFO    100% completed
2020-01-24 13:13:20.256+0000 INFO  Migrating Native label scan index (4/5):
2020-01-24 13:13:20.611+0000 INFO    10% completed
..
2020-01-24 13:13:20.941+0000 INFO    100% completed
2020-01-24 13:13:20.941+0000 INFO  Migrating Counts store (5/5):
2020-01-24 13:13:21.083+0000 INFO    10% completed
..
2020-01-24 13:13:21.509+0000 INFO    100% completed
2020-01-24 13:13:21.595+0000 INFO  Successfully finished upgrade of database, took 1s 892ms
2020-01-24 13:13:23.149+0000 INFO  Deprecated index providers in use: lucene-1.0 (32 indexes). Use procedure 'db.indexes()' to see what indexes use which index provider.
2020-01-24 13:13:29.214+0000 INFO  Bolt enabled on 127.0.0.1:7687.
2020-01-24 13:13:30.908+0000 INFO  Started.
2020-01-24 13:13:32.099+0000 INFO  Remote interface available at http://localhost:7474/

Upgrade from this (3.5) version to v4.0.0 gave the same error logs.

So for now, I have upgraded the database directly from v3.0.1 to v3.5.14 (Though this is not the recommended path yet I chose because of the confirmation from the logs generated)

Any reasons why its is not getting upgraded to v4.0.0? (In each step, the server was shutdown before copying the data folder.)

[keithave (Keith Van Eaton)]

Without knowing exactly what you're doing for 3.5.14 to 4.0.0, one item suggested to me was that 4.0 does not access the 'dot' db (i.e. graph.db) as a naming conversion seen in previous migrations. I was told to get rid of the '.db' portion of the database name. I'm essentially asking the same question here...

Migrate data from 3.5.1 Community to 4.0.0 Community

graph.db is not a valid database name in neo4j 4.0. That's a folder name in 3.5. Version 3.5 didn't have the concept of named graphs, so basically that was just a directory name. Try moving it to "graph" or the default which comes with Neo4j 4.0, which is "neo4j" Naming rules for databases are as follows: Length must be between 3 and 63 characters. The first character of a name must be an ASCII alphabetic character. Subsequent characters must be ASCII alphabetic or numeric characters, …

If you are able to successfully migrate from 3.5. to 4.0 please share your steps.

[coursera-ricardor]

I had a similar problem.

I can't warranty that will work in all cases.

I didn't notice that I had some databases with an older version, so I needed to downgrade first to the 3.5.x If it is the case, upgrade first the databases, some of worked perfectly, others not, reporting different problems

After several hours, I found this suggestion:

This looks like the transaction logs have only been partially written. If that's true the datastore itself is unlikely to be corrupt.
WARNING: before doing any manual change in the data/graph.db folder be sure to have a backup copy in place.
Before proceeding, read the above paragraph. Backup taken? If yes, try to remove the transaction log files in data/graph.db by rm neostore.transaction.db.*. Try to start Neo4j. If it still fails to start, paste a snippet of data/graph.db/messages.log containing the latest startup sequence - this should give more insight.

Repair corrupted neo4j database (2.3.2 community)

 Selected format from the store files: RecordFormat:StandardV4_0[SF4.0.0]
2020-02-20 16:50:03.642+0000 WARN [o.n.k.d.Database] [graphgain] Exception occurred while starting the database. Trying to stop already started components. Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@46468f0' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible. To force the database to start anyway, you can specify 'dbms.recovery.fail_on_missing_files=false'. This will create new transaction log and will update database metadata accordingly. Doing this means your database integrity might be compromised, please consider restoring from a consistent backup instead.".

I needed to modify the neo4j.conf

/etc/neo4j$ sudo nano neo4j.conf

# Enable this to be able to upgrade a store from an older version.
#dbms.allow_upgrade=true
dbms.allow_upgrade=true

#********************************************************************
# Migration  recovery
#********************************************************************
dbms.recovery.fail_on_missing_files=false

NOTE: Check your graph directory

Steps

/etc/neo4j$ sudo service neo4j stop
/var/lib/neo4j/data/databases/graphgain$ sudo cp -R /home/myuser/neo4j/data_3.5.14/databases/graphgain/* ./
/var/lib/neo4j/data/databases/graphgain$ sudo chown -R neo4j:neo4j *

/var/lib/neo4j/data/databases/graphgain$ sudo rm neostore.transaction.db.*
/var/lib/neo4j/data/databases/graphgain$ sudo rm ../../transactions/graphgain/neostore.transaction.db.0
/etc/neo4j$ sudo service neo4j start
/etc/neo4j$ tail -f /var/log/neo4j/debug.log

Reconfigure the server

/etc/neo4j$ sudo nano neo4j.conf

# Enable this to be able to upgrade a store from an older version.
#dbms.allow_upgrade=true

#********************************************************************
# Migration  recovery
#********************************************************************
#dbms.recovery.fail_on_missing_files=false

/etc/neo4j$ sudo service neo4j start

[laissaouichaima3 (chaima)]

i am facing up some issues so i have a graph DB in neo4j community edition 3.1.3 and I need to migrate to a higher version that supports gds, I already read some documentations about upgrading and migration and I know that I should follow the path: 3.1 --> 3.2 --> 3.3 --> 3.4 --> 3.5 latest then migrate to 4.0 and upgrade till achieve the version that supports gds. so what the thing i miss is how to upgrade like what i should do to upgrade and migrate i have no idea about how and about the steps to follow. can you help me please?