Can't un-quarantine a non-system database with dbms.quarantineDatabase command

Neo4j Graph Platform Cypher
1
  • neo4j version, 4.4.18

i am running 3 neo4j instances as a cluster
after intentionally make one neo4j instance disk full, it is quarantined,

  1. make one nj instance disk full using fallocate
nj-2:/data> fallocate -l 200G gentoo_root.img
fallocate: fallocate failed: No space left on device
  1. the instance is now quarantined
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name       | aliases | access       | address                                                                                               | role       | requestedStatus | currentStatus | error                                                                                                                                 | default | home |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "graph.db" | []      | "read-write" | "nj-2.xxxx.7687" | "unknown"  | "online"        | "quarantined" | "Quarantine marker file ('/data/databases/graph.db/quarantine_marker') cannot be written for database DatabaseId{62a99502[graph.db]}" | TRUE    | TRUE |
| "graph.db" | []      | "read-write" | "nj-0.xxx:7687" | "follower" | "online"        | "online"      | ""                                                                                                                                    | TRUE    | TRUE |
| "graph.db" | []      | "read-write" | "nj-1.xxxx:7687" | "leader"   | "online"        | "online"      | ""                                                                                                                                    | TRUE    | TRUE |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  1. i removed the gentoo_root.img file from the instance,
  2. i am using dbms.quarantineDatabase to recover the node, it does not work, it says the instance is 'Not quarantined previously'
neo4j@system> CALL dbms.quarantineDatabase("graph.db",false);
+-----------------------------------------------------------+
| databaseName | quarantined | result                       |
+-----------------------------------------------------------+
| "graph.db"   | FALSE       | "Not quarantined previously" |
+-----------------------------------------------------------+

still

------------------------------------------------------------------------------------------+
| "graph.db" | []      | "read-write" | "nj-2.xxxx:7687" | "unknown"  | "online"        | "quarantined" | "Quarantine marker file ('/data/databases/graph.db/quarantine_marker') cannot be written for database DatabaseId{62a99502[graph.db]}" | TRUE    | TRUE |

i need to delete pvc to restart the instance again to recover it...

2

@jeffwang313.work

When you connected to cypher-shell/browser and issues the CALL dbms.quarantineDatabase("graph.db",false); are you connecting as neo4j://<Neo4jHost>:7687 or bolt://<Neo4jHost>:7687

Per

It is recommended to run the quarantine procedure over 
the bolt:// protocol rather than neo4j://, which may route
 requests to unexpected instances.
1 Like
3

thanks Per!
it works now after using bolt:// and specify the host
db shows unknown state after running dbms.quarantineDatabase
then i did db restart and it went fine

i did this
1.
cypher-shell -u neo4j -p neo4j123 -a bolt://10.42.101.50:7687 -d system

neo4j@system> CALL dbms.quarantineDatabase("graph.db",false);
+------------------------------------------------------------------------------------+
| databaseName | quarantined | result                                                |
+------------------------------------------------------------------------------------+
| "graph.db"   | FALSE       | "RaftMessage applier failed: No space left on device" |
+------------------------------------------------------------------------------------+

| "graph.db" | [] | "read-write" | "nj-2:7687" | "unknown" | "online" | "initial" | "Database with name graph.db already exists." | TRUE | TRUE |

neo4j restart

+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name       | aliases | access       | address                                                                                               | role       | requestedStatus | currentStatus   | error | default | home |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "graph.db" | []      | "read-write" | "nj-2:7687" | "follower" | "online"        | "store copying" | ""    | TRUE    | TRUE |
| "graph.db" | []      | "read-write" | "nj-0:7687" | "follower" | "online"        | "online"        | ""    | TRUE    | TRUE |
| "graph.db" | []      | "read-write" | "nj-1:7687" | "leader"   | "online"        | "online"        | ""    | TRUE    | TRUE |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+