After updating a project from Neo4J 3.5.12 to 3.5.19 I found that the database would only start if I disabled the existing APOC plugin.
Not thinking much about it I went and go ahead and updated the APOC plugin as well.
Then the database would not start, after checking the plugins/ folder I found that both the old and new plugin jar were there. After removing the old one the database started without any problems.
It would be good to prompt the user to remove the old plugin.