Delete old Neo4j Ops Manager logs

Neo4j Graph Platform Neo4j Operations Manager (NOM)
1

I have NOM 1.10.0 installed and its database in Neo4j is growing over time (as expected).
Now, my NOM database is 27GB with 200M allocated IDs.

I do not need old logs.

Is there a way to delete old logs?
Is there a way to configure neo4j-ops-manager-server to keep only the last x days of logs?

2

Hi!

We recently fixed issues around logs not getting cleaned up properly, so I recommend you update to NOM 1.10.1.

Regarding your question:
I just looked at the code and currently, there is no way to adjust that. Since we have something similar for the metrics, I will suggest that the team implement the same for one of the next versions.

In the meantime, if you are feeling the same about metrics, here is how to cut their retention time:
You can adjust the time your logs will be kept by passing the following server configuration at the start: metrics.cleanup.retention-time . It's a duration, so the value needs to be something like P14D for instance.

So e.g. something like

java -jar ./lib/server.jar
... all the normal stuff ...
--metrics.cleanup.retention-time=P14D

That will eliminate all older metrics the next time the automatic cleanup happens.

Cheers,
Sascha

3

Hello,

Thanks for your answer.
Just to be clear, when I talk about logs here, I mean old data in the Neo4j NOM database (nodes and relationships).

I do not see this parameter on the documentation (nor any metrics.*):

I updated the server to 1.10.1 (which updated correctly the agents to the same version) and added the parameter for 7 days:

sudo nano /etc/systemd/system/neo4j-ops-manager-server.service

...
[Service]
User=neo4j
ExecStart=/var/lib/neo4j-ops-manager-server/bin/server \
    --spring.neo4j.uri=neo4j+s://xxxxxxxxxxxxxx \
    --spring.neo4j.authentication.username=nom \
    --spring.neo4j.authentication.password=xxxxxxxxxxxxxx \
    --server.port=xxxxxxxxxxxxxx \
    --server.ssl.key-store-type=PKCS12 \
    --server.ssl.key-store=xxxxxxxxxxxxxx \
    --server.ssl.key-store-password=xxxxxxxxxxxxxx \
    --grpc.server.port=xxxxxxxxxxxxxx \
    --grpc.server.security.key-store-type=PKCS12 \
    --grpc.server.security.key-store=xxxxxxxxxxxxxx \
    --grpc.server.security.key-store-password=xxxxxxxxxxxxxx \
    --jwt.secret=xxxxxxxxxxxxxx \
    --logging.file.name=/var/log/nom-server/nom-server.log \
    --logging.level.root=warn \
    --logging.level.com.neo4j=warn \
    --metrics.cleanup.retention-time=P7D

Restart=on-failure
...

This update and the new parameter didn't change anything: I still see 30 days old data in both 3 tabs Host, Instance, and Database.

We now have 1.5B properties, 2.3B relationships, and 293M nodes in the NOM DB for a store size of 179Gb.

4

Hello,

After some time, the visible data in both 3 tabs Host, Instance, and Database are available only for the last week. This corresponds to the metrics.cleanup.retention-time.

But the used space and allocated IDs did not decrease: now 217Gb for 357M nodes

5

Hi,

could you solve the problem with the growing NOM database?

If not, it would be interesting to know what goes wrong. Since you could see that the metrics are now only shown for 7 days, maybe the query log data is not being cleaned up correctly. Could you please execute the following statements on the NOM database and attach the results here?

CYPHER runtime = parallel
MATCH(n)
RETURN LABELS(n)[0] AS l0, LABELS(n)[1] AS l1, COUNT(*) AS r
ORDER BY r DESC;

MATCH (e:Execution:Single)
WHERE e.endTime IS NOT NULL AND e.endTime < (timestamp() - 86400000)
RETURN count(e) AS ExecOlderThan24hCount

MATCH (e:Execution:Single)
RETURN datetime({epochmillis:min(e.endTime)})

Thanks and best regards
Alexander

6

Hello,

Sorry for the delay.

We ended up with a script to reclaim the unused space like Space reuse - Operations Manual

  • Save and stop
cypher-shell --non-interactive \
    -a neo4j+s://<url>:<port> \
    -u "${neo4jUser}" -p "${neo4jPassword}" \
    -d nom \
    "CALL db.checkpoint(); STOP DATABASE nom;"
  • Copy in place
rm /var/lib/neo4j/nom-schema.cypher
cd /var/lib/neo4j
neo4j-admin database copy nom nom --compact-node-store=true --temp-path=/data/tmp --verbose
  • Start
cypher-shell --non-interactive \
    -a neo4j+s://<url>:<port> \
    -u "${neo4jUser}" -p "${neo4jPassword}" \
    -d system \
    "START DATABASE nom;"
  • Apply schema
cypher-shell --non-interactive \
    -a neo4j+s://<url>:<port> \
    -u "${neo4jUser}" -p "${neo4jPassword}" \
    -d nom \
    -f /var/lib/neo4j/nom-schema.cypher
  • Restart service
service neo4j-ops-manager-server restart