What does the following neo4j error mean?

cypher

(Andreas Manousakis) #1

I'm trying to run some queries on the date I've imported (while the rest are being imported) into neo4j but I'm getting the following error:

Something went wrong: "Error: Reference.set failed: First argument contains a string greater than 10485760 utf8 bytes in property 'users.auth0|5aeb486d249b5d6bdbe1a2d5.grass.0.data' ('{"0":"{","1":""","2":"0","3":""","4":":","5":""...')" and the application can't recover.

It doesn't happen all the time. When it does, I restart the PC, wait a bit and then queries run again until they don't. Meanwhile data are being imported while I process the already imported data.

PC specs:

  • core i5
  • 16GB RAM
  • 1TB HDD

OS: Ubuntu 18.04.1 LTS

Any suggestions?


(M. David Allen) #2

To be sure, we'd need to see your load code and how you're doing it. But from what I can see in the error message, it looks like (maybe) you're trying to process some file line by line, that file contains JSON in it, and maybe something about your input data is wonky.

Literally what the error is saying is that you're trying to pass as an argument that is a string value of about 10MB or so (10485760 bytes is about 10MB).

Hopefully it's not your intent to be putting such large string values into the database, as this isn't a good practice for indexing or performance anyway. Things like this sometimes will happen to me if say the linebreaks in my CSV are being misread or are malformatted or something like that.

This is just a guess in hopes it guides you the right way. If it's a bad guess, post your import code. :)


(Andreas Manousakis) #3

Hi David,

Thank you for the reply!

I used the following tutorial to import the whole bitcoin blockchain into neo4j. Everything was running smoothly, along with the queries, until they didn't.

Link: https://neo4j.com/blog/import-bitcoin-blockchain-neo4j/

The script keeps running fine. The blockchain is being imported, however queries won't run anymore as I get the error above. Sometimes they will, but 8/10 the neo4j browser crashes.

The activity monitor shows that RAM is almost maxed out.

Could hardware be the issue?

Any ideas or suggestions are welcome. :)


(Andreas Manousakis) #4

I restarted the PC and a log file was generated:

"There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 83558400 bytes for committing reserved memory.
Possible reasons:
The system is out of physical RAM or swap space
In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.

Out of Memory Error (os_linux.cpp:2743), pid=6378, tid=0x00007f1a85d83700

JRE version: (8.0_191-b12) (build )
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again"


(M. David Allen) #5

Well, that message is clear enough - you need to allocate more memory to Neo4j, which you can do by checking here:

https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/

The blockchain example dataset is quite large. I don't know off hand how much it needs but something like a minimum of 16gb or 32gb may be needed.


(Andreas Manousakis) #6

I re-did the whole procedure on a clean installation of ubuntu 18.04.1. The graph is minimal (some mb) but queries will still not work. The error is persistent while if it was memory issue it should have been resolved.

The only idea I am left with is delete the local directory of blockchain and download it again in case it s corrupted (it s stored in a hdd)


(Andreas Manousakis) #7

!Problem Solved!

The browser seems to be the issue. I don't really know why chromium started throwing errors. Chrome & Mozilla work, Chromium doesn't.