Now that we have been using the Neo4J graph database for a few years and have integrated and using the GDS library we are now moving to a causal cluster. But Neo4J has not integrated the GDS library to run on a cluster. What? GDS algorithms are allegedly parallelized, but they are unable to run on a cluster? I remember that being the entire point of using Hadoop and Spark - eliminate the transport of data across the network. Yet here we are again? Starting to look at TIgerGraph.
From the perspective of the core database and fabric, we use logical sharding which means that by design, there shouldn't be too many cross shard edges. However, you can match entities across shards in order to allow cross shard joins. This developer guide gives a great overview: https://neo4j.com/developer/neo4j-fabric-sharding/
As I'm sure you understand, many graph algorithms don't distribute very well (eg. https://www.cs.bgu.ac.il/~elkinm/book.pdf). That's why libraries like Spark's GraphX/GraphFrames only ever offered a few algorithms. Knowing this limitation, our focus has been on developing compression techniques and highly performant algorithms that can operate in a scale up context. We've successfully benchmarked our algorithms on graphs with hundreds of billions of nodes and relationships using widely available cloud compute instances.
I appreciate that this topic is a little dated now. So I was wondering if there has been any change/progress on this topic. As I understand it a limitation of the Neo4j sharding approach (please correct me if I am wrong) is that it is not possible to create cross shard relationships. @alicia_frame1 am I correct that the proposed approach (link above) requires that the entire graph being analysed is in a single shard? How is this scaling?
