Showing results for 
Search instead for 
Did you mean: 

Head's Up! Site migration is underway. Phase 2: migrate recent content

Thoughts on replacing RDS with neo4j for our use case


Who We Are: We are a startup web app serving mainly religious purposes.

What We Do: I guess the best publicly known term to describe our service is a CRM. More specifically we are branching into a ChMS (Church Management System). We have 13K+ organizations that we maintain data on and have 550+ users at this point. What we focus on is group shared data merged with private data. Users can create contacts and then share these contacts with other groups/users to admin/manage/view. And while everyone can have their own group level access rights, the individual user can add data points on the contacts that are like notes, tasks, events, etc. that are private/shared/public. This methodology helps serve all of the users and groups at a single time, while also helping users track their own statistics privately. Security is a must because we are storing financial data as well.

Our Current Stack: React UI -> GraphQL API -> RDS (MySQL 8.0.17). We manage access rights at the API level to request only the data the user has access to from RDS. For this we are using JSON field types to store layers of data for each row in the db.

How our UI works: we have data categories, Contacts, Tasks, Communication, Events, Finances. Now all of these are tied together but in a relational database they are each their own table structure. We have views associated with each data category List, Card, Table, Kanban, Calendar, Map. Our goal is to represent the same type of data in different views with the same filters. I built out a filter mechanism that is pretty advance and allows user to build filters at a 5 layer depth using conjunctions to group layers together. "Get all contacts that have addresses in Georgia that are tagged with X, Y, and Z, but not M, N, and O, and have either A or B". So the filters work and filter the data and now we are working on viewing that data. In a table view it is pretty basic. We can paginate the responses to keep API from overloading with trying to filter 27K contacts (organizations and individuals) and just filter and return 25 at a time. But when we push that over to a calendar view, now we want to show all of those contacts plotted on a month with their tasks, events, and activities. In order to get this data we have a few options but none are perfect. Either we get all of the contacts that match the filter, and then get all of their events, tasks, and activities, that are within a month date range. OR we get all of the events, tasks, and activities within a date range and then filter those based on the linked contacts that match the custom filter. Either way we are processing way more data than required to when the results may be a very small subset. If the filter returns a large number of contacts for instance 13K, then we are processing all 13K of those contacts for events, tasks, and activities that match our month.

Why switch to Neo4j? From trying to make this better I came across neo4j 4.0 boasts that it has schema based security so any property can be limited access (which we are doing right now with custom access checking) and that it can process related queries like this much much more optimized than can be done with a relational database. Right now I have to get filtered data and match it up against another set of filtered data to just get a single relational data set.

We already have an API that is 50% to completion with all of the data schema already built just working through building mutations. I know ways to get graphql to do what we want with RDS, it just doesn't seem very efficient. I think it sounds like a perfect case for neo4j, but I would love expert opinion.

Nodes 2022
NODES 2022, Neo4j Online Education Summit

All the sessions of the conference are now available online