About the job:
MemDuo is seeking a highly competent Machine Learning Engineer experienced in Neo4j, ChromaDB, and text chunking. The goal is to enhance our existing Neuromorphic GraphRAG project by implementing a strategy for chunking unstructured texts in vector storage, and representing summaries of the chunks in our Neo4j graph. This involves creating "Summary Nodes" that summarize the main ideas or themes of each chunk and storing them in Neo4j with appropriate metadata and links to their locations in the vector store, ChromaDB.
What you’ll deliver:
-
Implement a smart chunking strategy for unstructured texts found in vector storage.
Extract and summarize the main themes or ideas of each text chunk into a short summary string ("Chunk Summary").
Create Neo4j nodes ("Summary Nodes") containing the "chunk summary" string, specific metadata, and pointers to full chunk locations in ChromaDB, along with pointers to preceding and following chunks.
Advise on the proper (self hosted or other) LLM to mitigate rate limiting issues inherent in the process.
Contact info@memduo.com or www.linkedin.com/in/BrianFischman