Currently in my company, we import customer transaction data from oracle into neo4j using Luigi jobs on a scheduled manner for every one hour. We work on that data within neo4j to do graph analytics. The use case is fraud detection.
Process:
We make jdbc connections with oracle and use cypher queries inside the jobs to create nodes and relationships, do pre-preocessing and analytics for our process.
We schedule the jobs on an hourly basis since the whole workflow for the job takes 45-50 mins approx to complete.
Is there a way to create some sort of a python process to ingest data in real time or near real time without the use of Kafka (we already tried Kafka but we are having hard time implementing it)?
Because of this hourly schedule, we are missing valuable insights that we could generate if it had been done in near real time.