I have 2 needs as of now where I need some guidance desperately(I am new to Neo4J as well): So, for one of my projects, I need to build ETL pipelines within GCP for: 1. Transferring the Historical data from BigQuery to Neo4J 2. Transferring the Incremental data from BigQuery to Neo4J
Now, I am done with creating my pipelines in BQ for incremental load. How should I proceed? I can see some approaches suggesting: 1. Airflow. Will it work for huge amount of data? Will it be able to scale? Also, I think it will take time to load the data into Neo4J if the data volume is huge.(p.s.: Although, client doesn't have an existing composer instance; so, it will be painful to convince them for the same. If need be, and if there is no option, then I will do it). 2. The second approach that I could find was using Cloud DataProc where Apache Spark jobs can be written with Neo4J Connector.
Any suggestions on which approach should I follow? If there is any other approach than these, feel free to let me know. It will help me a ton!