First pentaho/kettle job ingest paginated REST API data into neo4j

I will draft a detailed tutorial in the future, but I've got some initial success!

I'm querying Auvik (a network & infrastructure discovery/monitoring platform) via their REST API which returns JSON results (limited to 100 per page).

First perform a cypher query and UNWIND a list of API urls we wish to query.
Get nextURL, and set it to a kettle variable
Get one page of API results
if the nextURL value is not empty go get another page (previous step - loop)
if nextURL is empty go get the next API url query from the cypher list then loop above steps again
if nextURL from cypher list is empty then end the job, we're all done.

I'm also exporting the JSON to a file for ingestion into our SQL datawarehouse.

Thanks for the help from community members!

1 Like