Taking long time to insert the data

General Random
1

Dear All,

We are trying to insert 85000 record using multiple JSON files using a loop. While the individual file execution without loop executes within a few seconds. When we load all the files in the loop the insertion is taking looong time. What are we missing?? Any help will be highly appreciated.

public static void runCyperQuery(String fileName, String path, String tenantId, String type)
	{
		loadQueries(fileName);
		Driver driver = GraphDatabase.driver(GlobalProperties.neo4j_url, AuthTokens.basic(GlobalProperties.neo4j_username, GlobalProperties.neo4j_password));
		try (Session session = driver.session())
		{
			if (mymap != null && mymap.size() > 0)
			{
				for (Map.Entry<String, String> entry : mymap.entrySet())
				{
					if (entry.getKey().equals("device"))
					{
						**File folder = new File(path);**
**						File[] listOfFiles = folder.listFiles();  /* Directory %/**
**						for (File temp : listOfFiles) /* For Loop */**
**						{**
**							if (temp.getName().startsWith("devices"))**
**							{**
**								String queryString = entry.getValue().replace("<FileName>", path + temp.getName());**
**								insertQuery(session, queryString);**
**							}**
**						}**
					} else
					{
						String queryString = entry.getValue().replace("<FileName>", path.replace("\\", "/"));
						insertQuery(session, queryString);  /* Insert Query Method Invoked */
					}
				}
				if (tenantId != null && type != null)
				{
					sendGET(type, tenantId);
				}
				logger.info("Inserted Successfully");
			}
			session.close();
		} catch (Exception e)
		{
			logger.error("Exceptin in runCyperQuery query ", e);
		}
		driver.close();
	}

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

public static void insertQuery(Session session, String queryString)
	{
		insertQuery(session, queryString, null);
	}

	@SuppressWarnings("unused")
	public static void insertQuery(Session session, String queryString, String tenantId)
	{
		try
		{
			if (session != null)
			{
				String greeting = session.writeTransaction(new TransactionWork<String>()
				{
					@Override
					public String execute(Transaction tx)
					{
						StatementResult result = null;
						result = tx.run(queryString);
						if (result != null && result.list().size() > 1 && result.single().values().get(0).asInt() == 0)
						{
							logger.error("** Query is not inserted in storage device : " + result.summary());
						}
						logger.info("Inserted Successfully");
						return result.toString();
					}
				});
			}
		} catch (Exception e)
		{
			logger.error("Exceptin in insert query ", e);
		}

	}
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2

Thanks for the question. I'm not seeing where any examples of the actual cypher queries being executed. Can you provide one of those please?

Note: in general, it looks like you might be trying to create a new Cypher transaction for each and every record being inserted. This will have bad performance. Suggest batching something like 10k records at a time per transaction.

Thanks!
-Ryan