Neo4j and Snowflake Cloud Database

Neo4j Graph Platform ETL-Tool
1

Trying to get Neo4j ETL to function with Snowflake. Is the source code available? or can the connection be evaluated? The system is unable to read the tables. It is able to identify the available schemas.

  • 90.9% - 00:00:00.010 -

  • 9.1% - 00:00:00.001 -

  • Crawling tables

  • Retrieving tables

  • Retrieving tables for schema <xxx.LXPRDUSRF>

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableRetriever.retrieveTablesFromMetadata(TableRetriever.java:242)
    at schemacrawler.crawl.TableRetriever.retrieveTables(TableRetriever.java:118)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$17(SchemaCrawler.java:585)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:582)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Processed 33 rows for

  • Retrieved 33 tables

  • Retrieving table columns

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

  • Could not log SQL warnings for result set
    net.snowflake.client.jdbc.SnowflakeSQLException: Result set has been closed.
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.raiseSQLExceptionIfResultSetIsClosed(SnowflakeBaseResultSet.java:74)
    at net.snowflake.client.jdbc.SnowflakeBaseResultSet.getWarnings(SnowflakeBaseResultSet.java:286)
    at sf.util.DatabaseUtility.logSQLWarnings(DatabaseUtility.java:248)
    at schemacrawler.crawl.MetadataResultSet.next(MetadataResultSet.java:485)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumnsFromMetadata(TableColumnRetriever.java:304)
    at schemacrawler.crawl.TableColumnRetriever.retrieveColumns(TableColumnRetriever.java:107)
    at schemacrawler.crawl.SchemaCrawler.lambda$crawlTables$18(SchemaCrawler.java:603)
    at sf.util.StopWatch.time(StopWatch.java:185)
    at schemacrawler.crawl.SchemaCrawler.crawlTables(SchemaCrawler.java:600)
    at schemacrawler.crawl.SchemaCrawler.crawl(SchemaCrawler.java:817)
    at schemacrawler.utility.SchemaCrawlerUtility.getCatalog(SchemaCrawlerUtility.java:75)
    at org.neo4j.etl.sql.DatabaseClient.getCatalog(DatabaseClient.java:148)
    at org.neo4j.etl.sql.DatabaseCatalog.(DatabaseCatalog.java:81)
    at org.neo4j.etl.sql.metadata.TableInfoAssembler.(TableInfoAssembler.java:35)
    at org.neo4j.etl.commands.DatabaseInspector.buildSchemaExport(DatabaseInspector.java:35)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:110)
    at org.neo4j.etl.commands.rdbms.GenerateMetadataMapping.call(GenerateMetadataMapping.java:31)
    at org.neo4j.etl.cli.rdbms.GenerateMetadataMappingCli.run(GenerateMetadataMappingCli.java:118)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:42)
    at org.neo4j.etl.util.CliRunner.run(CliRunner.java:35)
    at org.neo4j.etl.NeoIntegrationCli.main(NeoIntegrationCli.java:43)

2

Same issue here.

Is there a way to connect to snowflake and retrieve the metadata?

3

Hello David

Alternative Solution:
If you are using AWS cloud then you can eliminate all this by subscribing to other services from Amazon like Glacier or similar services in amazon ecosystems and then retrieve the relevant data from amazon using AWS command Line Interface.

But if you want to deal with snowflakes then it is better to get Professional Services help from Snowflakes CSR.

Thanks
Sameer

4

Hello @davidrsetti , do you want to retrieve the data or the metadata ? My alternative options (my 1st preference ) is data pipeline in python , ETL tools like Talend.

Talend -> https://www.talend.com/partners/snowflake/

If you just need metadata, I have created a blog for sql server. There are alot of data dictionary in snowflake (let me know if you need a query), that you can change and retrieve the metadata .

http://www.dominickumar.com/blog/sql-server-database-er-metadata-in-neo4j/

@sam_gijare -> AWS Glacier is an AWS archival storage. Accessing data in and out of Glacier is paid and there is a timeframe for the data availability. You need additional services like AWS glue to crawl, and you dont need additional services for metadata, since snowflake already has data dictionary .