I'm using the ETL tool to map my organization's relational database. Their database has decades of data across roughly 7000 tables.
I am currently doing the mapping step (not the importing) and it has been running for 6 hours.
I'm totally fine with waiting but just wanted to ask if the mapping step is potentially a long process? If not I can kill the process and investigate further.
Not sure what to expect, but it ran the mapping step for 2 days and it hadn't finished. I assumed something was wrong so I cancelled it and will investigate further.
Any advice as to whether or not the mapping stage should take minutes, hours, or days would be much appreciated.
Hi @susmitadaskgp, there is no static value. It all depends on your Data Model with Entities and Relationships.
What database are you using ? Before mapping did you had a check with the Data Dictionary ?
I highly recommend using a small sampling of the existing data before doing a massive import.
It may take multiple tries to debug it, so there will be occasions where you'll be waiting for a long time, only to discover the data wasn't imported correctly.
Also, you can get an estimate for the total time. i.e. if you load 1% of the data, you can estimate the total time by multiplying by 100.
The other thing I recommend is to clone the DB at each step as a backup if you do multiple steps. That way, if a step messes up the data, then you can go back to the previously good DB.
Hi @susmitadaskgp, are you migrating a OLTP or a OLAP database ? Maybe not all the 1000 tables, you have to migrate. You can create intermediate views also in the source database. There are always so many options to choose from.
Always have discovery and identifications of entities process first. This will save a lot of time.
@susmitadaskgp, like I guess, I think you are migrating an OLAP database. Sure, I can help you. Before I can advise more on this, I need more in-depth knowledge on the data. If you want we can connect in skype dominicvivek06 , and discuss there further.