SQL database ETL tricks and tips
by Mountain Computers Inc., Publication Date: Sunday, April 18, 2021
View Count: 1902, Keywords: SQL, MSSQL, ETL, Export, Transfer, Load, Tips, Tricks, Hashtags: #SQL #MSSQL #ETL #Export #Transfer #Load #Tips #Tricks
Here are some tips for my clients whom I am helping some some huge data sets this weekend.
1. head and tail your datasets to look at format before, during and after. look at the schema and mappings.
2. check for errors in your logs as your go... remember logs and triggers on issues need to be caught early.
3. make sure no indexes are enabled on bulk load and transfers. remember, if any indexes are enabled afterwards, the max is 3, and remember your foreign keys.
we have a 1.2TB dataset and that is going to be chugged in and out at about 50,000 records per second on one machine and over 500,000 records per second on a larger processor. the smallest data set is 180 million rows, so that won't take too long, regardless of the number of columns. the JSON sets will be fast!
The processing will take some 3 weeks to render... *ahem* down the results. talk about batching gone crazy!!
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.
© 2024 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.
