GoGreen PC Tune-Up™
Learn More

Insta-Install™
this is how ssl encrypt our websites
MTNCOMP | List View | Table View | myBlog (1747 Entries)
myBlog Home

Blog


Visual C# processing 10TB SQL database

SQL

by Mountain Computers Inc., Publication Date: Saturday, October 29, 2022

View Count: 270, Keywords: Visual C#, SQL 2019, 10TB, Data Processing, Hashtags: #VisualC# #SQL2019 #10TB #DataProcessing



Today was a big day,
 
I just had the chance to process work my dev team on a 10TB data set in SQL using Visual C#.
 
Amazing, 1 billion records up from the 500MB records I previously had to process.
 
Had a great time leading the team in the best and fastest way to process 500 million records in 40 seconds, and 1 billion records in 1 minute 20 seconds.  We even had to consider MPP techniques, yet yielded to just a massive core processing and blazing fast memory and storage.
 
Postmortem: The prototype in VBA once confirmed by stakeholders, it would take 1.5 to 2 months to process the results as calculated once the experiment started and it was stopped within 2 weeks to re-evaluate the expectations of the outcomes. The VBA solution was proven accurate by the statisticians and so I ported the code to accelerate the results even though we were working on a 32 core multi-processor system with the ability to life the country. Ironically, the port from VBA to C# moved the results to 45 minutes and with a few more tweaks to less than 15 minutes given the needs of the output and processing parameters.
 
REF: large data processing gone wild.
 
more to come...

if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.

© 2024 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.