Visual C# processing 10TB SQL database

Today was a big day,

 

I just had the chance to process work my dev team on a 10TB data set in SQL using Visual C#.

 

Amazing, 1 billion records up from the 500MB records I previously had to process.

 

Had a great time leading the team in the best and fastest way to process 500 million records in 40 seconds, and 1 billion records in 1 minute 20 seconds.  We even had to consider MPP techniques, yet yielded to just a massive core processing and blazing fast memory and storage.

 

Postmortem: The prototype in VBA once confirmed by stakeholders, it would take 1.5 to 2 months to process the results as calculated once the experiment started and it was stopped within 2 weeks to re-evaluate the expectations of the outcomes. The VBA solution was proven accurate by the statisticians and so I ported the code to accelerate the results even though we were working on a 32 core multi-processor system with the ability to life the country. Ironically, the port from VBA to C# moved the results to 45 minutes and with a few more tweaks to less than 15 minutes given the needs of the output and processing parameters.

 

REF: large data processing gone wild.

 

more to come...