Data OLAP OLTP ETL and other Data Operations on Full Scale Systems
by Mountain Computers Inc., Publication Date: Friday, November 1, 2019
View Count: 992, Keywords: Microsoft SQL, 2019, OLAP, OLTP, ETL, ITIL, SDLC, Full Scale, Hashtags: #MicrosoftSQL #2019 #OLAP #OLTP #ETL #ITIL #SDLC #FullScale
In the past 5 years, we have had to do some pretty massively scaled data sets for businesses, ranging from a few 100 megabytes to a few terabytes, to 500 terabytes.
First of all, know your data and who is using it and who is touching it,
when, from where, and what are the outcomes and goals. Data
availability in both terms of analytic processing, the OLAP side, and
then the transaction processing, the OLTP side, has to be fast and
efficient, very solid, and very strong to secure and protect.
How do you handle this much data safely and securely? Easy answer: you don't.
How do you trust your data to be complete? Does the technology take care of it for you?
What verification methods are best to ensure no data leakage? We have a few techniques and tools.
What about data transfer and load? we use to leverage BCP and now we have powerscripts to massively load and transfer data by the 100s of megabytes per second, and then it just depends on how distributed your data needs to be.
For us, we have no single point of failure in most systems. We have redundancy, but can you have redundancy on 100-500TB of data? Yes and No. You just have to trust the distributed data technology can self heal and self monitor and also self manage itself.
Where does that leave us today with modern RDBMS and other forms of Distributed Data Systems? We are not in a pickle, we just need to get used to accepting data from wherever, learning to keep and store what is essential and discard the rest.
Primary Data (research) versus Secondary Data (boiled data raw primary data) is a balancing act.
If you have compliance and retention needs, then address those as you go.
more to come...
if you found this article helpful, consider contributing $10, 20 an Andrew Jackson or so..to the author. more authors coming soon
FYI we use paypal or patreon, patreon has 3x the transaction fees, so we don't, not yet.
© 2024 myBlog™ v1.1 All rights reserved. We count views as reads, so let's not over think it.
