Asking Questions of Vastness (and can one make a living at it)

I had a conversation with an unemployed programmer friend today. He observed that the tricks we used as data wranglers and database pipeline engineers are changing as companies merge and become enormous. Asking standard business questions like "how are people using my website?" is no longer just about phrasing the question and making pretty reports, but being clever about how to juggle such unweildy data.

One thing I learned quickly at I/Pro was processing data takes up space and time. For some reason I had believed before that computers were magic and that data was quick to work with because it was not tangible. Wrong! There's a whole branch of Computer Science devoted to how long things take to compute, and processing techniques for speeding this up without just throwing more memory and Megahertz at them. Strangely, the I/Pro's founders (and 5+ successor CEO's) ignored this property of data and (to this day) believe it is possible to provide a scalable business as their clients' data gets bigger. Oops. While I/Pro could not count Yahoo! or Netscape fast enough, somehow these companies have to be able to count themselves (and foot the cost). Each seems to do so in their own way, as far as I know.

So as programmers, getting jobs with big companies means gaining a knack for Hugeness. Is this commonly taught in school? You never encounter terabytes of data (yet) when learning to code at home. As companies merge into fewer behemoths, I suppose the number of needed Hugeness experts will decrease. It may also mean that programmers used to smaller data sets will have a harder time as the small companies get assimilated.

At least rock stars and celebrities don't have to worry about constant mergers or scaling problems (dandruff not withstanding).