Big Data - salvation for the people or a big headache?

If you haven't heard about Big Data and you're working
in IT industry, you might want to check your pulse :)

If you haven't heard about Big Data, you're either far from the IT business, living in the hermit junction, or you should really check your pulse. As usual with the new trends, it shouldn't really be about the bits and bites. It's not either only about getting all the data together to be able to process it, but the understanding what it's all about and what could and should be done with it. So the crucial success factor will be the code that we create for the data. Gathering data into one place is easy. I do it everyday with my USB drive(s). I get them filled with data in no time. But different question is how am I able to benefit from the data I've stored. Do I have the means to find whatever I'm looking for, added with some intelligence, and creating something totally new from what I've got. Who are the most thrilled about this? Where's the money in short term? I don't think I'm far from the truth if I'd guess it's the advertising business. That's where the investments for Big Data will go first. I mean in private sector. I'm sure the intelligence agencies in several countries have done this forever and they're also the ones who have already, no doubt about it, put a lot of money and time to be the first ones to take advantage of it. But leaving them aside from this, after the advertisement industry the next ones that will follow are governmental usage, like taxation, healthcare, threat analyzes, science etc. And on private sector this would mean especially the larger companies who are trying to get more clear and accurate picture about the markets, the development needs of their products and the course the company is now vs where it should be going. So in general, everyone who would want to get better and smarter results from the data they have.

Having said that, to gather lots of data is not the biggest challenge. As so many times before, the technology itself, even though complex, will not be our biggest problem. Usually the big problem is everything else around it. People, the policies, laws and regulations, processes... and again people.
Bigger challenge will be the questions like where to get the information we'll need, when we need it, in a format that can be utilized? How to make sure the information will be valid and correct? Where and how to move and store the data? How to back it up? How to set the access levels and rights to meet both legal and ethical requirements?

Just to get the first steps right will have its challenges, not to mention what would follow. In a way, the whole thing works like a funnel. The stuff (data) you put from one end is the source for the data that comes out from the other end. So if we're having wrong data or we're missing some data in the first place, then the result will also be only partly true. And depending on the algorithms we're using and the code we've written, we might get good answers, bad answers, or in the worst case, very dangerous answers. You can easily think what might happen in army intelligence or in the health care business, if we get wrong and random answers on which we'll base our decisions. And when the data grows enough, we really WILL do our decisions based on that because there's no way of double checking it manually.

How to use it and how to make sure we handle and understand the data right. And how to process that information in a way that it will be useful. So that it will benefit us. And in the end in a way that it will benefit the mankind to put everything into broader picture as in the end, that's where it's going to affect eventually.

There are lots of opportunities with processing big amounts of data. We might find a whole lot of new kind of patterns that in smaller information junks would be invisible. This could lead into big steps in developing the urban environments in both logistical and environmental way. In healthcare - finding new kind of dependencies and relationships which might lead into new cures and treatments and longer life expectations. It could lead into understanding those physical and neurological areas as well as sociological dependencies that there are but that are impossible to see unless we have enough data. We might find answers to the food and water problem now and in the future. But... another side of the coin will be the misuse or the wrong interpretation of the results. And for some reason human beings have been always very good to misuse the good things. One example - and we're not far from there, looking what some countries like Iceland and Holland have done around genealogy - could be running searches within the data to find the hereditary illness or other genetic feature - racial as well. Putting people into different categories based on their genes. Does it ring any bell? For me it does, with a very bad echo.

Once again the technological evolution seems to take giant leaps as human beings are taking baby steps in their evolvement. Will there be a day when we compensate this difference with the technology like in the movies, having technological implants in our brains, that will be seen in the future. Some extra memory would come in handy sometimes :) Probably the answer is 'yes, that will happen'. But at the moment this gap between technological and human evolution is just growing, and the bigger the gap is, the bigger the risk will become we don't know how to use it the right way. Now is the time to take a mental step back, think, re-think, and then make the right decisions about how we'll utilize big data solutions. Decisions, that will carry us into the better future.