|If you haven't heard about Big Data and you're working|
in IT industry, you might want to check your pulse :)
Having said that, to gather lots of data is not the biggest challenge. As so many times before, the technology itself, even though complex, will not be our biggest problem. Usually the big problem is everything else around it. People, the policies, laws and regulations, processes... and again people.
Bigger challenge will be the questions like where to get the information we'll need, when we need it, in a format that can be utilized? How to make sure the information will be valid and correct? Where and how to move and store the data? How to back it up? How to set the access levels and rights to meet both legal and ethical requirements?
Just to get the first steps right will have its challenges, not to mention what would follow. In a way, the whole thing works like a funnel. The stuff (data) you put from one end is the source for the data that comes out from the other end. So if we're having wrong data or we're missing some data in the first place, then the result will also be only partly true. And depending on the algorithms we're using and the code we've written, we might get good answers, bad answers, or in the worst case, very dangerous answers. You can easily think what might happen in army intelligence or in the health care business, if we get wrong and random answers on which we'll base our decisions. And when the data grows enough, we really WILL do our decisions based on that because there's no way of double checking it manually.
How to use it and how to make sure we handle and understand the data right. And how to process that information in a way that it will be useful. So that it will benefit us. And in the end in a way that it will benefit the mankind to put everything into broader picture as in the end, that's where it's going to affect eventually.
There are lots of opportunities with processing big amounts of data. We might find a whole lot of new kind of patterns that in smaller information junks would be invisible. This could lead into big steps in developing the urban environments in both logistical and environmental way. In healthcare - finding new kind of dependencies and relationships which might lead into new cures and treatments and longer life expectations. It could lead into understanding those physical and neurological areas as well as sociological dependencies that there are but that are impossible to see unless we have enough data. We might find answers to the food and water problem now and in the future. But... another side of the coin will be the misuse or the wrong interpretation of the results. And for some reason human beings have been always very good to misuse the good things. One example - and we're not far from there, looking what some countries like Iceland and Holland have done around genealogy - could be running searches within the data to find the hereditary illness or other genetic feature - racial as well. Putting people into different categories based on their genes. Does it ring any bell? For me it does, with a very bad echo.
Once again the technological evolution seems to take giant leaps as human beings are taking baby steps in their evolvement. Will there be a day when we compensate this difference with the technology like in the movies, having technological implants in our brains, that will be seen in the future. Some extra memory would come in handy sometimes :) Probably the answer is 'yes, that will happen'. But at the moment this gap between technological and human evolution is just growing, and the bigger the gap is, the bigger the risk will become we don't know how to use it the right way. Now is the time to take a mental step back, think, re-think, and then make the right decisions about how we'll utilize big data solutions. Decisions, that will carry us into the better future.