Big Data is any amount of data so large that it is hard to process; too much information to handle. Yet what happens if you want a seemingly simple fact of trend, and have to look through a whole country’s census date to find it?
Big Data needs develop with the Cloud storage needs. More storage means there is more data collected and stored, but then in turn we need more storage space for the programs to analyse the initial data.
A few points:
“Data is best optimized when you know how it all fits together”.
It is useful to strip down the data to what is relevant to your needs, but there is no neat dividing line; everything interconnects. If it too simplified, you may leave out something useful, or that might be needed in the future, or that could lead you onto a new, overlooked market ; If it isn’t simplified enough you risk losing sight of the data, or taking so long to analysis it that the opportunities have already passed you by.
There is always the chance that the information you want falls in between the questions. We have all had the experience of survey forms that don’t give us the option that best reflects our opinion. Bear this in mind if you have any influence in how the data is collected. Asking the wrong questions will skew the data.
Companies need to collect all possible data, using tags, using data management platforms, and use a tool to sort, correlate and analysis it. It helps if the final representation is something that can be visualized.
Older methods of analysis used a representative sample of the data find patterns; hopefully this showed the pattern for the whole system, not just the sample at that particular time. Big Data aims to use every bit of relevant information, not just a token or (hopefully) typical sample. Problems can occur when two or three patterns exist at once; Big data aims to find the set of products or services that work for the different individuals in the market place, and avoid one answer that averages out to something that is not right for anybody.
The Analysis of Big Data reflects the immediate past, which is close to the present if the analysis programs are fast enough. How this relates to the future is never certain. We might follow trends with the data we have, but not pick up on some factor that abruptly changes everything. In fact, we need to know that if we act on the data patterns that we have, and make a change, then we might be the factor that changes everything! Similarly, analysing the past patterns may not give insight into how a new, future product will perform, though it may be helpful in showing how the product is adopted over time. New products and ideas mean new categories and questions for data collection; quite possibly questions that have not been previously asked. Combining analysis with computer simulations may alleviate some of the myopic tendencies here.