There is so much that has been written and spoken about ‘Big Data’, that anything more might be an overkill. A lot of companies have invested in tools to process big data because it is the rule of the day. For small and mid-sized businesses, investing in big data most often become sunken cost. They do not have the vast amounts of data that require large scale analysis and most often end up losing the competitive edge that they looked to get when investing. A smarter investment for them and most other organizations will be Small Data.
What is Small Data?
In a lot of cases, such as new exploratory experiments or medical trials, the amount of data at your disposal is limited. This is small data – totally contrary to the behemoth data sets that we call big data. Organizations often have to make inferences from small data because it is practically impossible to get a lot of data in these specific cases. The ability to draw conclusions from a limited data stream is important because it can assist in getting closer to a lot of breakthrough inventions.
Why is decision making harder with Small Data?
Imagine that you are a drug manufacturer, who is on the lookout for a breakthrough drug to treat a rare mosquito related disorder. Firstly, finding patients for clinical trials is going to be a mammoth task. Secondly, no one will be willing to let their bodies be used for experimentation for a prolonged period. You will have a few patients, and a few chances to conduct trials. You will have to determine the end result based on a limited amount of data. How will you achieve this?
Solving the Big Problem of Small Data – Cold Spring Harbor Laboratory
A lot of statistical methods for solving real-world problems were developed in an era prior to the computing age. But with supercomputers at our disposal, the need of the hour is to rework a lot of formulas and algorithms. The Cold Spring Harbor Laboratory, which comprises of a team of statisticians and scientists that do engineering abroad, has developed a software for determining probability distribution from a small amount of data.
This invention can prove to be phenomenal for fields like medical science, physics, chemistry, material science and even quantum mechanics. It gives a measure of how certain or uncertain an estimate is, which is essentially what statistics and mathematics have tried to answer all along. Known as Density Estimation using Field Theory, or DEFT, this tool is freely available and is an open source software.
The Road Ahead
Small data is going to be vital for small corporations that need a starting point for their marketing and business efforts. By ascertaining decisions taken on the basis of limited amounts of data, they can have a means of gaining competitive advantage. Small data is going to be critical. After all, if you compile a list of top 100 decisions ever taken, 60% of them will point towards small data*