What is Big Data?
As a customary, let us begin with a somewhat formal definition of big data. We have copied the below definition from Wikipedia. Read it anyway.
In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications
Don't worry if the above definition failed to incite your imagination. If you just need to copy the above definition and paste it in your presentation slide for management meeting tomorrow morning, then feel free to do so and leave this webpage. Otherwise, read on.
Okay. Now take a deep breath and look back to the above definition. It's actually a very good definition. All it says is, the term "Big Data" relates to any collection of data set which is so huge that traditional databases are unable to process them in timely manner. So how big is big enough? And what is the processing limit of the traditional databases?
How Big is Big?
Ask yourself this question.
What is the maximum database size you have seen so far?
Now I know if you are a professional in database and related subjects, you may be able to answer this question. But if you do not know anything about databases (then what are you doing here, anyway?), it's not possible for you to answer this. So let me ask an easier question. How much do you think is the size of the database where US government stores every citizen's social security number (SSN)? US population is somewhere around 330 million. If everybody has got one SSN, that database should approximately store 330 million records (I know it's over simplification, Bill, but there are people who use Apple). Is this database big enough? Probably not. Think of Facebook. As of 31 Dec 2012, they have 680 million monthly active users. Is that big enough? Not really. How about Google then? People perform 6,000 million searches per day in Google. Is that big data? Yep, may be - depending on how you store this data and what you intend do with this data. The thing is, storing just 6000 million rows is not a big thing in itself. You can do that using conventional technologies (read RDBMS) and tools (Oracle Database for example). But it can be more intriguing that that.
Endure this -
What if I ask you to store 6000 million of search phrases that are searched in Google everyday for 2 years continuously and at the end of it prepare a report on first 25 most searched keywords related to "football"?
Now you might (rightfully) say, “this is insane”. 6000 million for 2 years means (2 X 6000 X 365) 4,380,000,000,000 records. And even if we store it, how will we possibly perform analysis on that data? Gee! Welcome to the world of Big Data!
World of Big Data
Now I am sure, if you have been doing RDBMS and SQL all your life, you wouldn't give in so easily. While you understand that your favorite RDBMS database will never handle that much data, but you will definitely ask this question -
WHY on EARTH anybody needs to do that kind of WEIRD analysis in the first place?
Or you may also question the rationality of this -
What about the cost factor?
Because storing such a staggering amount of data and performing analysis on that will probably take hundreds of thousands of dollars. This means the return on investment will be pathetic!
And my answer will be - "yes, it would be pathetic. For you. Not for big data!" Because Big data does not use RDBMS, SQL queries or conventional databases. True, had we tried to tame big data using conventional technologies - we would have failed. And that is why scientists have found a different way of handling such a huge volume of data. I will show you how. But let me answer another question before that. The question is about the need.
Why do we need big data?
In order to run your business intelligibly, you have to have the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experiences. All of these abilities revolve around a simple fact - your ability to perform data driven analysis. Because that is how you discover knowledge. But the problem is, how do you get enough data? And if you happen to have enough data, how do you perform analysis on that data? These are 2 different questions. And answer to these questions will lead us to Big Data. SO let's explore them.
Conventionally, in order to have enough data, you will have to setup enough means to collect your data. That's not always easy and definitely not cheap.
Say, you are a cross-country fast food restaurant chain owner and you sell chicken burger and French fries. You are considering two new products, out of which you want to introduce one in your restaurants. The products are Mushroom burger and Pineapple Pie. If you are given the task to select one of these products over the other so as to maximize your profit, how would you take the decision? Clearly you have no other knowledge/experience/data about your customers liking or affinity towards one over the other.
If you have to take a data driven decision, the conventional way would be to run a survey campaign. Like, you might distribute a feedback form to all of your restaurant visitors to express their specific choices. And based on the data you collected thus, you may be able to take a decision. But there are 3 problems to this approach. First, the process is painfully slow. Second, the process is only limited to the choices of your existing customers and does not take into account your potential customers. And lastly, you need to design a computerized process to convert the results of your offline survey data to database records for analysis.
The above example demonstrates the typical scarcity of data when a business wants to grow on an uncharted territory. But this doesn't have to be like this always. With the advent of social media like Facebook and Twitter, it's relatively easy to get such data. Think of this, 680 millions of Facebook users are going to several restaurants every month, liking particular foods, commenting "wow! I like the Pineapple pie after this burger" or "I hate the smell of shroom pizza" now and then.
A Facebook Post showing one's affinity towards a food!
This can be an incredible source of information for your restaurant chain business. This data is already there. The question is how do we use it? Clearly, creating a conventional RDBMS-based data model to store this data is out of question because of the time and processing required to transform this unstructured textual data to a structured database schema. And even if you do that, there is this herculean task of analyzing this data out of that. Fundamentally, the conventional process requires a lot of data movement (read physical IO) which calls for a better process.
In case if you are still thinking, let me clarify that conventional data warehouse designs based on ER model or dimensional model are not suitable here. If you want to store this huge data in ER model - analytics will be dreadful due to huge number of joins that you need to traverse to retrieve records. If you want to ease out the data retrieval part by doing a dimensional modeling - the data will be unnecessarily voluminous because of data redundancy.
Big Data On Rescure - Map Reduce
So upto this point, we have established the need for big data for better business intelligence and we also understand the pain associated with lot of data. Many refer this as 3-Vs of Big Data, namely: Volume, Variety and Velocity (see below) which ultimately increases the complexity associated with Big Data processing.
Fortunately Big data comprises of methods and technologies to solve this problem. One of these methods is called - "Map Reduce". You may use Map Reduce framework to analyze the unstructured data through streaming. Streaming is a great thing - because when you stream data, you obviate the need for storing the data and while streaming you can actually analyze the data on the go to find the answers of your questions. "Map Reduce" is a separate topic and definitely outside the scope of this essay, but I intend to write on this when I have some time.
The important point to note is, big data provides us means to overcome the issue that were otherwise not solvable using conventional technique. The good thing is, once you start exploiting the potential of big data, both the scope and possibility of data driven analysis increases widely. Just like we saw in our restaurant chain example, leveraging big data analytics on social media data, now you can extend the scope of your analysis far more widely and perform it in comparatively lesser time.
Examples of Big Data Analytics
All these are not just theory. There are diverse use-cases of big data right now in this world. Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. And yet for every user they can perform market basket analysis to show you the list of products that you are most likely to buy, given your previous buying habits and buying habits of millions of customers like you. The NASA Center for Climate Simulation (NCCS) stores 32 peta-bytes of climate observations and simulations on the Discover supercomputing cluster - to enable various climate predictions. In physics, The Large Hadron Colliders in CERN, Geneva represent about 150 million sensors delivering data 40 million times per second. These data are processed to find out presence of new elementary particles in nature, among other things. All of these could not have been possible using conventional database technologies.
There is no doubt in my mind that Big Data is definitely the next big thing. However, like everything else, there are lots of hypes associated with it. In fact according to a Gartner analyst, Big Data is plunging into a “trough of disillusionment” that may dissuade many companies from pursuing analytics technology. But like everything else in this world, as a technologist we need to learn to discern the reality from hype.
Now that you know what big data is, I will conclude this article here. Here is a very useful infograph summarizing many concepts of Big Data, that you may wish to download. In my next article, we will discuss the implementation and use-cases of big data analysis.
Takwimu Kubwa ni nini? Kama kawaida, hebu kuanza na ufafanuzi kiasi fulani rasmi wa data kubwa. Tumekopisha ufafanuzi wa chini kutoka Wikipedia. Soma hata hivyo. Katika teknolojia ya habari, data kubwa ni mkusanyiko wa seti za data kubwa na ngumu sana kwamba inakuwa vigumu kusindika kwa kutumia zana za usimamizi wa hifadhidata au maombi ya jadi ya usindikaji wa data. Kama unataka tu kunakili ufafanuzi hapo juu na kuweka katika slide yako ya uwasilishaji kwa ajili ya mkutano wa usimamizi kesho asubuhi, basi kujisikia huru kufanya hivyo na kuondoka ukurasa huu. Vinginevyo, soma zaidi. Sawa basi. Sasa pumua kwa kina na ukumbuke ufafanuzi uliotajwa hapo juu. Kwa kweli ni ufafanuzi mzuri sana. "Hii inamaanisha kwamba ""data kubwa"" ni seti ya data ambayo ni kubwa sana kwamba hifadhidata za jadi haziwezi kuifanya kwa wakati." Kwa hiyo, ni kubwa kadiri gani? Na ni nini kikomo cha usindikaji wa hifadhidata za jadi? Kubwa Ni Kubwa Kadiri Gani? Jiulize swali hili. Ni nini ukubwa wa juu wa hifadhidata umeona hadi sasa? Kama wewe ni mtaalamu wa database na masuala yanayohusiana, unaweza kujibu swali hili. Lakini kama hujui chochote kuhusu hifadhidata (basi nini unafanya hapa, hata hivyo? (Sina uwezo wa kujibu swali hili. Kwa hiyo napenda kuuliza swali rahisi. Ni kiasi gani cha data ya serikali ya Marekani inahifadhi namba ya usalama wa kijamii ya kila raia? Idadi ya watu nchini Marekani ni karibu milioni 330. Ikiwa kila mtu ana SSN moja, hifadhidata hiyo inapaswa kuhifadhi rekodi milioni 330 (najua ni juu ya kurahisisha, Bill, lakini kuna watu ambao hutumia Apple). Je, hifadhidata hii ni kubwa vya kutosha? Yaelekea sivyo. Fikiria kuhusu Facebook. Kufikia Desemba 31, 2012, walikuwa na watumiaji milioni 680 wa kila mwezi. Je, hiyo ni kubwa vya kutosha? Si kweli. Vipi kuhusu Google basi? Watu wanafanya utafutaji wa milioni 6,000 kwa siku kwenye Google. Je, hiyo ni data kubwa? Ndiyo, inaweza kuwa - kulingana na jinsi ya kuhifadhi data hii na nini una nia ya kufanya na data hii. Kwa kweli, kuhifadhi safu milioni 6 sio jambo kubwa. Unaweza kufanya hivyo kwa kutumia teknolojia ya kawaida (soma RDBMS) na zana (Oracle Database kwa mfano). Lakini inaweza kuwa ya kuvutia zaidi kuliko hiyo. "Kama unajua, unaweza kuweka ""kuchunguza"" maneno ya utafutaji kwenye Google kila siku kwa miaka miwili na kisha kuandika ripoti juu ya maneno 25 ya kwanza ya utafutaji yanayohusiana na ""soka.""" Sasa unaweza (kwa haki) kusema, <unk>hii ni wazimu<unk>. "Mamilioni ya rekodi za ""mwaka"" ni sawa na ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa ""mwaka"" wa"" wa ""mwaka"" wa ""m." Na hata kama sisi kuhifadhi, jinsi gani tunaweza kufanya uchambuzi juu ya data hiyo? Je! Karibu katika ulimwengu wa Big Data! "Kama unajua, ""RDBMS"" ni jina la mfumo wa data kubwa, ambayo kwa kawaida hutumiwa na watu ambao wana uzoefu wa kuunda RDBMS na SQL, lakini kwa kweli hawawezi kuifanya iwe rahisi." "Kama unaelewa kwamba RDBMS yako favorite database kamwe kushughulikia data kwamba mengi, lakini wewe hakika kuuliza swali hili: ""Kwa nini juu ya dunia mtu yeyote anahitaji kufanya aina hiyo ya uchambuzi wa ajabu katika nafasi ya kwanza?""" Au unaweza pia kutilia shaka busara ya hii - Vipi kuhusu sababu ya gharama? Kuhifadhi na kuchunguza data nyingi kama hizo kwa gharama ya dola elfu nyingi. Hilo lamaanisha kwamba faida ya uwekezaji itakuwa ya kusikitisha! Na jibu langu litakuwa - "ndiyo, ingekuwa jambo la kusikitisha. Kwa ajili yako. Si kwa ajili ya data kubwa!" Kwa sababu data kubwa haitumii RDBMS, maswali ya SQL au hifadhidata za kawaida. Ni kweli, ikiwa tungejaribu kuondoa data kubwa kwa kutumia teknolojia za kawaida - tungekuwa tumeshindwa. Kwa hiyo, wanasayansi wamepata njia mpya ya kuchanganua data nyingi sana. Nitakuonyesha jinsi ya kufanya hivyo. Lakini acha nijibu swali jingine kabla ya hapo. Swali ni kuhusu uhitaji. Kwa nini tunahitaji data kubwa? Ili kuendesha biashara yako kwa uwazi, lazima uwe na uwezo wa kufikiri, kupanga, kutatua matatizo, kufikiri abstractly, kuelewa mawazo tata, kujifunza haraka na kujifunza kutoka uzoefu. Uwezo huu wote huzunguka ukweli rahisi - uwezo wako wa kufanya uchambuzi unaoendeshwa na data. Kwa sababu ndivyo unavyogundua maarifa. Lakini tatizo ni, unawezaje kupata data ya kutosha? Na kama kutokea kwa kuwa na data ya kutosha, jinsi gani unaweza kufanya uchambuzi juu ya data hiyo? Hizi ni maswali mawili tofauti. Majibu ya maswali haya yatatuongoza kwenye data kubwa. Kwa hiyo hebu tuchunguze. Kwa kawaida, ili kuwa na data ya kutosha, utakuwa na kuanzisha njia ya kutosha kukusanya data yako. Hilo si jambo rahisi sikuzote na kwa hakika si jambo la bei nafuu. Kwa mfano, unaweza kuwa na mkahawa wa chakula cha haraka na kuuza burger ya kuku na fries. Kwa mfano, unaweza kuwa na bidhaa mbili mpya ambazo unataka kuanzisha katika mikahawa yako. Bidhaa hizo ni hamburger ya uyoga na keki ya mananasi. Kama unapewa kazi ya kuchagua moja ya bidhaa hizi juu ya nyingine ili kuongeza faida yako, ungechukua uamuzi gani? Kwa kweli, hakuna ujuzi au uzoefu wa wateja wako kuhusu upendeleo wao kwa mmoja. Ikiwa unafanya maamuzi ya data, njia ya kawaida ni kufanya utafiti. Kwa mfano, unaweza kugawa fomu ya maoni kwa wageni wako wote wa mkahawa ili kuelezea chaguzi zao maalum. Kwa kuzingatia data zilizokusanywa, unaweza kufanya uamuzi. Lakini kuna matatizo matatu katika mbinu hii. Kwanza, mchakato huo ni wa polepole sana. Kwa upande mwingine, mchakato huu unahusisha tu chaguzi za wateja wako wa sasa na hauhesabu wateja wako wa uwezekano. Mwishowe, unahitaji kubuni mchakato wa kompyuta wa kubadilisha matokeo ya data yako ya uchunguzi wa nje ya mtandao kuwa rekodi za hifadhidata kwa uchambuzi. Mfano hapo juu unaonyesha upungufu wa kawaida wa data wakati biashara inataka kukua katika eneo lisilojulikana. Lakini si lazima iwe hivyo sikuzote. Kwa kuwa mitandao ya kijamii kama Facebook na Twitter imesitawi, ni rahisi kupata habari kama hizo. "Kama ilivyoelezwa hapo juu, watumiaji milioni 680 wa Facebook wanakwenda kwenye mikahawa kadhaa kila mwezi, wanapenda vyakula maalum, na kutoa maoni ""Wow!""" "Ninafurahia pie ya mananasi baada ya burger hii"" au ""Ninachukia harufu ya pizza ya uyoga"" sasa na kisha." Chapisho la Facebook linaloonyesha uhusiano wa mtu kuelekea chakula! Hii inaweza kuwa chanzo cha ajabu cha habari kwa ajili ya biashara yako ya mlolongo wa mgahawa. Takwimu hizi tayari zipo. Swali ni, tunatumiaje? Kwa kweli, kuunda RDBMS ya kawaida ya data ya kuhifadhi data hii ni nje ya swali kwa sababu ya muda na usindikaji unaohitajika kubadilisha data hii isiyo na muundo wa maandishi kuwa muundo wa hifadhidata. Hata hivyo, kwa sababu ya hali hii, ni vigumu sana kwa Hercules kuchanganua data kutoka kwa data. Kwa kweli, mchakato wa kawaida unahitaji harakati nyingi za data (soma IO ya kimwili) ambayo inahitaji mchakato bora. Kwa mfano, ikiwa unafikiria juu ya kuunda data, basi unaweza kuamua kuwa data warehouse ya kawaida ya ER haifai hapa. Kama unataka kuhifadhi data hii kubwa katika ER mfano - uchambuzi itakuwa ya kutisha kutokana na idadi kubwa ya kujiunga kwamba unahitaji kuvuka kupata rekodi. Kama unataka kupunguza nje ya sehemu ya upatikanaji wa data kwa kufanya modeling dimensional - data itakuwa unnecessarily kiasi kwa sababu ya data redundancy. Big Data On Rescue - Map Reduce: Kwa hivyo hadi wakati huu, tumeanzisha hitaji la data kubwa kwa ujasusi bora wa biashara na pia tunaelewa maumivu yanayohusiana na data nyingi. "Wengi huita hii ""V3 ya Big Data"" - kiasi, aina na kasi - ambayo kwa mwisho huongeza utata unaohusiana na usindikaji wa data kubwa." Kwa bahati nzuri Big data inajumuisha mbinu na teknolojia ya kutatua tatizo hili. Mojawapo ya mbinu hizo inaitwa - "Kupunguza Ramani". Unaweza kutumia Ramani Kupunguza mfumo wa kuchambua data unstructured kupitia utiririshaji. Streaming ni jambo kubwa - kwa sababu wakati wewe Streaming data, wewe kuepuka haja ya kuhifadhi data na wakati Streaming unaweza kweli kuchambua data juu ya kwenda kupata majibu ya maswali yako. "Map Reduce" ni mada tofauti na dhahiri nje ya wigo wa insha hii, lakini nina nia ya kuandika juu ya hili wakati nina muda. Jambo muhimu la kumbuka ni kwamba data kubwa inatupa njia ya kushinda suala ambalo kwa njia nyingine halingeweza kutatuliwa kwa kutumia mbinu ya kawaida. Jambo zuri ni kwamba mara tu unapoanza kutumia uwezo wa data kubwa, wote wigo na uwezekano wa uchambuzi unaoendeshwa na data huongezeka sana. Kama tulivyoona katika mfano wetu wa mlolongo wa mikahawa, kutumia uchambuzi wa data kubwa kwenye data ya media ya kijamii, sasa unaweza kupanua wigo wa uchambuzi wako kwa upana zaidi na kuifanya kwa muda mdogo. Mifano ya Big Data Analytics Hizi zote si nadharia tu. Kuna kesi mbalimbali za matumizi ya data kubwa sasa hivi katika ulimwengu huu. Amazon.com inashughulikia mamilioni ya shughuli za nyuma kila siku, pamoja na maswali kutoka kwa wauzaji zaidi ya nusu milioni ya watu wa tatu. Kwa kila mtumiaji, wanaweza kufanya uchambuzi wa kikapu cha soko ili kukuonyesha orodha ya bidhaa ambazo unaweza kununua, kulingana na tabia zako za ununuzi wa zamani na tabia za ununuzi wa mamilioni ya wateja kama wewe. Kituo cha NASA cha Simulation ya Hali ya Hewa (NCSC) huhifadhi 32 peta byte ya uchunguzi wa hali ya hewa na simulations kwenye Discover supercomputing cluster - kuwezesha utabiri mbalimbali wa hali ya hewa. Hatua ya kwanza ni kuanzisha kiwanda cha kuunganisha vipimo vya Hadron katika CERN, Geneva, Marekani, ambacho kina vifaa vya kuunganisha vipimo vya Hadron milioni 150 na data milioni 40 kwa sekunde. Takwimu hizi ni kusindika kupata uwepo wa chembe mpya za msingi katika asili, miongoni mwa mambo mengine. Yote haya hayangeweza kufanywa kwa kutumia teknolojia za kawaida za hifadhidata. Hakuna shaka katika akili yangu kwamba Big Data ni dhahiri kitu kikubwa ijayo. Hata hivyo, kama kila kitu kingine, kuna mengi ya hypes kuhusishwa na hilo. Kwa kweli kulingana na mchambuzi wa Gartner, Takwimu Kubwa zinaingia katika "mto wa kukata tamaa" ambayo inaweza kuwazuia kampuni nyingi kutoka kufuatilia teknolojia ya uchambuzi. Lakini kama kila kitu kingine katika ulimwengu huu, kama mtaalamu wa teknolojia tunahitaji kujifunza kutofautisha ukweli kutoka hype. Sasa kwa kuwa unajua nini data kubwa ni, mimi itabidi kumaliza makala hii hapa. Hapa ni infographic muhimu sana muhtasari dhana nyingi za Big Data, kwamba unaweza kutaka kupakua. Katika makala yangu ijayo, tutazungumzia utekelezaji na matumizi ya kesi ya uchambuzi wa data kubwa.