Introduction to big data

Course scopeSearch on YouTubeBookmarkSaved

This course covers Hadoop with HDFS, MapReduce and YARN. It then covers Pig, Hive and Spark with MLlib, GraphX and streaming. Later lectures give the full treatment of every tool. This page only sets up terms and scale.

What big data isSearch on YouTubeBookmarkSaved

Big data is a massive amount of data. Traditional tools cannot store, process and analyse it. That limit is the whole point of the course.

Big data analytics is the process that extracts useful insights from big data. Typical insights are hidden patterns, unknown correlations, market trends and customer preferences.

How big is big dataSearch on YouTubeBookmarkSaved

The slides list facts that show the scale. Every minute:

• Twitter users send 285,000 tweets.
• Facebook processes almost 350 GB of data.
• People and organisations launch 576 new websites.
• Google gets more than 2 million search queries.
• More than 100 million new emails are sent.
• Users upload 72 hours of new video to YouTube.
• Walmart processes almost 15,000 transactions.
• Sprint processes more than 300,000 phone calls.

A second slide adds more rates:

• About 2.9 million emails go out every second.
• About 50 million tweets go out per day.
• Households consume about 375 MB of data per day.
• Facebook users spend about 700 billion minutes per month.
• Users upload about 20 hours of video to YouTube every minute.
• Mobile internet users move about 1.3 EB of data.
• Google processes about 24 PB of data per day.
• Amazon takes about 72.9 items ordered per second.

Data sizesSearch on YouTubeBookmarkSaved

Data units grow in powers of 10. Each step up multiplies by $1{,}000$:

Brontobyte is an unofficial name. The numbers in parentheses on the slide count powers of 10 above KB: MB is 3, GB is 6, and so on.

Sources of big dataSearch on YouTubeBookmarkSaved

Big data comes from three sides: people, machines and organisations.

• People carry data-generating devices. Mobile phones, Facebook, Twitter, Instagram, GPS and cameras all emit data.
• The internet adds emails, websites, blogs, Google searches, uploaded photos and tweets.
• Machines and organisations add computing logs, sensing data, real-time traffic information, map data and IoT device streams. IoT devices alone generate huge amounts of data.

Big data vs traditional dataSearch on YouTubeBookmarkSaved

Traditional data lives in one central database. Big data lives in a distributed database across many nodes. The table below compares them feature by feature.

More data gives more accurate results. But check veracity before use. See Veracity.

Kinds of dataSearch on YouTubeBookmarkSaved

Data comes in three kinds. The kind decides which tool can process it.

Structured dataSearch on YouTubeBookmarkSaved

Structured data has a defined repeating pattern. It sits in a tabular format with a fixed schema. It is highly organised, so a simple search engine algorithm can store and fetch it.

Examples are the employee table of a company, a student table and a bank customer table. Sources are relational databases, CSV files and multi-dimensional files used in data warehouses.

Unstructured dataSearch on YouTubeBookmarkSaved

Unstructured data lacks any specific form or structure. It is heterogeneous: text files, images and videos mixed together. This makes it very hard and slow to process and analyse.

Unstructured data needs processing before it yields anything of interest. Feature extraction turns it into structured data. The catch is the near-infinite number of potential features.

Semi-structured dataSearch on YouTubeBookmarkSaved

Semi-structured data mixes both formats above. It does not live under a fixed repository such as a database. It still carries vital tags that separate its elements.

Typical examples are log files, JSON files, sensor data and CSV files. The tags give partial structure without a fixed schema.

The Vs of big dataSearch on YouTubeBookmarkSaved

The scale of the problem has four symptoms. Tweets create 12+ TB daily. Trade events pass 5+ million per second. Data comes in 100s of types. Only 1 in 3 decision makers trusts their information. Those four symptoms are volume, velocity, variety and veracity.

The slide deck grows the list step by step. Gartner named the 3 Vs in 2005. Later slides add a 4th, 5th, 6th and 7th V. The last slide lists 10 Vs. IDC projected that data generation would grow tenfold worldwide by 2020.

VolumeSearch on YouTubeBookmarkSaved

Volume describes the size and quantity of data. Three figures from the slides:

• The world made 90% of its data in the last two years (IBM study).
• World data grows from 33 ZB in 2018 to 175 ZB by 2025 (IDC).
• World digital data doubles every two years, faster than Moore's prediction (Intel report).

The 33 to 175 ZB jump is a factor of $175 / 33 \approx 5.3$ in 7 years.

VelocitySearch on YouTubeBookmarkSaved

Velocity is the rate of data flow: generation, then storage, then processing and analysis. Sources are mobile phones, social media, networks and servers.

Speed shapes value. A machine learning model may consume a stream of data. A social platform posts and uploads photos around the clock. Fast access drives timely and correct business decisions. A small amount of real-time data beats a large volume that arrives late.

The slide gives health care as the example. A pacemaker sends a patient's heart-rate data. Rapid analysis of that stream can prevent serious injury or death.

VarietySearch on YouTubeBookmarkSaved

Variety is the number of kinds of data. It reuses the three kinds above:

• Structured: organised data with a formal structure, stored in a relational database. Example: a bank statement with date, time and amount.
• Semi-structured: semi-organised data without a formal structure. Example: log files, JSON files, sensor data and CSV files.
• Unstructured: data with no rows-and-columns shape. Example: text files, emails, images, videos, voicemails and audio files.

About 80% of health-care data is unstructured. It holds handwritten practitioner notes, imaging data, video and audio files.

VeracitySearch on YouTubeBookmarkSaved

Veracity is the 4th V. It describes the accuracy and quality of data. Data comes from diverse sources, so it holds uncertainties, errors, redundancies, gaps and inconsistencies.

Veracity means assurance of quality, integrity, credibility and accuracy. Check the data for accuracy before using it for business insights.

ValueSearch on YouTubeBookmarkSaved

Value is the 5th V. A bulk of data with no value is no good to a company. Data must turn into something useful.

What counts is the amount of valuable, reliable and trustworthy data. Such data is stored, processed and analysed to find insights. Fast processing of varied high-quality data is great. But the most important thing is the value it brings to the business. Firms that collect data and never use it miss big opportunities.

VariabilitySearch on YouTubeBookmarkSaved

Variability is the 6th V. It asks how fast the structure of the data changes. It asks how often the meaning or shape of the data changes.

The slide gives a food example. Eating the same ice cream daily means little if the taste keeps changing. The name stays while the content shifts.

VisualizationSearch on YouTubeBookmarkSaved

Visualization is the 7th V. It is critical in today's world. Charts and graphs convey large complex data far better than spreadsheets full of numbers and formulas.

The slides show two examples. One is an Excel sheet of yearly sales per salesperson with a line chart. The other is pie charts of units sold per store and per flower: orchid, rose, sunflower and daisy across six stores.

Validity, venue, vocabulary and vaguenessSearch on YouTubeBookmarkSaved

The 10 Vs slide keeps volume, velocity, variety, veracity, value and variability. It adds four Vs:

• Validity: data quality, governance and master data management on massive data.
• Venue: distributed heterogeneous data from multiple platforms.
• Vocabulary: data models and semantics that describe data structure.
• Vagueness: confusion over the meaning of big data and the tools used.

Social media analytics toolsSearch on YouTubeBookmarkSaved

Social sites emit huge data and also sell tools to read it. The slides name three tools.

Meta Business SuiteSearch on YouTubeBookmarkSaved

Meta Business Suite manages ads, commerce and content publishing for Facebook and Instagram. Its key features:

• Insights: review post performances.
• Trends: track key trends to see patterns for success.
• Audience information: check demographics of the target audience.

KeyholeSearch on YouTubeBookmarkSaved

Keyhole tracks hashtags and keywords. It follows trends as they happen. Its key features:

• Sentiment analysis: see who talks about a brand and how they feel about it.
• Recommendations: get data-backed tips on time to post for followers, engagement or conversion.
• Competitor analysis: compare performance side by side with rivals.

Twitter AnalyticsSearch on YouTubeBookmarkSaved

Twitter Analytics is a free built-in tool for all users. It gives basic insights into the best-performing tweets. Its features:

• Monthly highlights: top tweet, mention and follower plus other monthly updates.
• Deep dives: engagement counts per tweet, split into likes, profile clicks, detail expands, retweets and replies.
• Averages: monthly engagement rates, reply counts, retweet counts and more.

Applications of big dataSearch on YouTubeBookmarkSaved

The overview slide names health care, finance, e-commerce, media and entertainment. The detail slides cover the five sections below.

Finance and bankingSearch on YouTubeBookmarkSaved

Banks use big data to generate more insights and revenue. Analytics rests on investment patterns, shopping trends, motivation to invest and personal or financial background inputs.

Uses are spotting fraud, analysing and streamlining transaction processing, improving understanding of users, perfecting trade execution and promoting an exceptional user experience.

Health careSearch on YouTubeBookmarkSaved

Big data cuts the cost of treatment. It predicts epidemic outbreaks. It prevents diseases. It improves quality of life.

Patient data helps doctors give better treatment. Data from IoT devices can predict disease and prevent it. A sensor near a patient or a newborn baby tracks heart-beat rate and blood pressure. When a value crosses the safe limit, an alarm reaches a doctor, who acts remotely at once.

E-commerceSearch on YouTubeBookmarkSaved

Shops track customer spending habits and shopping behaviour. Big retailers such as Amazon and Flipkart recommend products to each customer.

They record what a customer searches and recommend that type of product. Ads for the right product reach the right customer. YouTube recommends videos from past watched types. Sale days spike traffic and can crash a site. Big data helps organise and analyse the data for further use.

Government and militarySearch on YouTubeBookmarkSaved

The government and military use technology at high rates. A fighter plane must process petabytes of data. Agencies run utilities, manage traffic jams and fight crime such as hacking and online fraud.

Aadhar holds records of 1.21 billion citizens. That vast data needs analysis and storage. A traditional database cannot hold it. Big data analytics tools store and analyse it.

Social mediaSearch on YouTubeBookmarkSaved

Social media is the largest data generator. About 500+ TB of fresh data comes from social media daily, mainly on Facebook. The data holds videos, photos and message exchanges.

One activity on a social site generates much stored data. That data is processed when required. Storage in terabytes takes a lot of processing time. Big data is the solution to the problem.

ScalabilitySearch on YouTubeBookmarkSaved

One machine cannot hold and process growing data. That is the scalability issue. The slide asks two fill-in-the-blank words: scale up and scale out.

Scalability is the capability of a system to handle a growing amount of work. It is the potential to enlarge the system to fit that growth. Two methods exist:

Scale upSearch on YouTubeBookmarkSaved

Scale up is also called vertical scaling. It means upgrading hardware: buying and installing a more capable machine.

Scale outSearch on YouTubeBookmarkSaved

Scale out is also called horizontal scaling. It adds many lower-performance nodes to the current system.

Types of analyticsSearch on YouTubeBookmarkSaved

The deck lists four types of big data analytics. This is the short list. l02 gives the full treatment.

• Descriptive: business intelligence and data mining. It answers what happened in the business.
• Predictive: forecasting. It answers what could happen, using statistical models and forecast techniques.
• Prescriptive: optimisation and simulation. It answers what should be done, building on descriptive and predictive results. Example: the cheapest way to ship goods from a factory to its destination.
• Diagnostic: investigation of data and content. It answers why it happened by finding causes behind events.

Quick self-testSearch on YouTubeBookmarkSaved

1. Define big data in one sentence.

Big data is a massive amount of data that traditional tools cannot store, process and analyse. The tool limit is part of the definition.

2. Convert 2.5 GB to MB and to bytes.

$2.5 \times 1{,}000 = \mathbf{2{,}500}$ MB. $2.5 \times 10^9 = \mathbf{2{,}500{,}000{,}000}$ bytes. Each step down multiplies by $1{,}000$.

3. About 2.9 million emails go out per second. How many go out per day?

$2{,}900{,}000 \times 86{,}400 = 250{,}560{,}000{,}000$. That is about $\mathbf{2.51\times 10^{11}}$ emails per day.

4. The store table gives orchid sales of 3500, 3350, 6000, 6500, 7800 and 6800. Find the total and the share of Store 5.

Total $= \mathbf{33{,}950}$. Store 5 share $= 7800/33950 \approx 0.230 = \mathbf{23\%}$.

5. World data grows from 33 ZB in 2018 to 175 ZB in 2025. What is the growth factor?

$175/33 \approx \mathbf{5.3}$. Data grows more than fivefold in 7 years.

6. Differentiate structured, semi-structured and unstructured data with one example of each.

Structured data has a fixed tabular schema, such as an employee table. Semi-structured data carries tags without a fixed schema, such as a JSON log file. Unstructured data has no fixed form, such as video files.

7. Name the 7 Vs and state what each one asks.

Volume (how much data), variety (how many kinds), velocity (how fast it flows) and veracity (how accurate it is). Value (how useful it is), variability (how fast its shape changes) and visualization (how charts show it).

8. Differentiate scale up and scale out. Give one disadvantage of each.

Scale up (vertical) buys stronger hardware for one machine; its limit is the scalability ceiling with no fault tolerance. Scale out (horizontal) adds many small machines; its cost is harder maintenance with more space, power and cooling needs.

9. Why is veracity a problem, and what must be done about it?

Data comes from diverse sources, so it holds errors, redundancies, gaps and inconsistencies. Check the data for accuracy before using it for business insights.

10. Which analytics type answers "what should be done", and where is the full treatment?

Prescriptive analytics answers it with optimisation and simulation. The full treatment of all four types is in l02.