Big data foundations

This lecture sets up how big data applications are built. It covers application traits, data modelling, storage choices and the four analytics types. After this page you can classify any analytics task and trace each type on small numbers.

5. What is Big Data? Types and 5 V’s ExplainedOnTimeNotes · Kinds of data and the 5 Vs

4 Types of Data Analytics Explained with ExamplesFarhat Lectures. The # 1 CPA & Accounting Courses · The four analytics types with examples

Characteristics of big data applications Search on YouTube

A big data application differs from a conventional one by design, not only by size. Five traits separate it:

Massive, distributed data. The data is too large to fit or process in reasonable time on one node. It must be partitioned and computed in parallel.
Continuous, high-velocity ingestion. Data arrives constantly from sensor streams, clickstreams and transactions. It does not arrive as a neat periodic batch.
Heterogeneous variety. Structured tables, semi-structured logs and JSON, plus unstructured text, images and video coexist in the same pipeline.
Feedback-driven and iterative. Outputs feed back in and reshape future data. Recommendations, model scores and alerts change what the system sees next.
Fault tolerance is designed in. At this scale hardware failure is expected, not an edge case. Replication and recovery are core architecture.

Big data modelling Search on YouTube

Traditional databases enforce structure before data is written. Big data systems apply structure when data is read. That shift changes every modelling choice.

\text{Schema-on-write} = \text{structure first, then store} \qquad \text{Schema-on-read} = \text{store first, then structure}

Schema-on-write normalises data and enforces ACID transactions before each write. It optimises for consistency over scale. Schema-on-read stores raw data and applies structure at query time. It models for query patterns: denormalise and duplicate freely.

Schema-on-write checks data at the gate. Schema-on-read stores raw data and structures it per query.

Data model families Search on YouTube

Family	Systems	Idea
Key-value	Redis, DynamoDB	Each key maps to one value; fast lookups, no joins
Document	MongoDB	JSON-like documents; flexible fields per record
Columnar / wide-column	Cassandra, HBase	Rows keyed by a row key; columns grouped into column families; very high write throughput
Graph	Neo4j, Spark GraphX	Vertices plus edges; built for relationships

In this course, Hive partitioning and bucketing choices are modelling decisions. They lay out data by how it is queried, not by normalisation purity.

Name the trade when the exam asks. Schema-on-write gives consistency and clean reads at write cost. Schema-on-read gives scale and flexibility at query cost.

Big data storage Search on YouTube

Where data lives, and in what shape, matters as much as the compute engine that reads it. Three homes cover most systems:

Distributed file systems. HDFS splits data into blocks and replicates them across nodes. It answers the fact that no single disk is large enough.
Object storage. S3, Azure Blob and GCS offer near-infinite cheap storage, decoupled from compute. Object storage is the backbone of the modern lakehouse pattern.
NoSQL stores. Cassandra and HBase serve specific access patterns at massive write throughput. Each store fits one pattern well and others poorly.

Format matters as much as location Search on YouTube

Row-based formats favour fast writes. Columnar formats (Parquet, ORC) favour fast analytical reads and much better compression. The Hive module runs this comparison live.

The bigger trend is the lakehouse model: separate storage from compute instead of Hadoop's original co-located design. Each side then scales on its own.

Storage holds blocks, objects and NoSQL data. Compute scales apart from it. Columnar files speed up analytics.

Types of analytics Search on YouTube

Each analytics stage answers a harder question than the last. Each stage needs more data and more computation than the one before it.

Four stages in fixed order. Each stage builds on the output of the earlier ones.

Autonomous data agents increasingly chain all four stages automatically. They investigate, diagnose, forecast and recommend. A human reviews only the final call.

Describe, Dig, Predict, Prescribe

Descriptive (what happened) · Diagnostic (dig for why) · Predictive (what will happen) · Prescriptive (what to do). The order never changes.

Descriptive analytics Search on YouTube

Descriptive analytics (business intelligence and data mining) answers "what happened". It is the base for every advanced type. It turns historical or current data into trends, frequency of events, operation costs, and underlying reason of failure. It helps managers, planners and leaders build a road map for future action.

Daily examples are streaming dashboards and retail reports: a "what did users watch this week" report, or a daily sales-by-region summary. In this course, Hive and Pig deliver it. A query such as SELECT zone, COUNT(*), AVG(fare) FROM trips GROUP BY zone answers "what happened last month". That is descriptive analytics.

Worked example. Eight taxi trips form the table below. Group them by zone. Count the trips and average the fare per zone.

trip	zone	fare
1	A	10
2	A	12
3	A	14
4	B	20
5	B	22
6	C	8
7	C	9
8	C	7

SELECT zone, COUNT(*), AVG(fare) FROM trips GROUP BY zone;

Zone A: three trips.

\begin{aligned} \text{count} &= 3\\ \text{mean} &= (10 + 12 + 14)/3 = 36/3 = \mathbf{12.0} \end{aligned}

Zone B: two trips.

\begin{aligned} \text{count} &= 2\\ \text{mean} &= (20 + 22)/2 = 42/2 = \mathbf{21.0} \end{aligned}

Zone C: three trips.

\begin{aligned} \text{count} &= 3\\ \text{mean} &= (8 + 9 + 7)/3 = 24/3 = \mathbf{8.0} \end{aligned}

zone	COUNT	AVG(fare)
A	3	12.0
B	2	21.0
C	3	8.0

The output is a factual summary of the past. No cause and no forecast appear yet.

Diagnostic analytics Search on YouTube

Diagnostic analytics answers "why did it happen". It goes one level deeper than description for the company's customers, employees, products and more. It drills down, correlates and detects anomalies against a baseline to find the causal relationship. Treat it as detective work, not as a definition.

The slide example: an e-commerce platform sees cart abandonment spike 40% overnight. Diagnostic analysis correlates the spike against checkout load times, payment errors, or one device segment to isolate the cause.

In this course, diagnostic work means multi-table joins and correlation in Hive and Pig, plus Spark DataFrame operations. On the taxi data the question is: "fares in Zone X dropped 20% last month, why". The answer joins trips against weather or event-calendar data.

Worked example. A shop holds carts steady at $5{,}000$ per night. Baseline abandonment is $20\%$ . Tonight it is $28\%$ . Check the spike, then split by segment.

Confirm the 40% spike. Abandons rise from $1{,}000$ to $1{,}400$ .

\begin{aligned} \text{baseline rate} &= 1000/5000 = 0.20\\ \text{tonight rate} &= 1400/5000 = 0.28\\ \text{relative rise} &= (1400-1000)/1000 = 400/1000 = \mathbf{0.40} \end{aligned}

The relative rise is 40%. The absolute rise is only 8 points.

Drill down by segment. Desktop holds $3{,}000$ carts with $480$ abandons on both nights ( $16\%$ ). Mobile holds $2{,}000$ carts.

\begin{aligned} \text{mobile night 1} &= 520/2000 = 0.26\\ \text{mobile night 2} &= 920/2000 = 0.46\\ 480 + 920 &= \mathbf{1{,}400}\text{, which matches tonight} \end{aligned}

Isolate the cause. Desktop is flat, so the cause sits in the mobile segment. The next joins test mobile load times and payment errors for tonight.

Predictive analytics Search on YouTube

Predictive analytics answers "what will happen". It uses historical data, statistics and machine learning to forecast future outcomes. It predicts probabilities and trends with data mining and forecast techniques.

Hospital example: models flag high readmission-risk patients for proactive care. Logistics example: the UPS ORION system blends predictive and prescriptive routing and removes over 100 million miles from its fleet each year.

In this course, Spark MLlib runs the pipeline: feature engineering, then train/test split, then regression or classification, then evaluation. The taxi use is a fare or trip-duration forecast.

Worked example. A hospital scores $200$ discharged patients. It flags $50$ as high risk. Within 30 days, $30$ of the flagged patients return, and $10$ of the unflagged patients return. Score the forecast.

Precision: how many flags were right.

\begin{aligned} \text{precision} &= 30/50 = \mathbf{0.60} \end{aligned}

Recall: how many returns were caught. Total returns are $30 + 10 = 40$ .

\begin{aligned} \text{recall} &= 30/40 = \mathbf{0.75} \end{aligned}

The model catches 75% of returns, and 60% of its flags are right. Proactive care goes to the 50 flagged patients.

Prescriptive analytics Search on YouTube

Prescriptive analytics answers "what should we do". It goes one step past prediction. It recommends the best action under real constraints with optimisation, simulation and generative AI. Retailers use it to set the exact markdown price and timing for slow stock, and the per-store restock volume that avoids stockouts.

In this course, GraphX and Pregel serve this stage. Shortest-path and optimisation-style graph algorithms answer "what is the best route or allocation given this network". The slide example: find the cheapest way to ship goods from a factory to its destination.

Worked example. A factory must ship $100$ units to one city. Rail costs $5$ per unit for at most $70$ units. Road costs $8$ per unit with no limit. Find the cheapest mix.

Fill cheap rail first, then road.

\begin{aligned} \text{rail} &= 70 \times 5 = 350\\ \text{road} &= 30 \times 8 = 240\\ \text{total} &= 350 + 240 = \mathbf{590} \end{aligned}

Compare with all-road.

\begin{aligned} \text{all-road} &= 100 \times 8 = 800\\ \text{saving} &= 800 - 590 = \mathbf{210} \end{aligned}

The prescription is 70 units by rail and 30 by road. That mix minimises cost under the capacity limit.

Analytics types compared Search on YouTube

Type	Question	Example	Tool in this course
Descriptive	What happened	Streaming and retail dashboards	Hive, Pig
Diagnostic	Why did it happen	Cart-abandonment root cause	Hive and Pig joins, Spark DataFrames
Predictive	What will happen	Readmission risk, UPS ORION forecast	Spark MLlib
Prescriptive	What should we do	UPS ORION routing, digital-twin simulation	GraphX / Pregel

Each row reuses the slide examples. The tool column names the engine that runs each stage in this course.

Do not list predictive tools for a diagnostic question. Diagnostic work joins and correlates past data. Predictive work trains a model and forecasts. Examiners test that split.

Quick self-test Search on YouTube

1. Name the five traits of a big data application.

Massive distributed data, continuous high-velocity ingestion, heterogeneous variety, feedback-driven iteration and designed-in fault tolerance. Size alone is not the test.

2. Differentiate schema-on-write and schema-on-read in one line each.

Schema-on-write enforces structure before storage with normalised ACID writes. Schema-on-read stores raw data and applies structure at query time for each query pattern.

3. Name the four data model families with one system each.

Key-value (Redis or DynamoDB), document (MongoDB), columnar or wide-column (Cassandra or HBase) and graph (Neo4j or Spark GraphX).

4. Name the three storage homes and state when each fits.

HDFS stores blocks with replicas across nodes. Object storage holds cheap decoupled lakehouse data. NoSQL stores serve one access pattern at massive write throughput.

5. Trips in Zone D have fares 9, 12 and 15. Find the count and the mean.

Count $= \mathbf{3}$ . Mean $= (9+12+15)/3 = 36/3 = \mathbf{12.0}$ .

6. Abandonment moves from 20% to 28% on fixed carts. What is the relative rise?

$(28-20)/20 = 8/20 = \mathbf{0.40}$ , a 40% relative rise. The absolute rise is 8 points.

7. A model flags 60 patients and 45 return. Total returns are 60. Find precision and recall.

Precision $= 45/60 = \mathbf{0.75}$ . Recall $= 45/60 = \mathbf{0.75}$ .

8. Ship 60 units. Rail costs 4 per unit for at most 40 units. Road costs 7 per unit. Find the cheapest total and the saving over all-road.

Cheapest $= 40 \times 4 + 20 \times 7 = 160 + 140 = \mathbf{300}$ . All-road $= 420$ . Saving $= \mathbf{120}$ .

9. Differentiate diagnostic and predictive analytics with the taxi example.

Diagnostic asks why fares in Zone X dropped 20% and joins trips against weather data. Predictive forecasts next month fare or trip duration with a trained model.

10. Which analytics type sets restock volumes, and what does it build on?

Prescriptive analytics sets them with optimisation and simulation. It builds on descriptive and predictive results under real constraints.

A big data application has massive distributed data, high-velocity ingestion, heterogeneous variety, feedback loops and designed-in fault tolerance.
Schema-on-write structures data before storage for consistency. Schema-on-read stores raw data and structures it per query for scale.
The four model families are key-value, document, columnar (wide-column) and graph.
The three storage homes are HDFS blocks, object storage for the lakehouse and NoSQL stores for fixed access patterns.
Columnar formats (Parquet, ORC) favour analytical reads and compression. The lakehouse separates storage from compute.
Descriptive analytics answers what happened (Hive, Pig). Diagnostic answers why (joins, correlation, drill-down).
Predictive analytics answers what will happen (Spark MLlib: features, train/test split, model, evaluation).
Prescriptive analytics answers what to do (optimisation, GraphX shortest paths, e.g. cheapest shipment at 590).
The order is fixed: describe, dig, predict, prescribe. Data agents chain all four with a human at the end.

Big data foundations

Characteristics of big data applicationsSearch on YouTubeBookmarkSaved

Big data modellingSearch on YouTubeBookmarkSaved

Data model familiesSearch on YouTubeBookmarkSaved

Big data storageSearch on YouTubeBookmarkSaved

Format matters as much as locationSearch on YouTubeBookmarkSaved

Types of analyticsSearch on YouTubeBookmarkSaved

Descriptive analyticsSearch on YouTubeBookmarkSaved

Diagnostic analyticsSearch on YouTubeBookmarkSaved

Predictive analyticsSearch on YouTubeBookmarkSaved

Prescriptive analyticsSearch on YouTubeBookmarkSaved

Analytics types comparedSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page