Formula cheatsheet
Every key formula and rule from the course, one table per lecture. Click a heading to open the full lecture.
Introduction to big dataSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Unit ladder | KB MB GB TB PB EB ZB YB | Each step is ; KB up to YB |
| Bytes from a unit | counts steps above bytes (MB uses , GB uses ) | |
| Scale up (vertical) | stronger hardware on one node | Upgrades one machine; hits a limit with no fault tolerance |
| Scale out (horizontal) | more nodes in the cluster | Adds many small machines; gives true scalability with fault tolerance |
Big data foundationsSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Schema-on-write vs schema-on-read | structure first, then store vs store first, then structure | Write enforces ACID schema; read applies schema per query |
| Data model families | key-value, document, columnar, graph | Redis, MongoDB, Cassandra/HBase, Neo4j/GraphX |
| Storage homes | HDFS blocks, object store, NoSQL | Replicas across nodes; lakehouse backbone; fixed access patterns |
| Columnar formats | Parquet, ORC over row formats | Fast analytical reads with better compression |
| Descriptive | COUNT, AVG, GROUP BY | What happened; Hive and Pig summaries |
| Diagnostic | JOIN + correlate + drill down | Why it happened; split the spike by segment |
| Predictive | features, train/test split, model, evaluation | What will happen; Spark MLlib forecasts |
| Prescriptive | minimise cost under constraints | What to do; cheapest mix fills cheap capacity first |
Hadoop and HDFSSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Block count | blocks: 128, 128, 128, 128, 30 MB | |
| Raw storage | MB stored | |
| NameNode entries | 5 blocks × 3 replicas = 15 locations | One location per replica |
| Replica placement | local node, different rack, same rack as second | 3 copies over at least 2 racks |
| Daemon timers | heartbeat 3 s, block report 10 s, checkpoint 1 h | NameNode detects dead nodes by missing heartbeat |
| Read path | locations from NameNode, data from DataNodes | Bulk data never passes through the NameNode |
| Write path | `create(), pipeline, acks, complete()` | Packets flow down the chain, acks return up it |
| Config | dfs.replication 3, blocksize 268435456 = 256 MB | Web UI port is 50070 |
MapReduce and YARNSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Phase order | split, map, shuffle and sort, reduce | S-M-S-R; shuffle groups by key, reduce sums each group |
| Map task count | map tasks = input splits | 3 splits give 3 map tasks; a split is the work unit, not a block |
| Partition rule | One key lands whole in one partition; partitions reducers | |
| Combiner | pre-sum on the mapper node | Cuts network pairs (9 to 8 in the Deer example); final sums stay equal |
| Sentence split trace | (data, 3), (in, 2), rest 1 | data occurs once plus twice; in occurs twice |
| Deer trace | Bear 2, Car 3, Deer 2, River 2 | Sums of 9 map pairs: |
| Hadoop v1 | JobTracker plans, TaskTracker runs | One JobTracker per cluster; TaskTracker heartbeats the master |
| YARN daemons | Scheduler, Applications Manager, NodeManager, ApplicationMaster, container | Scheduler allocates; Manager accepts jobs; NodeManager launches containers |
| Execution order | submit, container, register, negotiate, launch, run, report, release | 8 steps from client submit to ApplicationMaster shutdown |
Pig data model and operatorsSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Nesting order | field → tuple → bag → relation | Field is one value, tuple is a row, bag holds tuples, relation is a named bag |
| Map | [key#value, ...] | Keys are unique chararray, values hold any type |
LOAD | A = LOAD 'f' USING PigStorage(',') AS (n:chararray, a:int); | Read a file into a relation with schema |
STORE | STORE A INTO 'out'; | Write a relation, fails if the output dir exists |
CROSS | Each tuple of A pairs with each tuple of B | |
DISTINCT | X = DISTINCT A; | Keep one copy of each full tuple |
ORDER BY | X = ORDER A BY a3 DESC; | Sort on fields, direction is ASC or DESC |
LIMIT | X = LIMIT A 3; | Keep at most 3 tuples, order not promised |
FILTER | X = FILTER A BY f3 == 3; | Keep tuples that pass the test |
FOREACH | X = FOREACH A GENERATE f1; | Project columns, like SELECT |
GROUP | B = GROUP A BY age; | One (group key, inner bag) tuple per key |
COUNT | FOREACH B GENERATE COUNT(A); | Count tuples per bag, skips NULL |
MAX | FOREACH B GENERATE group, MAX(A.a3); | Largest value per bag, skips NULL |
TOKENIZE | FOREACH A GENERATE TOKENIZE(a1); | Split each line into a bag of words |
| Word count | FLATTEN(TOKENIZE(line)) then GROUP then COUNT | One row per word, group equal words, count bags |
UNION | student = UNION student1, student2; | Stack all tuples, schemas need not match |
SPLIT | SPLIT d INTO s1 IF age < 23, s2 IF age > 25; | Route tuples by IF tests, drop the rest |
Pig statements, types and joinsSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Statement shape | LOAD -> transforms -> DUMP or STORE; | Every statement ends with ;; output action triggers the run |
| Keywords vs names | LOAD = load; students != Students | Keywords ignore case; relation, field and function names keep case |
REGISTER | REGISTER ./tutorial.jar; | Loads a JAR of UDFs for the script |
| NULL test | FILTER s BY city is not null; | is null / is not null only; == and != miss NULLs |
| Arithmetic | for , | + - * / %; , , , |
| Bincond | (a == 1) ? 20 : 30 | True branch before :, false branch after it |
CASE parity | CASE f2 % 2 WHEN 0 THEN 'even' WHEN 1 THEN 'odd' END | Cleaner than nested binconds for many branches |
| Regex test | f1 matches '.*tutorial.*' | True when f1 holds tutorial anywhere |
| Constructors | (Raju, 30), {(Raju,30),(Md,45)}, [name#Raja, age#30] | ( ) tuple, { } bag, [ ] map |
Inner JOIN | JOIN students BY id, fees BY id; | Keeps only keys in every relation; key 2 drops out |
LEFT OUTER | JOIN students BY id LEFT OUTER, fees BY id; | Keeps every left row; missing fees fields are NULL |
RIGHT OUTER | JOIN students BY id RIGHT OUTER, fees BY id; | Keeps every right row; missing students fields are NULL |
FULL OUTER | JOIN students BY id FULL OUTER, fees BY id; | Keeps every row of both sides |
COGROUP | COGROUP students BY id, fees BY id; | Returns (key, bag1, bag2); unmatched side is {} |
STREAM | STREAM raw_logs THROUGH `normalize.py`; | Pipes tuples through an external program |
DESCRIBE | DESCRIBE enriched; | Shows schema; Grunt shortcut \de |
EXPLAIN | EXPLAIN revenue; | Shows logical, physical, MapReduce plans; shortcut \e |
ILLUSTRATE | ILLUSTRATE enriched; | Traces a sample through the pipeline; shortcut \i |
| Revenue pattern | FILTER > GROUP > SUM > ORDER DESC > LIMIT | Hyderabad beats Delhi |
| Inactive pattern | JOIN customers BY id LEFT OUTER, valid BY id; | Keeps customers 2 and 4 with NULL order fields |
Pig UDFs and scriptingSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| UDF kinds | Eval / Filter / Aggregate / Load-Store | Eval per tuple, Filter boolean, Aggregate whole bag, Load/Store custom format |
| Java UDF base | class UPPER extends EvalFunc<String> | Override exec(Tuple input); return value is the new field |
| REGISTER | REGISTER myudfs.jar; | Makes the jar visible to the script |
| DEFINE alias | DEFINE UPPER myudfs.UPPER(); | Short name for the full class name |
| Call a UDF | FOREACH students GENERATE UPPER(name); | Runs exec() once per tuple |
| Piggybank | DEFINE ISOToUnix org.apache.pig.piggybank.evaluation.datetime.convert.ISOToUnix(); | Ready string, math and date UDFs |
| Scripting UDF | REGISTER 'myfuncs.py' USING jython AS myfuncs; | Python via Jython; also Rhino, JRuby, Groovy |
| MAPREDUCE operator | `com.example.LegacyScoringJob mr-input mr-output` | Backtick launch command inside MAPREDUCE 'job.jar' STORE ... LOAD ... |
| MAPREDUCE I/O | STORE cleaned_orders INTO 'mr-input' / LOAD 'mr-output' AS (customer_id:int, score:double) | Pig writes the input dir, reads the output dir with a declared schema |
| Local prototype | pig -x local script.pig | Fast sample runs without cluster queues |
| DESCRIBE | DESCRIBE emp; | Shows schema; runs no job |
| DUMP (safe) | DUMP (LIMIT emp 5); | Prints rows; every DUMP runs a mini job, so limit first |
| STORE clash | hadoop fs -rm -r dept_summary_output | Pig never overwrites output dirs; delete or use a fresh path |
| Generate pattern | f-string builds .pig text | Parametrizes thresholds and paths |
| Subprocess pattern | subprocess.run(['pig', '-x', 'local', 'f.pig']) | Check returncode; 0 means success |
| Embed pattern | P = Pig.compile(t) / P.bind({...}) / bound.runSingle() | Adds loops and conditions; run with pig, not python |
| PigUnit | test.assertOutput("students", input, "upper_names", output); | In-memory test with no cluster |
| Lab summary | Engineering 4, 59000.0, 71000 / Sales 3, 50666.67, 53000 / Marketing 3, 42000.0, 45000 | GROUP emp BY dept then COUNT, AVG, MAX |
Hive architecture and HiveQLSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Query path | Client → Driver → Metastore → Engine → HDFS | Driver checks Metastore before compiling the plan |
| Services | HiveServer2 / CLI / Web UI | Multi-user JDBC/ODBC/Thrift entry points |
| Metastore | schemas + partitions + HDFS paths | Catalog metadata; never holds the data |
| ARRAY access | tags[0] | Zero-indexed item lookup |
| MAP access | attributes['color'] | Key lookup in a map column |
| STRUCT access | dimensions.length | Dot notation into a struct field |
| Format pick | STORED AS ORC | Hive-only analytics and ACID tables |
| Cross-engine | STORED AS PARQUET | Shared reads by Spark, Trino, Presto |
| Evolving schema | STORED AS AVRO | JSON schema travels with row data |
| Raw landing | STORED AS TEXTFILE | Staging only; convert before querying |
| DATABASE synonym | CREATE SCHEMA = CREATE DATABASE | Full synonyms everywhere |
| DROP DATABASE | DROP DATABASE x CASCADE; | CASCADE drops every table inside |
| Managed DROP | DROP TABLE managed; | Deletes schema and files |
| External DROP | DROP TABLE ext; | Deletes metadata; files stay at LOCATION |
| TRUNCATE | TRUNCATE TABLE t; | Clears rows, keeps schema; managed tables |
| Partition | PARTITIONED BY (order_date STRING) | One HDFS dir per value; prune on filter |
| Bucket | CLUSTERED BY (customer_id) INTO 4 BUCKETS | bucket = hash(col) mod n; INT hash is the value |
| LOAD | LOAD DATA LOCAL INPATH 'f.csv' INTO TABLE t; | Copies files as-is; runs no job |
| Append vs replace | INSERT INTO / INSERT OVERWRITE | Append rows vs replace table or Hive partition |
| ACID recipe | STORED AS ORC + ('transactional'='true') | Required for UPDATE / DELETE |
| UPDATE / DELETE | UPDATE t SET b = b - 100 WHERE id = 42; | ACID tables; else error 10294 |
| Query shape | WHERE → GROUP BY → HAVING → ORDER BY → LIMIT | HAVING filters aggregates |
| ORDER BY | ORDER BY total DESC | One global order; slow on huge results |
| SORT BY | SORT BY amount; | Sorted within each reducer |
| DISTRIBUTE BY | DISTRIBUTE BY customer_id; | Same reducer, unsorted |
| CLUSTER BY | CLUSTER BY customer_id; | DISTRIBUTE BY x SORT BY x |
| Plain view | CREATE VIEW v AS SELECT ... WHERE amount > 400; | Reruns each read; always fresh; read-only |
| Materialized view | CREATE MATERIALIZED VIEW v AS SELECT ...; | Stored result; optimizer can auto-rewrite |
| REBUILD | ALTER MATERIALIZED VIEW v REBUILD; | Refreshes stale stored results |
| Index removal | HIVE-18448 (Hive 3.0) | No CREATE INDEX; use MVs, ORC stats, pruning |
| Statistics | ANALYZE TABLE t COMPUTE STATISTICS; | Feeds the cost-based optimizer |
Partitioning, bucketing and joinsSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Managed create | CREATE TABLE t (...) STORED AS ORC; | No LOCATION; Hive owns the warehouse path |
| External create | CREATE EXTERNAL TABLE t (...) LOCATION '/path'; | Hive keeps metadata plus pointer; files stay |
| DROP managed | DROP TABLE t; | Deletes metadata AND data |
| DROP external | DROP TABLE t; | Deletes metadata ONLY; files survive |
| Partition dirs | order_date=2026-08-10/ ... | One dir per value; partition column not in files |
| Static load | PARTITION (course='java') | Name each value by hand per load |
| Dynamic enable | SET hive.exec.dynamic.partition = true; + SET hive.exec.dynamic.partition.mode = nonstrict; | Lets Hive create dirs from row values |
| Dynamic insert | PARTITION (course) SELECT ..., course | Partition column last in SELECT |
| Pruning | WHERE order_date = '2026-08-11' | Opens only that dir; rest never touched |
| Good partition col | order_date, region, status | Low cardinality, often in WHERE |
| Bad partition col | customer_id, order_id, timestamp | High cardinality; use bucketing instead |
| Bucket rule | bucket_number = hash(value) mod n | INT: hash is value, so Id mod 3 |
| Bucket create | CLUSTERED BY (customer_id) INTO 4 BUCKETS | Fixed file count at CREATE TABLE time |
| Bucket enable | SET hive.enforce.bucketing = true; | Older decks set this before insert |
| Bucket map join | SET hive.optimize.bucketmapjoin = true; | Needs same join col + multiple counts |
| Sample | TABLESAMPLE(BUCKET 1 OUT OF 4 ON customer_id) | Reads one file, about 1/4 of rows |
| Combine | PARTITIONED BY (d) CLUSTERED BY (c) INTO 4 BUCKETS | Dir from partition col, file from bucket col |
| Inner join | JOIN ... ON a.id = b.id | Only matching rows; slide tables give 4 |
| Left / right / full | LEFT / RIGHT / FULL OUTER JOIN | Keeps left / right / both sides; NULLs fill gaps |
| Cross join | rows = m x n | 10 x 13 = 130; slide tables 7 x 4 = 28 |
| Id 5 in 3 buckets | 5 mod 3 = 2 | Bucket 2 |
Hive views and indexingSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| View idea | view = saved SELECT, no stored data | Virtual table; reruns query on every read |
| CREATE VIEW | CREATE VIEW v AS SELECT c1, c2 FROM t; | Save any result set; subset rows or columns |
| Filter view | CREATE VIEW v AS SELECT * FROM t WHERE s > 35000; | Strict test drops rows equal to the limit |
| ALTER query | ALTER VIEW v AS SELECT col1 FROM t; | Replace the defining query; base table kept |
| RENAME view | ALTER VIEW v RENAME TO w; | New name, same rows |
| When to index | large data + slow query + frequent WHERE | Index lives in a separate table; can partition it |
| Compact store | value + block id | Go straight to the block holding the value |
| Bitmap store | value + bit vector (1 bit per row) | 1 means the row holds the value |
| CREATE INDEX | CREATE INDEX i ON TABLE t(c) AS 'BITMAP'; | Type is handler class or BITMAP / COMPACT |
| Deferred rebuild | WITH DEFERRED REBUILD | Index starts empty until ALTER rebuilds it |
| SHOW INDEX | SHOW FORMATTED INDEX ON t; | Lists every index on the table |
| REBUILD | ALTER INDEX i ON t REBUILD; | Runs a MapReduce job; rebuild after append |
| ALTER syntax | ALTER INDEX i ON t REBUILD; | Missing ON raises ParseException |
| AND query | 0110 AND 0100 = 0100 | Intersect vectors bit by bit; row 2 wins |
| OR query | 1001 OR 0100 = 1101 | Union of row sets |
| NOT query | NOT 0110 = 1001 | Complement of the vector |
| Timing result | 276.815s compact vs 197.424s bitmap | Saving 79.391 s; ratio 1.40; same answer |
| Skip indexing | small data / unique rows / rare WHERE | Build cost beats the gain |
| DROP INDEX | DROP INDEX i ON t; | Deletes index and its table |
| Hive 3.0 | CREATE INDEX removed (HIVE-18448) | Use materialized views, ORC/Parquet stats |
| Cardinality | bucket high / Hive-partition low | Buckets need spread; partitions need few values |
ScalaSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| REPL divide | 10/6 gives 1 | Two Int values divide to Int; remainder dropped |
| Compile + run | scalac F.scala then scala F | Compile the file, run the class |
| Mutable | var data: Int = 100 | Reassignable; data = 101 works |
| Immutable | val myVal: String = "Foo" | Fixed; reassignment is an error |
| Inference | var myVar = 10 | Becomes Int by default |
| Multi-assign | val (a, b) = Pair(40, "Foo") | Tuple split into two val variables |
| String length | palindrome.length() | "Dot saw I was Tod" gives 17 |
| Concat | "a".concat("b") or "a" + "b" | New string with second added at end |
| Array literal | Array("Zara", "Nuha", "Ayan") | Fixed-size, one type |
| Array index expr | z(4/2) = "Ayan" | Index 2; any whole-number expression works |
| Matrix | Array.ofDim[Int](3, 3) | matrix; needs import Array._ |
| Arithmetic | a+b=30 / a-b=-10 / a*b=200 / b/a=2 / b%a=0 / c%a=5 | With a=10, b=20, c=25 |
| Range loop | for (a <- 1 to 10) | Inclusive range, 10 passes |
| Multi-range | for (a <- 1 to 3; b <- 1 to 3) | pairs, 18 print lines |
| Yield filter | for { a <- L if a != 3; if a < 8 } yield a | Drops 3 and values >= 8 |
| Basic class | class Student { var id: Int = 0 } | All fields need initial values |
| Anonymous object | new Arithmetic().add(10, 10) | No reference name; output sum = 20 |
| Inheritance | class Location(...) extends Point(xc, yc) | One superclass only; adds z, move gives 20, 30, 20 |
| Private | private def f() | Visible only inside defining class; (new Inner).f() fails |
| Protected | protected def f() in Super | Visible in subclasses only; Other fails, Sub works |
| Function | def addInt(a: Int, b: Int): Int | addInt(5, 7) returns 12 |
| Procedure | def printMe(): Unit | Returns nothing, like void |
| List build | List(1, 2, 3, 4) or 1 :: (2 :: Nil) | :: prepends, Nil is empty |
| head / tail | List(a, o, p).head is a | tail is List(o, p); Nil.isEmpty is true |
| Concat lists | fruit1 ::: fruit2 | Joins lists; .::: binds right so order flips |
| Set | Set(1, 3, 5, 7) | Pairwise different elements, one type |
| Map lookup | colors("red") | Value for key; keys unique, values repeat |
| Concat maps | colors1 ++ colors2 | Duplicate keys collapse to one |
| Tuple access | t._1 + t._2 + t._3 + t._4 | (4, 3, 2, 1) sums to 10 |
| Iterator | it.hasNext / it.next() | One-by-one walk; hasNext false ends loop |
Apache SparkSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| MapReduce passes | 2k passes for k iterations | One read plus one write per iteration |
| Cached Spark passes | 1 pass | Initial read, then reuse from RAM |
| 10 iters on 50 GB | 20 x 50 GB = 1000 GB = 1 TB vs 50 GB | Spark moves 20x less disk data |
| In-memory claim | Spark time = Hadoop time / 100 | 2014-era claim, varies by workload |
| On-disk claim | Spark time = Hadoop time / 10 | 2014-era claim, varies by workload |
| GraySort time | 72 / 23 = 3.13 | Same 100 TB, one third of time |
| GraySort nodes | 2100 / 206 = 10.19 | Same 100 TB, one tenth of nodes |
| Deployment modes | Standalone / YARN / Kubernetes | Own manager / shared / pods, no Hadoop |
| Removed options | Mesos removed in 4.0; SIMR obsolete | Do not list them as current |
| Spark Core | scheduling, memory, recovery, storage | Base for SQL, Streaming, MLlib, GraphX |
| Catalyst + AQE | plan + runtime re-optimize | AQE GA since Spark 3.0 |
| Streaming now | Structured Streaming (DataFrame) | DStreams is legacy maintenance |
| ML now | spark.ml (DataFrame Pipelines) | spark.mllib is legacy maintenance |
| Graph | GraphX via Pregel; GraphFrames alternative | GraphX stable, little new work |
| SparkR status | deprecated in Spark 4.0 | Use PySpark or Spark SQL |
| Word count | flatMap(_.split(" ")).groupBy("value").count() | Two Scala lines vs dozens of Java |
| Line count | spark.read.textFile("data/sample.txt").count() | Prints The file has 100000 lines. |
| Local run | .master("local[*]") | All cores; test and dev only |
| Setup | Java 17+ and Scala 2.13 for Spark 4.x | Check with java -version |
Machine learning with Spark MLlibSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Iterative disk cost | passes ≈ 2k | Each of k iterations writes plus reads HDFS |
| Cached Spark cost | 1 disk pass, k memory passes | Load once, cache, iterate in memory |
| K-means 20 iters | 2 x 20 = 40 vs 1 | MapReduce 40 passes; cached Spark 1 |
| Lineage recovery | replay DAG for lost partition | Recompute only lost partition from source |
| Mahout vs MLlib | 100x memory, 10x disk | Cited speedup; workload-dependent |
| DataFrame | spark.read.format("libsvm").load(path) | Table-like source for spark.ml |
| Transformer | .transform(df) | Appends columns; feature step or fitted model |
| Estimator | .fit(df) | Learns on train data; returns a Transformer |
| Pipeline | .setStages(Array(...)) | Ordered chain of Transformers and Estimators |
| PipelineModel | pipeline.fit(train) | Same stage count; Estimators became Transformers |
| MinMax range | rescaled = min + (max-min)(v-min_i)/(max_i-min_i) | Defaults min 0.0, max 1.0, shared by features |
| MinMax slide rows | [1,0.1,-1] -> [0,0,0] | Minima map to 0; maxima [3,10.1,3] -> [1,1,1] |
| StandardScaler | z = (x - mu) / sigma | withStd divides; withMean subtracts |
| VectorAssembler | .setInputCols(Array("hour","mobile","userFeatures")) | Merges columns into one vector |
| StringIndexer | .setInputCol("category").setOutputCol("categoryIndex") | Most frequent label gets index 0 |
| Slide indices | a -> 0, c -> 1, b -> 2 | Counts 3, 2, 1 |
| OneHotEncoder | one binary col per category | At most one 1 per row group |
| Binarizer | 1.0 if value > threshold else 0.0 | Threshold 0.5: 0.8 -> 1.0; rest 0.0 |
| VectorSlicer | .setIndices(Array(1)) | Keeps positions 1-2: [-2,2.3,0] -> [2.3,0] |
| VectorIndexer | .setMaxCategories(4) | Above 4 distinct values stays continuous |
| Pearson | r = sum((xi-xbar)(yi-ybar)) / sqrt(sum(xi-xbar)^2 sum(yi-ybar)^2) | Linear fit on raw values |
| Spearman | rho = 1 - 6 sum(d^2) / n(n^2-1) | Monotonic fit on ranks |
| Slide ranks rho | 1 - 120/990 = 0.879 | Ten ranks; sum d² = 20 |
| Age ranks rho | 1 - 48/504 = 0.905 | Eight people; sum d² = 8 |
| Outlier effect | pearson -0.205 vs spearman 0.697 | One x=1000 point flips Pearson |
| Outlier removed | pearson 0.943 vs spearman 0.992 | Both agree without the extreme point |
| LogisticRegression | .setMaxIter(10).setRegParam(0.3).setElasticNetParam(0.8) | Seed 5043; tune threshold by max F-Measure |
| Decision tree | .setMaxCategories(4) | Index labels and features; 70/30 split |
| Random forest | .setNumTrees(10) | Same stages as tree with rf swapped in |
| K-means legacy | KMeans.train(data, 2, numIterations = 20) | RDD API; reports sum of squared errors |
| K-means modern | .setK(2).setSeed(1L) | Score with Silhouette, squared euclidean |
| Naive Bayes | randomSplit(Array(0.7, 0.3), 1234L) | Fit, predict, score accuracy |
| LinearSVC | .setMaxIter(10).setRegParam(0.1) | Inspect coefficients and intercept |
| MLP layers | Array(4, 5, 4, 3) | Block 128, seed 1234, maxIter 100, 60/40 split |
Graph processing with GraphXSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Property graph | directed multigraph + properties on every vertex and edge | Parallel edges allowed; immutable, distributed, fault-tolerant |
VertexId | type VertexId = Long (64-bit) | One entry per vertex in VertexRDD |
| Graph build | Graph(verRDD, edgeRDD) | Parallelize vertex and edge arrays, then build |
| Imports | `import org.apache.spark.graphx._` | Plus Edge, Graph, GraphLoader as needed |
| Vertex filter | `graph.vertices.filter { case (id, (city, pop)) => pop > 50000 }` | Keeps 5 cities; drops Harrisburg (49528) |
| Edge filter | `graph.edges.filter { case Edge(a, b, d) => d < 150 }` | Keeps 5 edges: 113, 106, 128, 39, 130 |
| Triplets | `graph.triplets.collect` | One record per edge: srcAttr, attr, dstAttr |
| Degrees | deg(v) = indeg(v) + outdeg(v) | inDegrees, outDegrees, degrees; sums equal numEdges |
| Neighbour sum | `ugraph.aggregateMessages[Int](ectx => ectx.sendToSrc(ectx.dstAttr._2), _ + _)` | Union with graph.reverse.edges first for both directions |
| Neighbour totals | Vertex 3 | |
| Basic operators | `numEdges` `numVertices` `inDegrees` `outDegrees` `degrees` | 5 basic operators |
| Property operators | `mapVertices` `mapEdges` `mapTriplets` | New graph, same structure |
reverse | `graph.reverse` | Flip every edge direction; count and properties unchanged |
subgraph | `graph.subgraph(epred, vpred)` | Keep vertices and edges that pass the predicates |
mask | `graph.mask(other)` | Keep vertices and edges also present in other |
groupEdges | `graph.groupEdges(merge)` | Merge parallel edges with merge |
joinVertices | `graph.joinVertices(table)(map)` | Update matched vertices only; same vertex type |
outerJoinVertices | `graph.outerJoinVertices(table)(map)` | Update all vertices (Some/None); may change type |
| PageRank | PR(v) = (1-d)/N + d\sum PR(u)/\text{outdeg}(u) | graph.pageRank(0.0001); tolerance on rank change |
| Connected components | `graph.connectedComponents().vertices` | Each piece labelled with an id |
| Triangle count | `graph.triangleCount().vertices` | Needs canonical orientation () plus partitionBy |
Stream concepts and algorithmsSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Batch vs stream | bounded + exact vs unbounded + per-window | Minutes-hours exact vs ms-s incremental |
| Stream traits | Continuous, Unbounded, Time-ordered, Fast, Transient | Cut Fast Traffic; seen once |
| Tumbling counts | [0,5):3, [5,10):2, [10,15):3 | 8 events, no overlap |
| Sliding counts | [0,6),[3,9),[6,12),[9,15): 3 each | Size 6 s, slide 3 s |
| Session counts | gap 7.5->11 = 3.5s closes: 5 + 3 | 3 s gap rule |
| Watermark | W = max(event time) - 10 min | Accept > W, drop < W |
| Reservoir keep | P(keep i) = k/i; P(in sample) = k/n | k=2, n=5 gives 2/5 each |
| Bloom hashes | h1 = x mod 10; h2 = (2x+3) mod 10 | 25->bits 5,3; 42->bits 2,7 |
| Bloom rule | 0 bit = absent; all 1 = possibly present | No false negatives |
| FM distinct | n-hat = 2^R | R = longest trailing-zero run |
| DGIM | sum buckets except oldest + oldest/2 | Powers of two, at most two per size |
| Count-Min | query = min over rows | Never undercounts |
| Lambda | batch + speed -> merge | Logic twice, batch fixes errors |
| Kappa | one log + one job, replay from 0 | Logic once, swap when caught up |
| Record vs micro-batch | ms (Flink/Storm) vs ~interval (Spark) | Snapshots vs lineage recovery |
| At-most-once | commit offset before processing | Loss possible, no duplicates |
| At-least-once | commit offset after processing | Duplicates possible, no loss |
| Exactly-once | checkpoints + replay + idempotent sink | Atomic output and offset |
| Checkpoint recovery | restore at 120, re-read 120->150 | No loss, no double count |
| Scaling | partition by key; 1 Kafka partition -> 1+ task | Same key stays ordered |
| Backpressure | maxOffsetsPerTrigger; Flink flow control | Slow intake, Kafka buffers bursts |
Streaming pipelines and KafkaSearch on YouTube
| Topic | Formula | Meaning |
|---|---|---|
| Pipeline stages | sources / ingest / process / store / serve | S-I-P-S-S; log gives decoupling, fan-out, replay |
| Fraud rules | amount > 5x avg OR two cities in 10 min | Key by card; watermark on ts; alerts + Delta Lake |
| DStream | one RDD per batch interval | Legacy spark.streaming; reduceByKey is per batch |
| Unbounded table | groupBy(word).count() | Spark keeps counts as state per trigger |
| Complete mode | write entire result table | Small aggregate tables |
| Update mode | write only changed rows | Large aggregate tables |
| Append mode | write only new final rows | Needs watermark for aggregates |
| Watermark | max event time seen - delay | Older events drop to late-data path |
| Slide windows | tumbling 3-2-3; sliding 3-3-3-3; sessions 5-3 | 8 events at 1..14 s; sliding double-counts |
| Kafka partition | hash(key) mod p | One key stays in one partition, in order |
| Ordering | within partition only | No order across partitions |
| Retention | delete by age or size | Reads never delete events |
| Groups | share within, broadcast across | Extra consumers in a group sit idle |
| Replication | leader + followers; ISR | min.insync.replicas = 2, acks = all survives 1 failure |
| acks | 0 = nothing / 1 = leader / all = in-sync | N-L-E: safest is all, fastest is 0 |
| Exactly-once writes | enable.idempotence + transactions | Kill retry duplicates; atomic across partitions |
| Compaction | keep latest value per key | Tombstone (null) deletes a key |
| ksqlDB window | WINDOW TUMBLING (SIZE 1 MINUTE) | Per-sensor 1-min AVG(temp) table |