Pig statements, types and joins

Pig Latin Data Model and Data TypesCode With Z · Pig data model and typesApache Pig LatinAcademy of Fine Learning · Pig Latin data types

Pig Latin statementsSearch on YouTubeBookmarkSaved

A Pig Latin statement is an operator that takes a relation as input and produces a relation as output. LOAD and STORE are the exception. They read from the file system and write to it instead of mapping relation to relation.

Statements can span several lines. Every statement must end with a semicolon (;). A script has a fixed shape: LOAD, then a series of transformations, then DUMP or STORE.

DUMP or STORE is required to get output. Without one, Pig checks the script but never runs it. This split matters for debugging: a script with no output action only gets validated.

A = LOAD 'student' USING PigStorage()
    AS (name:chararray, age:int, gpa:float);
B = FOREACH A GENERATE name;

A = LOAD 'student' USING PigStorage()
    AS (name:chararray, age:int, gpa:float);
B = FOREACH A GENERATE name;
DUMP B;

(John)
(Mary)
(Bill)
(Joe)

Comments, case and REGISTERSearch on YouTubeBookmarkSaved

Line comments start with --. Block comments sit between /* and */. Keywords such as LOAD and FOREACH are case-insensitive. Relation names, field names and function names are case-sensitive, so Students and students are different relations.

REGISTER makes a JAR file available to a script. The usual use is custom UDFs (Part 3 covers them). After registering, the script calls the UDF by its fully qualified class name inside FOREACH ... GENERATE.

-- single-line comment
/* multi-line
   comment block */
REGISTER ./tutorial.jar;

Multi-query executionSearch on YouTubeBookmarkSaved

By default Pig runs all statements of a script together. It looks for ways to share work across the several DUMP and STORE outputs of one script. Batch runs of a .pig script are thus more efficient than typing the same statements one by one in Grunt.

Data typesSearch on YouTubeBookmarkSaved

Every field has a type from the table below. LOAD ... AS sets types explicitly. Otherwise Pig infers them. The complex types (tuple, bag, map) repeat the Part 1 model. A field (also called atom) builds a tuple. Tuples build a bag. Bags build a relation. A map holds key-value pairs.

Complex shapes: tuple (raja, 30), bag {(raju,30),(md,45)}, map [name#Raju, age#30].

NULL valuesSearch on YouTubeBookmarkSaved

Every type, scalar or complex, can hold NULL. A NULL is an unknown or missing value. It serves as a placeholder for optional data. Nulls come from the source data itself. They also come from operations: the unmatched side of an outer JOIN, or a field that a UDF could not compute. Pig treats nulls much like SQL does.

clean = FILTER students BY city is not null;

clean = FILTER students BY city is not null;

Expression operatorsSearch on YouTubeBookmarkSaved

Arithmetic and conditional operatorsSearch on YouTubeBookmarkSaved

Suppose a = 10 and b = 20. Pig provides +, -, *, / and % (remainder). The bincond ?: picks one of two values from a boolean test. CASE ... WHEN ... THEN ... END chains several branches and reads better than nested binconds.

x = (a == 1) ? 20 : 30;
CASE f2 % 2
  WHEN 0 THEN 'even'
  WHEN 1 THEN 'odd'
END

Comparison and type construction operatorsSearch on YouTubeBookmarkSaved

f1 matches '.*tutorial.*' is true when f1 holds tutorial anywhere. The dots match any characters and the stars repeat them.

The three bracket pairs build complex values inline:

These literals have the same shapes as the data types above. Use them to write tuple, bag and map values direct in a script.

Inner JOINSearch on YouTubeBookmarkSaved

JOIN combines tuples from two or more relations with matching values on a key. By default it is an inner join: only keys present in every relation appear in the result. A tuple with no match on the other side drops out entire. That drop is what makes it "inner".

students = LOAD 'students.txt' USING PigStorage(',')
    AS (id:int, name:chararray, city:chararray);
fees = LOAD 'fees.txt' USING PigStorage(',')
    AS (id:int, amount:int);
paid = JOIN students BY id, fees BY id;

(1,Rajiv,Hyderabad,1,500)
(3,Rajesh,Delhi,3,700)

A relation can join with itself (self join). Load it twice under two aliases. Then JOIN the aliases by the shared key.

Outer JOINsSearch on YouTubeBookmarkSaved

Outer joins keep the tuples that an inner join would drop. The missing side fills with NULL.

left_j = JOIN students BY id LEFT OUTER, fees BY id;
right_j = JOIN students BY id RIGHT OUTER, fees BY id;
full_j = JOIN students BY id FULL OUTER, fees BY id;

(1,Rajiv,Hyderabad,1,500)
(2,Omer,Kolkata,,)
(3,Rajesh,Delhi,3,700)

(1,Rajiv,Hyderabad,1,500)
(3,Rajesh,Delhi,3,700)

(1,Rajiv,Hyderabad,1,500)
(2,Omer,Kolkata,,)
(3,Rajesh,Delhi,3,700)

COGROUPSearch on YouTubeBookmarkSaved

GROUP groups one relation by a key. COGROUP groups two or more relations by a shared key at once. Each key gives one result tuple of the shape below.

cogrouped = COGROUP students BY id, fees BY id;

(1, {(1,Rajiv,Hyderabad)}, {(1,500)})
(2, {(2,Omer,Kolkata)}, {})
(3, {(3,Rajesh,Delhi)}, {(3,700)})

Key 2 still appears, with an empty bag on the fees side. COGROUP never drops a key in silence. That is the core difference from an inner JOIN.

The slides give the orders/customers case: joining orders with customers by customer_id merges each order and its customer into one flat row. COGROUP instead gives one bag of orders and one bag of customer info per customer_id. That shape suits a per-customer total computed before any flattening.

STREAMSearch on YouTubeBookmarkSaved

STREAM sends a relation through an external program in any language. Each tuple goes to the program's stdin, one line at a time. Each line of stdout becomes a tuple of the output relation.

cleaned = STREAM raw_logs THROUGH `normalize.py`;

Reasons to use it:

• Reuse an existing Perl, Python or shell script instead of rewriting logic as a UDF.
• Plug in text-processing or cleansing tools already in use outside Pig.
• Prototype a transform fast before writing it as a formal Java UDF.

The slides reuse a Python cleaner this way. It applies a messy-city-name script direct, with no Java rewrite.

cleaned = STREAM orders THROUGH `clean_city.py`;

DESCRIBE, EXPLAIN and ILLUSTRATESearch on YouTubeBookmarkSaved

These three commands show what a script does without full runs:

Grunt shortcuts for the same commands:

DESCRIBE enriched;

enriched: {valid::order_id: int, valid::customer_id: int,
  valid::city: chararray, valid::amount: double,
  customers::customer_id: int, customers::name: chararray,
  customers::signup_year: int}

Practice pipeline: e-commerce ordersSearch on YouTubeBookmarkSaved

The slides end with a bookstore pipeline over two small CSV files. It answers business questions step by step.

Pipeline part 1: revenue by citySearch on YouTubeBookmarkSaved

orders = LOAD 'orders.csv' USING PigStorage(',')
    AS (order_id:int, customer_id:int, city:chararray, amount:double);
valid = FILTER orders BY amount > 0 AND city is not null;
by_city = GROUP valid BY city;
revenue = FOREACH by_city GENERATE
    group AS city, SUM(valid.amount) AS total;
ranked = ORDER revenue BY total DESC;
top5 = LIMIT ranked 5;

Hyderabad: {(101,1,Hyderabad,450.0), (103,1,Hyderabad,220.5)}
Delhi:     {(104,3,Delhi,180.0)}

(Hyderabad,670.5)
(Delhi,180.0)

Pipeline part 2: customers with and without ordersSearch on YouTubeBookmarkSaved

customers = LOAD 'customers.csv' USING PigStorage(',')
    AS (customer_id:int, name:chararray, signup_year:int);
enriched = JOIN valid BY customer_id, customers BY customer_id;
inactive = JOIN customers BY customer_id LEFT OUTER, valid BY customer_id;
DESCRIBE enriched;
EXPLAIN revenue;
ILLUSTRATE enriched;

(101,1,Hyderabad,450.0,1,Rajiv,2021)
(103,1,Hyderabad,220.5,1,Rajiv,2021)
(104,3,Delhi,180.0,3,Rajesh,2020)

(1,Rajiv,2021,101,1,Hyderabad,450.0)
(2,Omer,2022,,,,)
(3,Rajesh,2020,104,3,Delhi,180.0)
(4,Divya,2023,,,,)

Quick self-testSearch on YouTubeBookmarkSaved