Mastering Your Data

Part of the free Generative AI course on LogicWiz, module: Leveling Up Your Powers.

Episode 7: Mastering Your Data

"Data without analysis is just noise. Data with analysis is intelligence."

Nova's Analytical Brain

Nova can load data and filter it. But your manager wants more: "Show me the average response time per category. Which topics get the most questions? What's trending this week?"

This requires aggregation and precise selection — the ability to group, summarize, and slice data with surgical precision. These are essential for:

  • Summarizing data by categories
  • Building features for ML models
  • Creating reports and dashboards

This is the last Python skill you need before Nova comes alive.

Grouping with groupby() and agg()

groupby() groups rows by one or more columns, then agg() applies aggregation functions:

df.groupby("Pclass").agg({"Age": "mean", "Fare": "median"})

This groups by Pclass and calculates:

  • Mean of Age
  • Median of Fare

Multiple Aggregations on One Column

Pass a list of function names:

df.groupby("Pclass").agg({"Age": ["mean", "median", "count", "sum"], "Fare": "median"})

This creates a multi-level column header: Age has sub-columns mean, median, count, sum.

Grouping by Multiple Columns

Pass a list of column names to groupby():

df.groupby(["Pclass", "Sex"]).agg({
    "Age": "mean",
    "Fare": "median",
    "Survived": "mean"
})

This creates a multi-level row index (Pclass then Sex nested under each class). You can report the average age, median fare, and survival rate for each combination.

{{visual:groupby-walkthrough}}

{{cell:l7-try-groupby}}

.loc and .iloc: Surgical Precision

Sometimes Nova needs exactly row 5, columns 2-4. Or all rows where the category is "AI", but only the title and date columns. These are powerful methods for selecting specific rows and columns.

.loc (Label-Based)

Selects by row labels and column names:

customer.loc[0:1, ["Age", "Education"]]
  • First part (before comma): row selection (labels 0 and 1, inclusive)
  • Second part (after comma): column selection by name

.iloc (Integer Position-Based)

Selects by integer positions:

customer.iloc[[0, 2], [0, 1, 3]]
  • Selects rows at positions 0 and 2
  • Selects columns at positions 0, 1, and 3

When to Use Which?

  • Use .loc when you know column names (more robust if order changes)
  • Use .iloc when you know exact index positions
  • .iloc is always relative to the current index (even after filtering)

{{visual:loc-iloc-walkthrough}}

{{cell:l7-try-loc-iloc}}

Creating New Columns: Computed Intelligence

Nova often needs derived data — like a "relevance score" computed from multiple columns. Assign a value to a new column name:

# Scalar value for all rows
customer["creditScore"] = 710

# Based on another column
customer["creditScore"] = customer["Age"]

# Based on a formula
customer["creditScore"] = customer["Age"] + 10 * customer["Income"]

{{visual:new-col-walkthrough}}

{{cell:l7-try-new-col}}

Summary of Key Pandas Operations

Operation Code
Load CSV pd.read_csv(url)
First N rows df.head(n)
Last N rows df.tail(n)
Random rows df.sample(n)
Column info df.info()
Statistics df.describe()
Value counts df["col"].value_counts()
Filter df[df["col"] == value]
AND filter df[(cond1) & (cond2)]
Group + aggregate df.groupby("col").agg({...})
Label selection df.loc[rows, cols]
Position selection df.iloc[rows, cols]
New column df["new"] = value

Mission 7: Make Nova Data-Fluent

Your final Python mission:

  • Group data and compute aggregates
  • Select specific rows and columns
  • Add computed columns

Complete this mission to fully earn the title: Knowledge Architect

What comes next: You've mastered Python. You can store data, manipulate text, make decisions, build functions, use libraries, and analyze data. Now it's time for the moment you've been building toward — Nova comes alive. In the next module, you'll build your first AI system from scratch. The chatbot. The LLM. The agent. This is where AI stops being a concept and becomes YOUR creation.