Data with Pandas

From Arrays to Tables

NumPy arrays are powerful but anonymous — a matrix of numbers with no column names, no row labels, no concept of "this column is salary, that column is department."

Real data has structure. It has named columns, labelled rows, mixed types, and missing values. Pandas brings all of this together in a data structure called a DataFrame — essentially a table, like a spreadsheet, but programmable.

Pandas is built on NumPy and inherits its performance, while adding the expressiveness needed for real data analysis.

The DataFrame

A DataFrame is a collection of columns. Each column is a Series — a labelled 1D array. The row labels on the left (0, 1, 2...) form the index.

shape gives (rows, columns). dtypes shows the type of each column. describe() gives instant summary statistics — count, mean, standard deviation, percentiles — for all numeric columns. Three lines of information that would take twenty lines of manual code.

Selecting Columns and Rows

A single column name returns a Series. A list of column names returns a DataFrame.

Rows are selected with .loc (by label) and .iloc (by integer position):

Filtering — Boolean Indexing

Just as in NumPy, a condition produces a boolean Series, which is then used as a mask:

Multiple conditions using & (and) and | (or) — with each condition in parentheses:

Adding and Transforming Columns

New columns are created by assignment:

Each new column is derived from existing ones — the operation applies to every row simultaneously.

Sorting

sort_values returns a new DataFrame — the original is untouched. Pass ascending=False for descending order.

Grouping and Aggregation

groupby is one of Pandas' most powerful features — splitting a DataFrame into groups and applying a function to each:

groupby("dept") splits the DataFrame by department. ["salary"] selects the salary column. .agg(["mean","min","max","count"]) computes all four statistics for each group. What would take a dozen lines with a dictionary and a loop is expressed in one line.

Reading CSV Data

In real work, data comes from files. pd.read_csv() loads a CSV file into a DataFrame:

On a real machine, you would write pd.read_csv("products.csv") directly with a file path. The io.StringIO wrapper is used here because the sandbox has no local file system to read from — the CSV data is embedded as a string instead. Everything else is identical.

Handling Missing Data

Real data is rarely complete. Pandas represents missing values as NaN (Not a Number):

• isnull().sum() — counts missing values per column
• dropna() — removes any row with at least one missing value
• fillna(value) — replaces missing values with a specified value or dictionary of values

A Short Analysis Pipeline

Data in, insights out — reading, deriving a column, computing aggregates, finding maxima per group, displaying summaries. This is the shape of most data analysis work.

What You Have Learned

Pandas brings labelled, structured data analysis to Python — built on NumPy, expressed in a clean, readable API.

The key ideas:

• A DataFrame is a table of named columns and labelled rows
• describe() gives instant summary statistics for all numeric columns
• Select columns with df["col"] or df[["a","b"]]; rows with .iloc (position) or .loc (label)
• Boolean indexing filters rows: df[df["col"] > value]
• New columns are derived by assignment: df["new"] = df["a"] + df["b"]
• groupby("col").agg([...]) splits, applies, and combines in one expression
• read_csv() loads files; isnull(), dropna(), fillna() handle missing data
• Pandas operations return new DataFrames — the original is untouched by default

In the final lesson of this series, you'll learn testing — how to write code that verifies your code, and why this habit transforms the way you program.