Data Analysis with pandas DataFrames

Data Analysis with pandas DataFrames

pandas turns raw CSV, Excel, and database exports into a labeled table called a DataFrame, with names on both rows and columns. Nearly every ML pipeline starts by loading data into a DataFrame and ends by reshaping it.

Load and Peek

import pandas as pd

df = pd.read_csv("titanic.csv")
print(df.head(10))
print(df.info())
print(df.describe())

head shows sample rows, info shows column types and missing counts, and describe summarizes numeric columns. These three calls answer most questions about a new dataset.

Select and Filter

Pick a column with df["age"], several with a list, and rows with a condition. Boolean conditions are the pandas replacement for spreadsheet filters.

adults = df[df["age"] >= 18]
survivors = df[df["survived"] == 1]
print(adults.shape, survivors.shape)

Group and Aggregate

Grouping splits the data and applies a function to each group:

by_class = df.groupby("pclass")["survived"].mean()
print(by_class)

The result shows survival rate by passenger class in one line, and it is the seed of most exploratory analysis.

Handle the Basics

Missing values show up as NaN. isna() counts them, dropna() removes rows, and fillna() replaces gaps with a fixed value or a statistic. Do not remove data casually; check both the count and the meaning before choosing a strategy.

print(df.isna().sum())
df2 = df.copy()
df2["age"] = df2["age"].fillna(df2["age"].median())

Save Progress

Export the cleaned frame with .to_csv("clean.csv", index=False) so the next stage of the pipeline starts from a known state.

Key Points

• DataFrames are labeled tables for analysis.
• Head, info, and describe reveal the data quickly.
• Boolean filters and groupby cover most queries.
• Handle NaN deliberately; do not just drop it.