pandas essentials

Lesson 43 of 62, about 4 min

pandas gives you the DataFrame: a table with named columns, like a spreadsheet you drive with code. Almost every machine learning project starts by loading data into a DataFrame and cleaning it there.

Series and DataFrame

A Series is one column. A DataFrame is a collection of Series sharing an index.

python

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Bilal", "Chitra", "Dev", "Esha"],
    "city": ["Jaipur", "Delhi", "Jaipur", "Pune", "Delhi"],
    "marks": [88, 72, 95, 60, 79],
    "age": [21, 22, 20, 23, 21],
})
print(df)
print(type(df["marks"]))

Looking at the data

python

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Bilal", "Chitra", "Dev", "Esha"],
    "city": ["Jaipur", "Delhi", "Jaipur", "Pune", "Delhi"],
    "marks": [88, 72, 95, 60, 79],
    "age": [21, 22, 20, 23, 21],
})
print(df.head(3))
print(df.shape)
print(df.dtypes)
print(df.describe())
print(df["city"].value_counts())

Real projects usually start with pd.read_csv("file.csv"); df.to_csv("out.csv", index=False) writes one.

Selecting columns and rows

python

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Bilal", "Chitra", "Dev", "Esha"],
    "city": ["Jaipur", "Delhi", "Jaipur", "Pune", "Delhi"],
    "marks": [88, 72, 95, 60, 79],
})
print(df["marks"])                 # one column, a Series
print(df[["name", "marks"]])       # several columns, a DataFrame
print(df.iloc[0])                  # row by position
print(df.loc[2, "name"])           # row label and column name
print(df.iloc[1:3, 0:2])           # slices by position

Filtering

Boolean masks, exactly like NumPy.

python

import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Bilal", "Chitra", "Dev", "Esha"],
    "city": ["Jaipur", "Delhi", "Jaipur", "Pune", "Delhi"],
    "marks": [88, 72, 95, 60, 79],
})
print(df[df["marks"] >= 75])
print(df[(df["city"] == "Delhi") & (df["marks"] > 75)])
print(df[df["city"].isin(["Jaipur", "Pune"])]["name"].tolist())

Use &, | and ~ with parentheses around each condition; and, or, not do not work on Series.

Adding and changing columns

python

import pandas as pd

df = pd.DataFrame({"name": ["Asha", "Bilal"], "marks": [88, 72]})
df["percent"] = df["marks"] / 100
df["grade"] = df["marks"].apply(lambda m: "A" if m >= 80 else "B")
df["name"] = df["name"].str.upper()
print(df)

Sorting

python

import pandas as pd

df = pd.DataFrame({"name": ["Asha", "Bilal", "Chitra"], "marks": [88, 72, 95]})
print(df.sort_values("marks", ascending=False))

Grouping and summarising

groupby splits the table by a column, applies a function to each group, and joins the results back together.

python

import pandas as pd

df = pd.DataFrame({
    "city": ["Jaipur", "Delhi", "Jaipur", "Pune", "Delhi"],
    "marks": [88, 72, 95, 60, 79],
})
print(df.groupby("city")["marks"].mean())
print(df.groupby("city").agg(count=("marks", "size"), best=("marks", "max")))

Missing values

python

import pandas as pd
import numpy as np

df = pd.DataFrame({"a": [1, np.nan, 3], "b": ["x", "y", None]})
print(df.isna())
print(df.isna().sum())
print(df.fillna({"a": df["a"].mean(), "b": "unknown"}))
print(df.dropna())

Combining tables

python

import pandas as pd

students = pd.DataFrame({"id": [1, 2, 3], "name": ["A", "B", "C"]})
marks = pd.DataFrame({"id": [1, 2, 4], "marks": [90, 80, 70]})
print(pd.merge(students, marks, on="id", how="left"))
print(pd.concat([students, students], ignore_index=True))

From DataFrame to model

scikit-learn accepts DataFrames directly. The convention is X for the feature columns and y for the target column.

python

import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.DataFrame({"hours": [1, 2, 3, 4, 5], "marks": [52, 58, 66, 71, 80]})
X = df[["hours"]]
y = df["marks"]
model = LinearRegression().fit(X, y)
print(model.predict(pd.DataFrame({"hours": [6]})).round(1))

Practice

Build a DataFrame of six products with category and price. Print the average price per category, sorted from highest to lowest.
Add a column that marks products above the overall mean price as "premium".
Introduce a missing price, then fill it with the median of its category.