Filtering & Sorting

This lesson on Filtering & Sorting is hands-on and example-driven. You will learn to isolate specific rows in a Pandas DataFrame using conditional filtering and logical operators. You will also master the .sort_values() method to order your data based on single or multiple column priorities. These skills are essential for preparing data for analysis.

What You'll Be Able To Do

• Filter a DataFrame using a single numerical condition.
• Apply multiple logical conditions using the OR operator (|).
• Sort a DataFrame by the values in a designated column.
• Define multi-level sorting priorities using list syntax.
• Write Boolean masks to select specific rows.

Detailed Concept Walkthrough

1. Conditional Filtering

Filtering uses a Boolean mask—a Series of True/False values—to select only the rows where the condition evaluates to True. This is the fundamental mechanism for row selection in Pandas.

• Mechanism: When you apply a condition (e.g., df['Score'] > 80), Pandas checks every row and returns a Series of Booleans.
• Under the Hood: This Boolean Series is passed back into the DataFrame indexer (df[...]), which extracts only the rows corresponding to True values.
• Syntax Rule: The condition must reference a column name and use standard Python comparison operators (>, <, ==, etc.).

filtered_df = df[df['Score'] > 80] # Selects rows where the 'Score' column value is greater than 80

Key Takeaway: Filtering is achieved by indexing the DataFrame with a Series of True/False values.

2. Logical OR Filtering

To select rows that meet one condition OR another, you combine Boolean masks using the vertical bar (|), which represents the logical OR operator.

• Mechanism: Each individual condition must be wrapped in parentheses to ensure correct order of operations before the OR logic is applied.
• Best Practice: Use | for OR conditions and & for AND conditions when combining Pandas Boolean Series, as standard Python or and and operators do not work on Series.
• Execution Flow: Pandas evaluates the first mask, then the second mask, and finally combines them, returning True if either mask was True for that row.

math_or_science = df[(df['Subject'] == 'Math') | (df['Subject'] == 'Science')] # Selects rows where Subject is Math OR Science

Key Takeaway: Use parentheses and the | operator to combine multiple filtering conditions effectively.

3. Sorting DataFrames

The .sort_values() method reorders the DataFrame rows based on the values present in one or more specified columns.

• Mechanism: You must specify the by parameter, which takes the name of the column(s) to sort by.
• Best Practice: By default, sorting is ascending (smallest to largest). To sort descending, you must explicitly set ascending=False.
• Under the Hood: Pandas creates a new DataFrame with the rows reordered; the original DataFrame remains unchanged unless you assign the result back.

sorted_df = df.sort_values(by='Score') # Sorts the DataFrame rows based on the 'Score' column, ascending by default

Key Takeaway: Use df.sort_values(by='ColumnName') to reorder the entire dataset based on column values.

4. Multi-Column Sorting Priority

When sorting by multiple columns, Pandas uses a list to define the sorting hierarchy: it sorts by the first column, and then uses the second column to break ties in the first.

• Syntax Rule: Pass a list of column names to the by parameter: by=['Col1', 'Col2'].
• Execution Flow: All rows are sorted by 'Col1'. If two rows have the same value in 'Col1', their relative order is then determined by their values in 'Col2'.
• Best Practice: The order of columns in the list dictates the strict priority of the sort operation.

multi_sorted = df.sort_values(by=['Subject', 'Score']) # Sorts by Subject first, then by Score within each Subject group

Key Takeaway: List order in the by parameter defines the strict priority for multi-level sorting.

Topics Covered in Filtering & Sorting

1. Logical OR Filtering (1:21 - 2:30) — The instructor demonstrates combining two conditions using the vertical bar operator to select rows matching either criteria.
2. Conditional Filtering (2:31 - 3:50) — Syntax for selecting rows based on numerical column values using comparison operators is shown.
3. Single Column Sort (4:00 - 4:40) — The basic usage of the .sort_values() method is introduced using one column name in the 'by' parameter.
4. Multi-Column Sort (4:40 - 5:30) — The method for defining sorting priority using a list of column names is explained to handle tie-breaking.

Python Cheat Sheet

• df[condition] — Selects rows where the condition is True

  df[df['Score'] > 90]
  

• (cond1) | (cond2) — Combines two conditions using logical OR

  df[(df['Subject']=='Math') | (df['Score'] < 70)]
  

• df.sort_values(by='Col') — Sorts DataFrame by a single column

  df.sort_values(by='Score')
  

• by=['Col1', 'Col2'] — Defines multi-level sorting priority

  df.sort_values(by=['Grade', 'Score'])
  

• df['Col'] > value — Creates a Boolean Series mask

  df['Score'] >= 80
  

Comparison Table

Common Pitfalls

• Mistake: Using Python's or operator inside Pandas filtering. Avoid: Always use the bitwise operator | for logical OR.
• Mistake: Forgetting parentheses around complex conditions. Avoid: Wrap each individual condition in parentheses (cond1).
• Mistake: Assuming sort is permanent without assignment. Avoid: Assign the result back to the DataFrame: df = df.sort_values(...).
• Mistake: Sorting by multiple columns without a list. Avoid: Pass the column names as a list: by=['Col1', 'Col2'].

FAQs

• How do I sort in descending order? Add the argument ascending=False inside the sort_values() method call to reverse the default order.
• What if I need both conditions to be true? Use the bitwise AND operator (&) instead of the OR operator (|) to ensure both conditions are met simultaneously.
• Does filtering change the original DataFrame? No, filtering always returns a new view or copy of the data; the original DataFrame is preserved unless you explicitly reassign it.