Managing Memory and Copies in Pandas 3
What you'll learn: How to modify DataFrames correctly under Copy-on-Write and measure actual memory usage rather than relying on inplace=True guarantees.
Copy-on-Write: Logical vs. Physical Copies
Pandas 3.0 uses Copy-on-Write (CoW) by default. When you subset a DataFrame, pandas creates a logical copy. It behaves entirely independently of the original, meaning modifying the subset will never alter the parent DataFrame.
However, under the hood, pandas delays creating a physical copy in memory. The subset shares the same memory buffers with the parent DataFrame until you attempt to modify it. Only upon writing does pandas physically allocate new memory and copy the data.
Updating Data Properly
Because derived objects act as independent copies, "chained assignment" will only modify the temporary subset, leaving the parent unchanged. To modify the parent DataFrame directly, always use a single .loc operation:
import pandas as pd
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
# Under CoW, modifying a subset does NOT modify the parent
subset = df["A"]
subset.iloc[0] = 99 # 'df' remains completely unchanged
# Correct way to modify the parent directly in a single step
df.loc[0, "A"] = 99
The Reality of inplace=True
Many pandas methods offer an inplace=True parameter (like df.drop(columns=["B"], inplace=True)). While it remains part of the API and is not deprecated, it does not guarantee lower physical memory allocation. In many cases, pandas still creates intermediate copies under the hood to safely complete the operation.
Measuring True Memory Impact
It is common to see df.memory_usage(deep=True) used to assess memory, but this only reports the size of the objects themselves. It does not reflect peak process memory or tell you whether buffers are being physically shared across DataFrames. To accurately measure whether operations (including inplace=True) are actually saving memory, you must profile allocation directly using tools like Python's built-in tracemalloc:
import tracemalloc
# Start tracking memory allocations
tracemalloc.start()
# Perform an operation
df.drop(columns=["B"], inplace=True)
# Check the peak memory usage during the operation
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"Peak memory usage: {peak} bytes")
Key Takeaway: Pandas 3 Copy-on-Write shares memory automatically until a write occurs; to update data safely, use single .loc assignments on the parent, and use profiling tools like tracemalloc rather than assuming inplace=True guarantees memory efficiency.