所有的代码分析基于 polars 1.43.0 版本
分析polars的Series
Series 是 Polars 的核心列式数据结构,上接 Python API,下接 Arrow 内存格式和 ChunkedArray。本文分析其从 Python 到 Rust 的完整架构和设计亮点。
架构总览
1. Python 层:用户入口
Python 的 Series 类定义在:
展开折叠代码 (173-253 行,共 81 行)
@expr_dispatch
class Series:
"""
A Series represents a single column in a Polars DataFrame.
Parameters
----------
name : str, default None
Name of the Series. Will be used as a column name when used in a DataFrame.
When not specified, name is set to an empty string.
values : ArrayLike, default None
One-dimensional data in various forms. Supported are: Sequence, Series,
pyarrow Array, and numpy ndarray.
dtype : DataType, default None
Data type of the resulting Series. If set to `None` (default), the data type is
inferred from the `values` input. The strategy for data type inference depends
on the `strict` parameter:
- If `strict` is set to True (default), the inferred data type is equal to the
first non-null value, or `Null` if all values are null.
- If `strict` is set to False, the inferred data type is the supertype of the
values, or :class:`Object` if no supertype can be found. **WARNING**: A full
pass over the values is required to determine the supertype.
- If no values were passed, the resulting data type is :class:`Null`.
strict : bool, default True
Throw an error if any value does not exactly match the given or inferred data
type. If set to `False`, values that do not match the data type are cast to
that data type or, if casting is not possible, set to null instead.
nan_to_null : bool, default False
In case a numpy array is used to create this Series, indicate how to deal
with np.nan values. (This parameter is a no-op on non-numpy data).
Examples
--------
Constructing a Series by specifying name and values positionally:
>>> s = pl.Series("a", [1, 2, 3])
>>> s
shape: (3,)
Series: 'a' [i64]
[
1
2
3
]
Notice that the dtype is automatically inferred as a polars Int64:
>>> s.dtype
Int64
Constructing a Series with a specific dtype:
>>> s2 = pl.Series("a", [1, 2, 3], dtype=pl.Float32)
>>> s2
shape: (3,)
Series: 'a' [f32]
[
1.0
2.0
3.0
]
It is possible to construct a Series with values as the first positional argument.
This syntax considered an anti-pattern, but it can be useful in certain
scenarios. You must specify any other arguments through keywords.
>>> s3 = pl.Series([1, 2, 3])
>>> s3
shape: (3,)
Series: '' [i64]
[
1
2
3
]
"""
# NOTE: This `= None` is needed to generate the docs with sphinx_accessor.
_s: PySeries = None # type: ignore[assignment]关键设计:
_s: PySeries— 内部持有的是 Rust 侧的PySeries对象(通过 PyO3 构建),Python 本身不存数据__init__多路分发 — 根据values类型(Sequence、numpy.ndarray、pa.Array、torch.Tensor等)走不同的*_to_pyseries构造路径,最终都调用 Rust 端的PySeries::new@expr_dispatch装饰器 — 大部分方法被expr_dispatch包装,使得Series和Expr共享同一套接口逻辑
2. Python ↔ Rust 桥接
#[pyclass(frozen, from_py_object)]
#[repr(transparent)]
pub struct PySeries {
pub series: RwLock<Series>,
}
impl Clone for PySeries {
fn clone(&self) -> Self {
Self {
series: RwLock::new(self.series.read().clone()),
}
}
}
impl From<Series> for PySeries {
fn from(series: Series) -> Self {
Self::new(series)
}
}
impl PySeries {
pub(crate) fn new(series: Series) -> Self {
PySeries {
series: RwLock::new(series),
}
}
}PySeries 是对 Rust Series 的简单封装:
#[pyclass(frozen)]— PyO3 标记,表示 Python 不可变(实际内部用RwLock支持 Rust 层可变)series: RwLock<Series>— 读写锁,Python GIL + RwLock 保证线程安全
ToSeries / ToPySeries trait 提供 Vec<PySeries> 和 Vec<Series> 之间的双向转换,简化批量操作。
3. Rust 核心:Series + SeriesTrait
3.1 Series 定义
#[derive(Clone)]
#[must_use]
pub struct Series(pub Arc<dyn SeriesTrait>);核心设计模式:类型擦除 + 虚方法调用 (trait object)
Series 本身只是一个 Arc<dyn SeriesTrait> 的包装,不感知具体类型。所有操作通过 SeriesTrait 的虚函数分派到具体实现。
3.2 Deref 到 SeriesTrait
impl Deref for Series {
type Target = dyn SeriesTrait;
fn deref(&self) -> &Self::Target {
self.0.as_ref()
}
}通过 Deref,Series 可以直接调用 SeriesTrait 的所有方法(如 filter、take、cast 等),无需手动解引用。
3.3 SeriesTrait
pub trait SeriesTrait:
Send + Sync + private::PrivateSeries + private::PrivateSeriesNumeric
{SeriesTrait 约定了所有类型必须实现的操作接口:
| 类别 | 方法 |
|---|---|
| 元数据 | name()、dtype()、len()、null_count()、chunks() |
| 切片/过滤 | slice()、filter()、take()、head()、tail() |
| 排序/唯一 | sort_with()、unique()、arg_unique() |
| 聚合 | sum_reduce()、min_reduce()、max_reduce()、mean_reduce() |
| 类型转换 | cast()、to_physical_repr() |
| 空值处理 | is_null()、is_not_null()、drop_nulls() |
| 遍历 | get()、get_unchecked() |
PrivateSeries 子 trait 提供内部使用的操作(agg、hash、arithmetic 等),对外部隐藏。
设计亮点:所有方法都有默认实现(polars_bail! 或返回全 Null),每个类型只需覆盖自己支持的操作。新类型加入时,只需实现需要的方法,其余自动 fallback。
4. 具体类型的实现
4.1 SeriesWrap + impl_dyn_series! 宏
#[repr(transparent)]
pub(crate) struct SeriesWrap<T>(pub T);
impl<T: PolarsDataType> From<ChunkedArray<T>> for SeriesWrap<ChunkedArray<T>> {
fn from(ca: ChunkedArray<T>) -> Self {
SeriesWrap(ca)
}
}SeriesWrap<T> 是一个 #[repr(transparent)] 的新类型包装,所有 ChunkedArray<T> 通过它来实现 SeriesTrait。
impl_dyn_series! 宏为每种物理类型生成 PrivateSeries + SeriesTrait 的实现,包括:
impl_dyn_series!(Int32Chunked, Int32Type);
impl_dyn_series!(UInt32Chunked, UInt32Type);
impl_dyn_series!(Float64Chunked, Float64Type);
impl_dyn_series!(BooleanChunked, BooleanType);
// ...等
核心方法实现模式——委派给 ChunkedArray 的同名方法:
fn filter(&self, filter: &BooleanChunked) -> PolarsResult<Series> {
ChunkFilter::filter(&self.0, filter).map(|ca| ca.into_series())
}
fn sort_with(&self, options: SortOptions) -> PolarsResult<Series> {
Ok(ChunkSort::sort_with(&self.0, options).into_series())
}
4.2 浮点类型的独立实现
浮动类型(Float16/Float32/Float64)的 impl_dyn_series! 在 floats.rs 而非 mod.rs,因为它们的 mean_reduce 和 median_reduce 返回原地类型而非 Float64:
// mod.rs (整数类型):
fn mean_reduce(&self) -> PolarsResult<Scalar> {
Ok(Scalar::new(DataType::Float64, self.mean().into()))
}
// floats.rs (浮点类型):
fn mean_reduce(&self) -> PolarsResult<Scalar> {
let mean = self.mean().map(AsPrimitive::as_);
Ok(Scalar::new(self.dtype().clone(), mean.into()))
}
4.3 Boolean 的独立实现
Boolean 的算术操作只有 add_to(因为布尔值支持按位逻辑运算),且 and_reduce/or_reduce/xor_reduce 使用了 polars_compute::bitwise::BitwiseKernel 进行位运算聚合。
4.4 逻辑类型的实现(Datetime 示例)
对于 DatetimeChunked 这样的逻辑类型,实现模式是所有操作在物理层完成,再封装回逻辑类型:
fn filter(&self, filter: &BooleanChunked) -> PolarsResult<Series> {
self.0.physical().filter(filter).map(|ca| {
ca.into_datetime(self.0.time_unit(), self.0.time_zone().clone())
.into_series()
})
}
算术操作也遵循类型语义:
Datetime - Datetime = DurationDatetime ± Duration = Datetime
fn subtract(&self, rhs: &Series) -> PolarsResult<Series> {
match (self.dtype(), rhs.dtype()) {
(DataType::Datetime(tu, tz), DataType::Datetime(tur, tzr)) => {
assert_eq!(tu, tur);
assert_eq!(tz, tzr);
let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
let rhs = rhs.cast(&DataType::Int64).unwrap();
Ok(lhs.subtract(&rhs)?.into_duration(*tu).into_series())
},
(DataType::Datetime(tu, tz), DataType::Duration(tur)) => {
assert_eq!(tu, tur);
let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
let rhs = rhs.cast(&DataType::Int64).unwrap();
Ok(lhs
.subtract(&rhs)?
.into_datetime(*tu, tz.clone())
.into_series())
},
(dtl, dtr) => polars_bail!(opq = sub, dtl, dtr),
}
}
fn add_to(&self, rhs: &Series) -> PolarsResult<Series> {展开折叠代码 (134-146 行,共 13 行)
match (self.dtype(), rhs.dtype()) {
(DataType::Datetime(tu, tz), DataType::Duration(tur)) => {
assert_eq!(tu, tur);
let lhs = self.cast(&DataType::Int64, CastOptions::NonStrict).unwrap();
let rhs = rhs.cast(&DataType::Int64).unwrap();
Ok(lhs
.add_to(&rhs)?
.into_datetime(*tu, tz.clone())
.into_series())
},
(dtl, dtr) => polars_bail!(opq = add, dtl, dtr),
}
}4.5 NullChunked — 特殊的独立实现
NullChunked 不是一个 SeriesWrap<ChunkedArray<T>>,而是一个独立的结构体,直接实现 SeriesTrait:
#[derive(Clone)]
pub struct NullChunked {
pub(crate) name: PlSmallStr,
length: usize,
// we still need chunks as many series consumers expect
// chunks to be there
chunks: Vec<ArrayRef>,
}所有操作都是空操作或返回空结果:
take→ 返回等长的 Null Seriesfilter→ 计算保留的数量cast→ 调用Series::full_nullsort→ 返回自身
5. 从 Arrow 数据构造 Series
pub unsafe fn from_chunks_and_dtype_unchecked(
name: PlSmallStr,
chunks: Vec<ArrayRef>,
dtype: &DataType,
) -> Self {
use DataType::*;
match dtype {
Int8 => Int8Chunked::from_chunks(name, chunks).into_series(),
Int16 => Int16Chunked::from_chunks(name, chunks).into_series(),
Int32 => Int32Chunked::from_chunks(name, chunks).into_series(),
Int64 => Int64Chunked::from_chunks(name, chunks).into_series(),
UInt8 => UInt8Chunked::from_chunks(name, chunks).into_series(),
UInt16 => UInt16Chunked::from_chunks(name, chunks).into_series(),
UInt32 => UInt32Chunked::from_chunks(name, chunks).into_series(),
UInt64 => UInt64Chunked::from_chunks(name, chunks).into_series(),展开折叠代码 (75-158 行,共 84 行)
#[cfg(feature = "dtype-i128")]
Int128 => Int128Chunked::from_chunks(name, chunks).into_series(),
#[cfg(feature = "dtype-u128")]
UInt128 => UInt128Chunked::from_chunks(name, chunks).into_series(),
#[cfg(feature = "dtype-date")]
Date => Int32Chunked::from_chunks(name, chunks)
.into_date()
.into_series(),
#[cfg(feature = "dtype-time")]
Time => Int64Chunked::from_chunks(name, chunks)
.into_time()
.into_series(),
#[cfg(feature = "dtype-duration")]
Duration(tu) => Int64Chunked::from_chunks(name, chunks)
.into_duration(*tu)
.into_series(),
#[cfg(feature = "dtype-datetime")]
Datetime(tu, tz) => Int64Chunked::from_chunks(name, chunks)
.into_datetime(*tu, tz.clone())
.into_series(),
#[cfg(feature = "dtype-decimal")]
Decimal(precision, scale) => Int128Chunked::from_chunks(name, chunks)
.into_decimal_unchecked(*precision, *scale)
.into_series(),
#[cfg(feature = "dtype-array")]
Array(_, _) => {
ArrayChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone())
.into_series()
},
List(_) => ListChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone())
.into_series(),
String => StringChunked::from_chunks(name, chunks).into_series(),
Binary => BinaryChunked::from_chunks(name, chunks).into_series(),
#[cfg(feature = "dtype-categorical")]
dt @ (Categorical(_, _) | Enum(_, _)) => {
with_match_categorical_physical_type!(dt.cat_physical().unwrap(), |$C| {
let phys = ChunkedArray::from_chunks(name, chunks);
CategoricalChunked::<$C>::from_cats_and_dtype_unchecked(phys, dt.clone()).into_series()
})
},
Boolean => BooleanChunked::from_chunks(name, chunks).into_series(),
#[cfg(feature = "dtype-f16")]
Float16 => Float16Chunked::from_chunks(name, chunks).into_series(),
Float32 => Float32Chunked::from_chunks(name, chunks).into_series(),
Float64 => Float64Chunked::from_chunks(name, chunks).into_series(),
BinaryOffset => BinaryOffsetChunked::from_chunks(name, chunks).into_series(),
#[cfg(feature = "dtype-extension")]
Extension(typ, storage) => ExtensionChunked::from_storage(
typ.clone(),
Series::from_chunks_and_dtype_unchecked(name, chunks, storage),
)
.into_series(),
#[cfg(feature = "dtype-struct")]
Struct(_) => {
let mut ca =
StructChunked::from_chunks_and_dtype_unchecked(name, chunks, dtype.clone());
StructChunked::propagate_nulls_mut(&mut ca);
ca.into_series()
},
#[cfg(feature = "object")]
Object(_) => {
if let Some(arr) = chunks[0].as_any().downcast_ref::<FixedSizeBinaryArray>() {
assert_eq!(chunks.len(), 1);
// SAFETY:
// this is highly unsafe. it will dereference a raw ptr on the heap
// make sure the ptr is allocated and from this pid
// (the pid is checked before dereference)
{
let pe = PolarsExtension::new(arr.clone());
let s = pe.get_series(&name);
pe.take_and_forget();
s
}
} else {
unsafe { get_object_builder(name, 0).from_chunks(chunks) }
}
},
Null => new_null(name, &chunks),
Unknown(_) => {
panic!("dtype is unknown; consider supplying data-types for all operations")
},
#[allow(unreachable_patterns)]
_ => unreachable!(),
} }from_chunks_and_dtype_unchecked 是系列构造的统一入口。它接受 Vec<ArrayRef> 和 DataType,据此分派到对应的 ChunkedArray::from_chunks:
- 物理类型(Int32、Float64 等)→ 直接构造
- 逻辑类型(Date、Datetime 等)→ 构造物理 ChunkedArray +
.into_date()等转换 - 嵌套类型(List、Struct)→ 递归处理
此外还提供了 _try_from_arrow_unchecked 方法,可以直接从 Arrow 的 DataType 创建 Series,自动处理 Arrow ↔ Polars 类型的映射(如 ArrowDataType::Date32 → DataType::Int32 + into_date())。
6. 设计哲学:Python 不可变 vs Rust 可变
PySeries 用了 #[pyclass(frozen)](Python 层面不可变)+ RwLock<Series>(内部可变),这和直觉相反,却是刻意的设计:
1. 跟计算模型一致
Polars 的查询引擎基于表达式 + 惰性计算,数据流本质上是函数式的:s.sort() 返回新 Series,不改原值。Python 暴露可变接口会破坏查询优化的推理(谓词下推、公共子表达式消除等)。
2. 线程安全
Rust 侧执行时会释放 GIL 做并行计算。如果 PySeries 本身可变,另一个线程通过 DataFrame 引用同一块数据就可能产生数据竞争。frozen + RwLock 让 Python 层面安全共享,Rust 内部可控地修改。
3. 跟 Arrow 内存模型对齐
Arrow 数组本身不可变,“追加” 是追加新 chunk 而非修改已有内存。Rust 侧的可变性服务于:追加 chunk、改名字、标记排序状态等元数据操作,大块数据从不原地修改。
4. 接口更干净
Python 用户不需要区分 s.sort()(原地)和 s.sorted()(返回新值),永远只有 s.sort() 返回新 Series。
一句话:对外 (Python) 承诺不可变让代码安全可预测;对内 (Rust) 保留可变能力满足实际需求。
设计亮点总结
-
类型擦除 + 虚方法分派 —
Series(pub Arc<dyn SeriesTrait>)是核心,一个统一类型包装所有具体类型,通过 trait object 实现动态分派 -
宏驱动实现 —
impl_dyn_series!宏为每种物理类型生成 ~400 行样板代码,只需一两行宏调用;特殊类型(float、boolean、datetime)覆盖宏生成的默认实现 -
物理/逻辑类型分离 — 逻辑类型(Date、Datetime、Duration)在物理类型(Int32、Int64)之上添加语义,通过
self.0.physical()获取底层数据操作,结果再封装回逻辑类型 -
NullChunked 零成本表示 — Null 类型不存储实际数据,只需要
length和chunks字段,所有操作都是 O(1) -
Arc 共享 + COW —
Series内部用Arc共享数据,_get_inner_mut在需要修改时才 clone(写时复制),保证零拷贝切片的高效性 -
Python↔Rust 零拷贝 — 数据始终在 Rust 端,Python
Series.__init__的所有路径最终都调用 Rust 的构造方法,不经过中间序列化