Filtering, ranking, and deduplication

Filtering

Expressions support comparisons, &, |, ~, is_null(), and is_in(). Parenthesize each comparison when combining expressions because Python operator precedence still applies.

eligible = frame.filter(
    (cx.col("priority") >= 5)
    & cx.col("trust").is_in({cx.Trust.TRUSTED, cx.Trust.UNTRUSTED})
)

cx.now() captures one UTC time when the expression is created, making every row in that execution compare against the same instant.

Ranking

The built-in lexical ranker case-folds text, extracts \w+ terms, and scores the fraction of distinct query terms present in a Text or JSON payload. It does not use embeddings, language models, stemming, or network calls.

rank() stores relevance without reordering records. Packing uses the score when allocating preferred and optional capacity, then returns selected items in original conversational order.

For semantic or domain-specific relevance, implement the batched Ranker protocol shown in Writing plugins.

Exact deduplication

dedupe() groups identical canonical content_hash values. It is not semantic deduplication.

The representative is the oldest created_at, then the lexicographically smallest ID. Duplicate groups merge conservatively:

Field	Merge rule
sources	Deduplicated deterministic order
merged_from_ids	Every original and previously merged ID
parent_ids	Union
trust	quarantined over untrusted over trusted
sensitivity	Most restrictive
retention	required over preferred over optional
expires_at	Earliest finite expiry
priority	Maximum
truncatable	True only when every duplicate is truncatable
min_tokens	Maximum
Metadata conflict	Representative value retained; all conflicting values recorded in the manifest

Single-item groups are unchanged. A merged group is attached to the eventual PackManifest.dedupe_groups; use that manifest rather than relying on underscore-prefixed internal metadata.

See Frame operations for the complete tabular operation guide and Lineage and manifests for audit examples.