# Filtering, ranking, and deduplication

## Filtering

Expressions support comparisons, `&`, `|`, `~`, `is_null()`, and `is_in()`. Parenthesize each comparison when combining expressions because Python operator precedence still applies.

```python
eligible = frame.filter(
    (cx.col("priority") >= 5)
    & cx.col("trust").is_in({cx.Trust.TRUSTED, cx.Trust.UNTRUSTED})
)
```

`cx.now()` captures one UTC time when the expression is created, making every row in that execution compare against the same instant.

## Ranking

The built-in lexical ranker case-folds text, extracts `\w+` terms, and scores the fraction of distinct query terms present in a Text or JSON payload. It does not use embeddings, language models, stemming, or network calls.

`rank()` stores relevance without reordering records. Packing uses the score when allocating preferred and optional capacity, then returns selected items in original conversational order.

For semantic or domain-specific relevance, implement the batched `Ranker` protocol shown in {doc}`plugins`.

## Exact deduplication

`dedupe()` groups identical canonical `content_hash` values. It is not semantic deduplication.

The representative is the oldest `created_at`, then the lexicographically smallest ID. Duplicate groups merge conservatively:

| Field | Merge rule |
| --- | --- |
| `sources` | Deduplicated deterministic order |
| `merged_from_ids` | Every original and previously merged ID |
| `parent_ids` | Union |
| `trust` | `quarantined` over `untrusted` over `trusted` |
| `sensitivity` | Most restrictive |
| `retention` | `required` over `preferred` over `optional` |
| `expires_at` | Earliest finite expiry |
| `priority` | Maximum |
| `truncatable` | True only when every duplicate is truncatable |
| `min_tokens` | Maximum |
| Metadata conflict | Representative value retained; all conflicting values recorded in the manifest |

Single-item groups are unchanged. A merged group is attached to the eventual `PackManifest.dedupe_groups`; use that manifest rather than relying on underscore-prefixed internal metadata.

See {doc}`operations` for the complete tabular operation guide and {doc}`lineage` for audit examples.