# Token profiles and budget packing

Packing converts a frame into the items that are eligible and affordable for one model call. It also creates a `PackManifest` explaining the decision.

## Processing order in the 0.1 series

1. Evaluate expiry, quarantine, boundary rules, and custom policies.
2. Exclude optional/preferred violations or raise `PolicyViolation` for a required violation.
3. Exclude optional/preferred Binary content or raise `UnsupportedContent` for required Binary content.
4. Count eligible item costs using the selected `TokenProfile`.
5. Reserve every required item.
6. If required content exceeds the budget, apply the explicitly selected overflow strategy.
7. Consider preferred, then optional items by rank score, priority, score-to-cost density, and ID.
8. Return selected items in their original input order.

The token-profile base cost is counted once. Each item's content cost plus its configured envelope cost is then charged against the budget.

## Retention is a contract

- `required`: must be selected or packing fails.
- `preferred`: considered before every optional item, but can be excluded for policy or budget reasons.
- `optional`: selected from the remaining capacity.

High priority never overrides a `required` item, and a high-priority optional item does not jump ahead of a preferred item.

## Defaults

```python
pack = frame.pack(budget=8_000)
```

is equivalent to choosing:

```python
pack = frame.pack(
    budget=8_000,
    token_profile=cx.profiles.approximate(),
    boundary=cx.Boundary.internal("application"),
    overflow=cx.Overflow.error(),
)
```

Use an explicit external boundary before rendering data for a model provider.

## Token profiles

| Profile | 0.1-series behavior | Manifest |
| --- | --- | --- |
| `profiles.approximate()` | At least one token per item, estimated from UTF-8 bytes | `estimated=True` |
| `profiles.openai("o200k_base")` | Rust-native content count in binary wheels; conservative base and per-item envelope | `estimated=True` |
| `profiles.openai("cl100k_base")` | Same behavior for `cl100k_base` | `estimated=True` |
| `profiles.huggingface(tokenizer_json)` | Rust `tokenizers` count from a tokenizer JSON definition | `estimated=False` for the configured profile |
| `profiles.huggingface(python_tokenizer)` | Batched Python fallback with a first-use `PerformanceWarning` | Caller-defined profile |

`estimated=False` means the configured profile was counted exactly. It does not prove that the profile includes every provider-specific request field. The built-in Hugging Face profile has no provider envelope cost by default.

`profiles.openai()` deliberately remains estimated even when content tokenization is exact because provider request-envelope rules can change. Do not treat an estimated budget as a provider-enforced hard limit; leave operational headroom and handle provider errors.

Passing an encoding other than `o200k_base` or `cl100k_base` is unsupported by the 0.1-series native core.

## Inspect the result

```python
manifest = pack.manifest.to_dict()
print(manifest["budget"])
print(manifest["total_tokens"])
print(manifest["token_profile_id"])
print(manifest["estimated"])
print(manifest["selected"])
print(manifest["excluded"])
```

Never branch on a translated exception string or a formatted log line. Use manifest fields, diagnostic codes, and structured exception attributes.

## Required overflow

The safe default never removes required content. See {doc}`overflow` for a complete recovery example. Applications may use `demote()` before packing when old conversation turns are no longer mandatory; Cognoxium never makes that authorization decision automatically.