Token profiles and budget packing

Packing converts a frame into the items that are eligible and affordable for one model call. It also creates a PackManifest explaining the decision.

Processing order in the 0.1 series

  1. Evaluate expiry, quarantine, boundary rules, and custom policies.

  2. Exclude optional/preferred violations or raise PolicyViolation for a required violation.

  3. Exclude optional/preferred Binary content or raise UnsupportedContent for required Binary content.

  4. Count eligible item costs using the selected TokenProfile.

  5. Reserve every required item.

  6. If required content exceeds the budget, apply the explicitly selected overflow strategy.

  7. Consider preferred, then optional items by rank score, priority, score-to-cost density, and ID.

  8. Return selected items in their original input order.

The token-profile base cost is counted once. Each item’s content cost plus its configured envelope cost is then charged against the budget.

Retention is a contract

  • required: must be selected or packing fails.

  • preferred: considered before every optional item, but can be excluded for policy or budget reasons.

  • optional: selected from the remaining capacity.

High priority never overrides a required item, and a high-priority optional item does not jump ahead of a preferred item.

Defaults

pack = frame.pack(budget=8_000)

is equivalent to choosing:

pack = frame.pack(
    budget=8_000,
    token_profile=cx.profiles.approximate(),
    boundary=cx.Boundary.internal("application"),
    overflow=cx.Overflow.error(),
)

Use an explicit external boundary before rendering data for a model provider.

Token profiles

Profile

0.1-series behavior

Manifest

profiles.approximate()

At least one token per item, estimated from UTF-8 bytes

estimated=True

profiles.openai("o200k_base")

Rust-native content count in binary wheels; conservative base and per-item envelope

estimated=True

profiles.openai("cl100k_base")

Same behavior for cl100k_base

estimated=True

profiles.huggingface(tokenizer_json)

Rust tokenizers count from a tokenizer JSON definition

estimated=False for the configured profile

profiles.huggingface(python_tokenizer)

Batched Python fallback with a first-use PerformanceWarning

Caller-defined profile

estimated=False means the configured profile was counted exactly. It does not prove that the profile includes every provider-specific request field. The built-in Hugging Face profile has no provider envelope cost by default.

profiles.openai() deliberately remains estimated even when content tokenization is exact because provider request-envelope rules can change. Do not treat an estimated budget as a provider-enforced hard limit; leave operational headroom and handle provider errors.

Passing an encoding other than o200k_base or cl100k_base is unsupported by the 0.1-series native core.

Inspect the result

manifest = pack.manifest.to_dict()
print(manifest["budget"])
print(manifest["total_tokens"])
print(manifest["token_profile_id"])
print(manifest["estimated"])
print(manifest["selected"])
print(manifest["excluded"])

Never branch on a translated exception string or a formatted log line. Use manifest fields, diagnostic codes, and structured exception attributes.

Required overflow

The safe default never removes required content. See Overflow recovery for a complete recovery example. Applications may use demote() before packing when old conversation turns are no longer mandatory; Cognoxium never makes that authorization decision automatically.