Installation¶

Recommended: uv 0.12.1¶

The uv workflow documented for Cognoxium 0.1.1 requires uv 0.12.1. The pip workflow below does not require uv. Add Cognoxium to an existing uv project:

uv add cognoxium

For a disposable project:

uv init cognoxium-demo
cd cognoxium-demo
uv add cognoxium
uv run python -c "import cognoxium as cx; print(cx.__version__)"

Expected output for this documentation version:

0.1.1

uv manages the project environment; it is not a runtime dependency of Cognoxium.

pip¶

Standard Python environments remain supported:

python -m pip install cognoxium
python -c "import cognoxium as cx; print(cx.__version__)"

Optional integrations¶

The core package has no mandatory Python dependencies. Install only the interoperability layer you need.

Extra	Enables	uv	pip
`arrow`	Arrow conversion, IPC, and Parquet	`uv add "cognoxium[arrow]"`	`python -m pip install "cognoxium[arrow]"`
`pandas`	`to_pandas()`	`uv add "cognoxium[pandas]"`	`python -m pip install "cognoxium[pandas]"`
`polars`	`to_polars()`	`uv add "cognoxium[polars]"`	`python -m pip install "cognoxium[polars]"`
`tokenizers`	Python-side tokenizer interoperability	`uv add "cognoxium[tokenizers]"`	`python -m pip install "cognoxium[tokenizers]"`

Binary wheels include the Rust extension. The Python tokenizers extra is not required for the built-in Rust o200k_base and cl100k_base paths.

Supported wheels¶

Cognoxium 0.1.1 supports CPython 3.10 through 3.14. The release workflow builds wheels for:

Linux x86-64 and ARM64
Windows x86-64
macOS x86-64 and Apple Silicon

On another platform, the installer may fall back to the source distribution. A source build requires a current stable Rust toolchain and maturin.

Development checkout¶

Clone the repository, reproduce the locked Python environment, and build the PyO3 extension:

git clone https://github.com/Nue-Japan/cognoxium.git
cd cognoxium
uv sync --extra dev
uv run maturin develop

The repository commits both Cargo.lock and the cross-platform uv.lock. Use uv lock --check and uv sync --locked --extra dev to fail when project metadata and the reviewed lockfile differ.

Run the principal checks:

uv run pytest
uv run ruff check python tests scripts
uv run mypy
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
uv run sphinx-build -W -b html docs docs/_build/en/latest
uv run sphinx-build -W -b html -D language=ja docs docs/_build/ja/latest

uv does not install Rust. Install Rust separately before maturin develop or any Cargo command.