Configuration reference¶

Module: piighost.config

A configuration file describes a whole pipeline declaratively. piighost reads it as TOML or JSON, chosen by the file suffix, validates it with Pydantic, and builds the pipeline the file describes. This page documents every section and every component type.

from piighost.config import load_config, load_pipeline, load_thread_pipeline

The config extra is required (pip install piighost[config]), which pulls in pydantic-settings. Unknown keys are rejected, so a typo fails validation rather than being ignored. A component type can need an extra of its own, named in the Extra column of the table that documents it.

Entry points¶

load_config parses and validates a file into a PipelineConfig without building any component, so no model loads. load_pipeline builds a stateless AnonymizationPipeline and raises ConfigError if the file declares a [memory] section, since a memory describes a thread pipeline. load_thread_pipeline builds a ThreadAnonymizationPipeline and raises ConfigError if the file declares no [memory] section.

from piighost.config import load_pipeline, load_thread_pipeline

stateless = load_pipeline("pipeline.toml")       # no [memory]
thread = load_thread_pipeline("thread.toml")     # has [memory]

File format¶

The suffix picks the parser. A .json suffix is read as JSON, compared without regard to case, and anything else as TOML. The two formats carry the same schema. A section is a TOML table or a JSON object.

[detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

[linker]
type = "exact"

[anonymizer.placeholder]
type = "redact"

{
  "detector": { "type": "regex", "patterns": { "EMAIL": "[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}" } },
  "linker": { "type": "exact" },
  "anonymizer": { "placeholder": { "type": "redact" } }
}

Environment overrides¶

Every top-level key accepts an override from an environment variable prefixed PIIGHOST_, whether it holds a scalar or a whole section. PIIGHOST_NAME overrides the name scalar, and PIIGHOST_DETECTOR overrides the [detector] section with a JSON object, rejected as a validation error when it is not valid JSON. Overrides layer above the file key by key, so an environment value wins over the file value and the keys it leaves out keep theirs.

export PIIGHOST_NAME="local-en"
export PIIGHOST_DETECTOR='{"type": "exact", "values": {"Patrick": "PERSON"}}'

No nested delimiter is configured, so a variable such as PIIGHOST_DETECTOR__TYPE names no field, and it is ignored without an error rather than reaching the type key. A section is overridden by its JSON object only.

Secrets are never read from the file. Each is read from its own environment variable at build time, and a missing one raises ConfigError from build().

Sections¶

The top-level keys of a PipelineConfig.

[detector]¶

Discriminated on type. Required.

type = "regex"¶

Matches PII by one regex per label, pulled from inline patterns, named catalogs, or both. Catalogs merge first, then inline patterns, so an inline pattern overrides a catalog pattern on the same label. At least one inline pattern or one catalog is required. Each pattern is validated as a compilable regex at load time, then compiled under re.ASCII, so \d matches 0-9 and a shape class stops at the first non-ASCII character. A value such as prénom@corp.com is therefore matched from nom onwards.

[detector]
type = "regex"
catalogs = ["generic", "fr"]
patterns = { EMPLOYEE_ID = 'EMP-[0-9]{4}' }

type = "composite"¶

Runs child detectors together and merges their detections.

[detector]
type = "composite"

[[detector.detectors]]
type = "regex"
catalogs = ["generic"]

[[detector.detectors]]
type = "exact"
values = { Patrick = "PERSON" }

type = "exact"¶

Finds occurrences of literal values, each mapped to a label.

[detector]
type = "exact"
values = { Patrick = "PERSON", Lyon = "LOCATION" }

type = "chunked"¶

Wraps a detector with a splitter that cuts long text into overlapping chunks.

[detector]
type = "chunked"
chunk_size = 2000
chunk_overlap = 200

[detector.detector]
type = "spacy"
model = "en_core_web_sm"

Model-backed detectors¶

Each needs its own extra, and every one but presidio needs a model. labels accepts a list or an {emitted: internal} map. max_concurrency caps concurrent inferences, or None for unbounded.

[detector]
type = "gliner2"
model = "fastino/gliner2-multi-v1"
labels = ["PERSON", "LOCATION"]
threshold = 0.5

The transformers detector passes aggregation_strategy to its token-classification pipeline, which groups sub-word tokens into whole entities.

The presidio detector takes no model key, since the config path builds Presidio's default English AnalyzerEngine with its default recognizers. Another language, a custom recognizer, or a custom NLP engine is the programmatic path, constructing the engine and passing it to PresidioDetector.

The llm detector reads its provider credential from the provider's own environment variable, never from the file.

[linker]¶

Optional. Defaults to ExactEntityLinker. One linker exists, so type names it rather than discriminating a union.

[linker]
type = "exact"

[anonymizer]¶

Optional. Defaults to an Anonymizer with a label-counter factory. When present it carries one [anonymizer.placeholder] table selecting the placeholder factory, discriminated on type.

[anonymizer.placeholder]
type = "label_counter"

The middleware needs a delimited factory, so redact, label, label_counter, or label_hash. The mask factory produces P***, which keeps no delimiters and has no recognizer.

[overlap_resolver]¶

Optional in the file, but the stage runs either way. Omitting the section builds a ConfidenceOverlapResolver, and there is no supported way to disable the stage, since the render stage assumes disjoint spans. One resolver exists, so type names it rather than discriminating a union.

[overlap_resolver]
type = "confidence"

[expander]¶

Optional, and disabled when omitted. One expander exists, so type names it rather than discriminating a union.

[expander]
type = "word_boundary"
case_sensitive = false

[entity_resolver]¶

Optional. Discriminated on type.

[entity_resolver]
type = "fuzzy"
threshold = 0.85

[guard]¶

Optional. Discriminated on type. Re-checks the de-identified output for residual PII and refuses it when PII remains.

type = "detector"¶

Re-runs a detector on the output. Carries a nested [guard.detector] config.

[guard]
type = "detector"

[guard.detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

type = "llm"¶

Prompts a chat model to find residual PII.

type = "moderation"¶

Scores the output with a Mistral moderation model. The credential is read from MISTRAL_API_KEY at build time, and build() raises ConfigError when it is unset.

[override]¶

Optional. Forces detections through a whitelist and vetoes them through a blacklist. Each list is a detector config, [override.whitelist] and [override.blacklist], and both are optional.

[override]
blacklist_strategy = "value"

[override.whitelist]
type = "regex"
patterns = { CODENAME = 'ACME-[A-Z]+' }

[override.blacklist]
type = "exact"
values = { "public@corp.com" = "EMAIL" }

[observation_redactor]¶

Optional. A placeholder factory config, same type values as [anonymizer.placeholder], redacting the payloads sent to a tracing backend so a trace holds tokens, not raw values.

[observation_redactor]
type = "label"

Omitting the section traces the clear text and the detection values, and a live tracer then emits a PIIGhostSecurityWarning. The pipeline's trace_clear_text flag, which silences that warning, has no key in a configuration file, so a file-built pipeline cannot acknowledge clear-text tracing. Passing trace_clear_text=True to the pipeline is the programmatic path.

[memory]¶

Optional. Its presence makes the pipeline a ThreadAnonymizationPipeline keeping per-thread state. Discriminated on type.

The token_memo_ttl scalar goes with it, at the top level rather than in this section, since it bounds the pipeline's own memoized token map and not the store. Setting it without a [memory] raises, because a stateless pipeline memoizes nothing. Why it matters on a multi-worker deployment is in Multi-instance deployment.

type = "in_memory"¶

A process-local store, lost on restart and not shared across workers.

[memory]
type = "in_memory"

type = "redis"¶

A persistent, multi-worker store, optionally keying each stored message with a hasher and encrypting each stored value with a cipher.

Configure both [memory.hasher] and [memory.cipher], or neither. With neither, the backend stores the mapping in clear and warns. With exactly one, build() raises ConfigError.

The hasher, [memory.hasher], is discriminated on type.

The cipher, [memory.cipher], has one type.

The hasher reads its pepper from PIIGHOST_HASH_PEPPER and the cipher reads its base64 key from PIIGHOST_CIPHER_KEY, both at build time. A missing or malformed value raises ConfigError.

[memory]
type = "redis"
url = "redis://localhost:6379/0"
namespace = "piighost"
ttl = 3600

[memory.hasher]
type = "argon2"

[memory.cipher]
type = "aesgcm"

type = "sqlalchemy"¶

A durable, multi-worker store backed by any SQLAlchemy-supported database (SQLite, PostgreSQL, ...). It reads the database URL from an environment variable rather than the config file, so the URL and its password stay out of version control. An optional hasher and cipher protect the stored values exactly as they do for Redis.

Configure both [memory.hasher] and [memory.cipher], or neither, exactly as for Redis. With neither, the backend stores the mapping in clear and warns. With exactly one, build() raises ConfigError.

The URL must use an async driver, for example postgresql+asyncpg://... or sqlite+aiosqlite://.... A missing environment variable raises ConfigError at build time. Call await memory.create_schema() once at startup to create the table.

[memory]
type = "sqlalchemy"
url_env = "PIIGHOST_DATABASE_URL"
table_name = "piighost_conversation_messages"

[memory.hasher]
type = "argon2"

[memory.cipher]
type = "aesgcm"

Full example¶

The keys of examples/config/pipeline.toml, a stateless pipeline pulling a catalog, adding one inline pattern, and enabling several optional stages. The file itself carries the same keys with a comment on each stage.

[detector]
type = "regex"
catalogs = ["generic"]
patterns = { EMPLOYEE_ID = 'EMP-[0-9]{4}' }

[overlap_resolver]
type = "confidence"

[expander]
type = "word_boundary"

[entity_resolver]
type = "fuzzy"
threshold = 0.85

[linker]
type = "exact"

[anonymizer.placeholder]
type = "label_counter"

[override.whitelist]
type = "regex"
patterns = { CODENAME = 'ACME-[A-Z]+' }

[guard]
type = "detector"

[guard.detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

[observation_redactor]
type = "label"

The same content in JSON, chosen by a .json suffix, is equivalent. A table becomes an object, an inline table becomes a nested object, and an array of tables becomes an array of objects.

Errors¶

ConfigFileError and ConfigValidationError are subclasses of ConfigError, so catching ConfigError covers all three. The classes live in piighost.exceptions, so a caller can catch them without the config extra.

See also¶

• examples/config/ in the repository for six runnable files, detector_only.toml, minimal.toml, minimal.json, pipeline.toml, thread_redis.toml and thread_sqlalchemy.toml, all six loaded by examples/config/run.py.
• Command-line interface for validating a file from the shell.
• Detectors reference for the detector each type builds.
• LangChain middleware reference for driving a thread pipeline in an agent.