Configuration file¶
You will describe a whole pipeline in a TOML file, growing it from three lines to a conversational pipeline that keeps a token stable across the turns of a conversation. Each step changes one thing in the file, then you check the file and run it to see what changed.
Prerequisites
piighost installed with the config extra, pip install "piighost[config]", see Installation. Every step runs without a model and without network access. Step 6 adds the fuzzy extra.
1. Set up the check loop¶
Two commands drive every step below. Start with a pipeline.toml that is wrong on purpose, with pattern where the schema expects patterns.
Validate it.
The output should be:
invalid configuration in pipeline.toml: 1 validation error for PipelineConfig
detector.regex.pattern
Extra inputs are not permitted [type=extra_forbidden, input_value={'EMAIL': '[a-z0-9._%+-]+...a-z0-9.-]+\\.[a-z]{2,}'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
The command names the section and the key it choked on, and exits 1, which also makes it a CI gate. Run it after every edit below. It builds no component, so it loads no model.
Dump the schema once and point your editor at it for completion on the section and key names.
Both commands are documented in the command-line interface.
2. Build a pipeline from three lines¶
Fix the key, patterns with an s. The file now carries one section, and that is enough to build a pipeline.
Write run.py next to it. It loads the file and de-identifies the text you pass on the command line, and every later step reuses it unchanged.
import asyncio
import sys
from piighost.config import load_pipeline
async def main() -> None:
pipeline = load_pipeline("pipeline.toml")
result = await pipeline.anonymize(sys.argv[1])
print(result.text)
asyncio.run(main())
The output should be:
The token names the label and numbers it although the file declares no anonymizer, and both occurrences of one address would share that token although the file declares no linker. Each of those two stages falls back to its default, and so does overlap resolution. This file is on disk as examples/config/detector_only.toml.
3. Pick the token¶
Ask for a plain redaction instead of the numbered token, with an [anonymizer.placeholder] section.
[detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }
[anonymizer.placeholder]
type = "redact"
The output should be:
The address is gone and its label with it. examples/config/minimal.toml carries this file with the default linker written out, and examples/config/minimal.json carries it in JSON, the suffix picking the parser. The configuration reference lists every token style.
4. Pull a prebuilt catalog¶
Your pattern covers email only, so the IP address in the sample text went through in clear. Replace the inline pattern with the generic catalog, which carries email, URL, IPv4 and credit card. Four labels now reach the anonymizer, so put the numbered token back to tell them apart.
The output should be:
The IP address is covered now, and prénom@corp.com only half of it. The catalog patterns match ASCII shapes, so the match starts after the accent. Add an inline pattern on the same label, and it overrides the catalog's.
[detector]
type = "regex"
catalogs = ["generic"]
patterns = { EMAIL = '[A-Za-zÀ-ÿ0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' }
[anonymizer.placeholder]
type = "label_counter"
The output should be:
The whole address is a token now. Catalogs merge first, then your inline patterns, so a label declared in both takes your pattern.
5. Run two detectors at once¶
The catalog matches formats, and a first name has no format. Declare the names you already know in a second detector, and let a composite detector run both and merge what they return.
[detector]
type = "composite"
[[detector.detectors]]
type = "regex"
catalogs = ["generic"]
patterns = { EMAIL = '[A-Za-zÀ-ÿ0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' }
[[detector.detectors]]
type = "exact"
values = { Patrick = "PERSON", Patrik = "PERSON" }
[anonymizer.placeholder]
type = "label_counter"
The output should be:
The names and the formats are caught in one pass. One person spelled two ways still gets two tokens, <<PERSON:1>> and <<PERSON:2>>, which the next step settles.
6. Merge the near-duplicate entities¶
Patrick and Patrik are the same person, and a model reading two tokens follows two people. Install the fuzzy extra.
Append an [entity_resolver] section to the file, which clusters the entities whose values are close enough to each other.
The output should be:
Both spellings share <<PERSON:1>>. Drop the section and the stage is gone again, as it is for every optional stage.
7. Keep the tokens across a conversation¶
Each run of run.py restarts the numbering, since the pipeline keeps nothing from one call to the next. Append a [memory] section, which gives it a per-thread store and changes the loader you call.
The file is valid, and run.py now refuses it.
The traceback ends on:
A file carrying a memory describes a thread pipeline, so it takes load_thread_pipeline. Write thread.py, which sends two messages on the thread "thread-42".
import asyncio
from piighost.config import load_thread_pipeline
async def main() -> None:
pipeline = load_thread_pipeline("pipeline.toml")
first = await pipeline.anonymize("Patrick writes to alice@corp.com.", "thread-42")
print(first.text)
second = await pipeline.anonymize("Patrik answers from 10.0.0.7.", "thread-42")
print(second.text)
asyncio.run(main())
The output should be:
The second message reuses the <<PERSON:1>> assigned by the first. The two loaders refuse each other's files, so load_thread_pipeline on a file without a memory raises this configuration declares no memory; use load_pipeline.
What's next¶
- Configuration reference for every section, every
typeand every key. - Deploy a production pipeline for a memory shared between workers, Redis or a SQL database, with the stored values encrypted at rest. The two files are
examples/config/thread_redis.tomlandexamples/config/thread_sqlalchemy.toml. - Force a detection or keep a value in clear for the whitelist and the blacklist.