Detectors reference¶

Module: piighost.components.detector

A detector is the detect stage of a pipeline. It reads a text and returns the PII it finds. Every detector satisfies the AnyDetector port and returns a list of Detection, whatever backend it wraps.

from piighost.components.detector import (
    ChunkedDetector,
    CompositeDetector,
    ExactMatchDetector,
    LLMDetector,
    RegexDetector,
)
from piighost.components.detector.ner import (
    Gliner2Detector,
    Gliner2PiiDetector,
    PresidioDetector,
    SpacyDetector,
    TransformersDetector,
)

The NER detectors each need their own extra (gliner2, spacy, transformers, presidio). LLMDetector needs the llm extra plus a provider package.

AnyDetector (protocol)¶

The port every detector implements. A single async method, so an implementation can await I/O such as a model server or an LLM API without blocking the pipeline.

class AnyDetector(Protocol):
    async def detect(self, text: str) -> list[Detection]: ...

detect returns detections in any order. Overlaps and duplicates are resolved by later pipeline stages, not by the detector.

Detection¶

Each detector returns a list of Detection, a frozen dataclass carrying where the match sits, what it matched, its label, and its confidence.

RegexDetector¶

Finds PII by matching one regex pattern per label. Each pattern is compiled once at construction, under re.ASCII, so \d and the other shape classes match ASCII only. A Unicode digit look-alike such as an Arabic-Indic numeral does not match, since a PII format uses ASCII digits. detect emits one detection per non-overlapping match at a flat confidence of 1.0.

It carries no checksum validator, so it matches on shape alone. A structured value mangled by OCR is kept rather than dropped, because dropping a real value would leak it.

Constructor¶

RegexDetector(patterns: dict[str, str])

from piighost.components.detector import RegexDetector

detector = RegexDetector({"EMAIL": r"[\w.+-]+@[\w.-]+\.\w{2,}"})
detections = await detector.detect("write to alice@example.com")
# [Detection(span=Span(9, 26), text="alice@example.com", label="EMAIL", confidence=1.0)]

from_hub¶

RegexDetector.from_hub(ref: str, *, hub: str | None = None) -> RegexDetector

Builds a detector from the regexes a piighost hub reference carries. The hub is a registry of tested de-identification regexes, addressed by namespace/name and an optional selector: a tag, or the eight hex characters of a commit.

from piighost.components.detector import RegexDetector

detector = RegexDetector.from_hub("piighost/logs:fd79aec6")
detections = await detector.detect("mail me at a@b.co from 10.0.0.1")

A reference pinned to a commit is immutable, so the answer is cached under ~/.cache/piighost/hub and read from disk on every later call. A reference pointing at a tag or at latest moves, so it is fetched every time: serving a stale one would quietly detect less than the caller asked for.

The call raises a subclass of HubError (piighost.hub) when the reference does not parse, the hub cannot be reached, or the reference resolves to something other than a plain regex detector. That last case covers a reference carrying a model detector: taking its regexes alone would detect less than the reference promises, so it fails instead of returning half of it.

It uses the standard library only, so the core install needs no extra.

CompositeDetector¶

Runs several detectors over the same text and merges their detections. It is itself an AnyDetector, so it composes with the pipeline unchanged. It runs every child concurrently and concatenates their results in child order. It does not deduplicate. Overlaps and duplicates flow to the span-conflict stage.

Constructor¶

CompositeDetector(detectors: list[AnyDetector])

from piighost.components.detector import CompositeDetector, RegexDetector
from piighost.components.detector.ner import Gliner2Detector

email_detector = RegexDetector({"EMAIL": r"[\w.+-]+@[\w.-]+\.\w{2,}"})
person_detector = Gliner2Detector(model="fastino/gliner2-multi-v1", labels=["PERSON"])
detector = CompositeDetector([email_detector, person_detector])

ExactMatchDetector¶

Finds whole-word occurrences of configured literal values. It scans the text for each value and emits one detection per occurrence at confidence 1.0. Matching is on word boundaries, so a value does not fire inside a longer word (Ann does not match inside Anne), and case-insensitive by default, so a value matches whatever its casing while the detection keeps the text as it appears. It carries no model and no optional dependency, which makes it the detector of choice for exercising the pipeline in tests.

Constructor¶

ExactMatchDetector(values: dict[str, str], case_sensitive: bool = False)

from piighost.components.detector import ExactMatchDetector

detector = ExactMatchDetector({"Patrick": "PERSON", "Lyon": "LOCATION"})
detections = await detector.detect("Patrick lives in Lyon")

ChunkedDetector¶

Runs a wrapped detector over each chunk of a long text. It is a decorator and itself an AnyDetector. It splits the text into overlapping chunks, runs the wrapped detector on each, and remaps every detection back to the original text. Strictly identical detections produced by the overlap are dropped. Label conflicts and differing confidences flow to the span-conflict stage.

Constructor¶

ChunkedDetector(detector: AnyDetector, splitter: AnySplitter | None = None)

from piighost.components.detector import ChunkedDetector
from piighost.components.detector.ner import SpacyDetector

spacy_detector = SpacyDetector(model="en_core_web_sm")
detector = ChunkedDetector(spacy_detector)

LLMDetector¶

Detects PII with a LangChain chat model via structured output. Needs the llm extra plus a provider package. The model is asked to extract (text, label) pairs against a schema whose label field is constrained to the configured labels. Each extracted value is then located in the source text by word-boundary search, so a value the model invented but absent from the text yields nothing. labels is required, since the schema is built from it. The source text is wrapped in <text_to_analyze> tags and the system prompt instructs the model to treat the tagged content as data, never as instructions, so a prompt-injection attempt inside the text cannot steer the extraction.

Constructor¶

LLMDetector(
    model: BaseChatModel | str,
    labels: list[str] | dict[str, str],
    prompt: str | None = None,
    provider: str | None = None,
    confidence: float = 1.0,
)

A custom prompt must contain a {labels} placeholder and, per LangChain's f-string format, double any other literal curly brace as {{ or }}.

from piighost.components.detector import LLMDetector

detector = LLMDetector(
    model="gpt-5.6-terra",
    labels=["PERSON", "EMAIL"],
    provider="openai",
)

NER detectors¶

The model-backed detectors extend BaseNERDetector, which handles label mapping and filtering (see below). Each needs its own extra and takes a loaded model or a model name to load, except PresidioDetector, which takes a constructed AnalyzerEngine.

Gliner2Detector¶

A zero-shot GLiNER2 model. Needs the gliner2 extra. labels is required, because GLiNER2 is queried with the internal labels. A str model is loaded with GLiNER2.from_pretrained.

Gliner2Detector(
    model: GLiNER2 | str,
    labels: list[str] | dict[str, str],
    threshold: float = 0.5,
    max_concurrency: int | None = None,
    max_chars: int | None = None,
    auto_chunk: bool = True,
)

Gliner2PiiDetector¶

A ready-to-use Gliner2Detector over fastino's GLiNER2 model fine-tuned for PII, with a preset label map so neither a model id nor a labels argument is needed. The preset spans the model's taxonomy, from names and contact details to identifiers, payment data, digital identity, secrets, and sensitive dates. Pass labels to narrow or extend the set, or model to inject a loaded instance, for example in a test, so no weights are downloaded.

Gliner2PiiDetector(
    model: GLiNER2 | str | None = None,
    labels: list[str] | dict[str, str] | None = None,
    threshold: float = 0.5,
    max_concurrency: int | None = None,
    max_chars: int | None = None,
    auto_chunk: bool = True,
)

SpacyDetector¶

A spaCy NER model. Needs the spacy extra. labels is optional. When omitted, every entity spaCy produces is kept with its spaCy label. A str model is loaded with spacy.load.

SpacyDetector(
    model: Language | str,
    labels: list[str] | dict[str, str] | None = None,
    max_concurrency: int | None = None,
)

TransformersDetector¶

A Hugging Face token-classification pipeline. Needs the transformers extra. labels is optional, kept native when omitted. A str pipeline is loaded as an ner pipeline. An entity scoring below threshold is dropped.

TransformersDetector(
    pipeline: TokenClassificationPipeline | str,
    labels: list[str] | dict[str, str] | None = None,
    threshold: float = 0.0,
    max_concurrency: int | None = None,
    aggregation_strategy: str = "simple",
    max_chars: int | None = None,
    auto_chunk: bool = True,
)

PresidioDetector¶

Wraps a Presidio AnalyzerEngine so a caller reuses Presidio's recognizers. Needs the presidio extra. The analyzer is injected, since an engine is assembled from an NLP engine and a recognizer registry, not loaded from a name. labels is optional, kept native when omitted. An entity scoring below threshold is dropped by Presidio.

PresidioDetector(
    analyzer: AnalyzerEngine,
    labels: list[str] | dict[str, str] | None = None,
    language: str = "en",
    threshold: float = 0.0,
    max_concurrency: int | None = None,
)

From a config, the presidio detector type builds Presidio's default English AnalyzerEngine. For another language or custom recognizers, construct the engine yourself and use PresidioDetector directly.

Long-text handling¶

Gliner2Detector and TransformersDetector take max_chars with auto_chunk (default True). A text longer than max_chars is split into overlapping chunks, scanned separately, and remapped back onto the original text. With auto_chunk off, a text over the bound raises TextTooLongError instead. max_chars defaults to None, so there is no bound and the whole text is scanned in one pass. SpacyDetector and PresidioDetector do not expose these.

Label mapping¶

BaseNERDetector normalizes the labels argument into an external-to-internal map, then maps and filters the detections the model produces. It distinguishes the label a model uses natively from the label emitted in Detection.label.

• A list, ["PERSON", "LOCATION"], maps each label to itself.
• A map, {"PERSON": "PER"}, takes the emitted label as its key and the model's native label as its value, so a detection the model labels PER is emitted as PERSON. A native label absent from the map values is dropped.
• None or an empty map applies no mapping, so every detection is kept with the label the model gave it.

Two external labels mapping to one internal label raise LabelMappingError, since the reverse lookup would be ambiguous.

from piighost.components.detector.ner import TransformersDetector

detector = TransformersDetector(
    pipeline="dslim/bert-base-NER",
    labels={"PERSON": "PER", "LOCATION": "LOC"},
)

Pattern catalogs¶

Reusable regex pattern sets for RegexDetector. Each catalog is a plain dict[str, str] mapping a PII label to a regex pattern string. Patterns match on shape alone, with no checksum validation.

from piighost.components.detector.patterns import (
    EU_PATTERNS,
    FR_PATTERNS,
    GENERIC_PATTERNS,
    US_PATTERNS,
)

Feed a catalog to a RegexDetector, or merge several by dict merge, an inline pattern on the same label taking precedence.

from piighost.components.detector import RegexDetector
from piighost.components.detector.patterns import FR_PATTERNS, GENERIC_PATTERNS

detector = RegexDetector({**GENERIC_PATTERNS, **FR_PATTERNS})

Every catalog pattern is tested against catastrophic backtracking, so an adversarial input cannot turn a scan into a denial of service.

The GENERIC_PATTERNS labels are country-agnostic. The others are prefixed (US_, FR_) so they do not collide when catalogs are merged. EU_PATTERNS carries the ISO 13616 IBAN shared across member states. For country-specific numbers, use a per-country catalog.

Pulling catalogs from a config¶

A regex detector config pulls catalogs via catalogs. An entry is either a prebuilt name, among generic, us, eu, fr, or a hub reference written hub:namespace/name with an optional :selector. The catalogs merge in order, then any inline patterns, so an inline pattern overrides a catalog pattern on the same label. A regex detector config needs at least one inline pattern or one catalog.

[detector]
type = "regex"
catalogs = ["generic", "fr"]

[detector.patterns]
INTERNAL_ID = "EMP-\\d{6}"

A hub reference names a reviewed catalogue instead of carrying a copy of it, so the config stays short and the patterns stay auditable at their source:

[detector]
type = "regex"
catalogs = ["hub:piighost/logs:fd79aec6"]

A hub catalog is fetched when the config is built, not when it is parsed, and a reference pinned to a commit is cached on disk afterwards. Set PIIGHOST_HUB_URL to pull from a private registry. An unknown name or a malformed reference fails at load time rather than as a bad URL later.

See also¶

• Pipeline reference for the pipeline that drives the detector.
• Pre-built detectors for composing catalogs in practice.
• TOML configuration for the declarative build.
• Extending PIIGhost for writing your own detector.
• Data models reference for the full shape of a Detection.