# Dataset contract

## When the dataset is needed

Do not upload a dataset at project start. The first dataset handoff occurs only after the synthetic pipeline, dependency diagnostics, unit tests, end-to-end tests, and visual PDF verification are green. This milestone is named **D1 - External calibration ready**.

The initial handoff must be synthetic or already lawfully de-identified. Do not upload raw patient documents to chat, email, a public repository, or this source tree.

## First handoff: calibration set

Recommended initial size:

- 50-100 documents
- 150-300 pages
- 10-15 common MIKA document types
- Russian printed text as the majority
- At least 20 percent photographs or scans
- At least 10 percent intentionally poor-quality pages
- At least one barcode example per relevant document type

## Required structure

```text
mika-deid-calibration-v1/
  README.md
  manifest.csv
  documents/
    doc-000001.pdf
    doc-000002.jpg
  annotations/
    doc-000001.json
    doc-000002.json
```

Filenames must be random and must not contain a name, diagnosis, date of birth, hospital number, phone, email, or original filename.

## Manifest columns

```csv
document_id,file_path,sha256,document_type,language,page_count,source_kind,quality_tier,contains_handwriting,contains_barcode,split_group
```

- `document_id`: random dataset identifier
- `file_path`: relative path inside the package
- `sha256`: file checksum
- `document_type`: controlled type such as discharge_summary, lab, pathology, imaging_report
- `language`: ru, en, or mixed
- `source_kind`: native_pdf, scanned_pdf, photo, or image
- `quality_tier`: good, medium, or poor
- `contains_handwriting`: true/false
- `contains_barcode`: true/false
- `split_group`: groups related pages or versions so they never cross train/calibration/test splits

## Annotation schema

Coordinates are normalized to the full rendered page from 0 to 1.

```json
{
  "schema": "mika-deid-annotation-v1",
  "document_id": "doc-000001",
  "pages": [
    {
      "page": 1,
      "width": 1.0,
      "height": 1.0,
      "regions": [
        {
          "region_id": "r-0001",
          "category": "person_name",
          "box": {"x0": 0.10, "y0": 0.08, "x1": 0.42, "y1": 0.12},
          "required_action": "redact",
          "legibility": "clear",
          "notes": "synthetic label only"
        }
      ]
    }
  ]
}
```

Annotation notes must never repeat the actual identifier text.

## Allowed categories

- person_name
- birth_date
- phone
- email
- snils
- passport
- insurance_policy
- medical_record_id
- address
- barcode
- signature
- face
- organization
- event_date
- indirect_identifier

## Dataset stages

1. `synthetic-dev`: generated examples used during implementation.
2. `external-calibration`: synthetic or lawfully prepared documents used to adjust rules; results may influence code.
3. `locked-validation`: never used to adjust rules; used for release metrics.
4. `controlled-pilot`: real-world documents only after legal, privacy, security, consent, and access gates pass.

No document or patient may appear in more than one split. Related versions, pages, and visits share a `split_group`.

## Handoff checklist

- Dataset owner and lawful basis recorded
- No raw patient material in the source repository
- No identifiers in filenames or annotation notes
- Checksums complete
- Malware and file-type scan complete
- Annotation performed by two reviewers for critical categories
- Disagreements adjudicated
- Clinical utility fields separately marked
- Secure transfer destination approved
- Retention and deletion date approved

