JSON Sidecar Schema & Automated Validation

1. Purpose and Workflow Integration

When delivering digitized assets to the Library, external vendors must supply a structured JSON sidecar file alongside every individual digital asset, including preservation masters, production/service files, access derivatives, and OCR text files.

While the Inventory Spreadsheet establishes the initial item-level metadata provided by the Library at intake, the JSON sidecar schema governs the asset-level, technical, and operational metadata generated by the vendor during digitization.

The authoritative JSON Schema is maintained as a separate, publicly accessible file. Vendors must integrate this schema into their automated quality-control pipelines before delivery.

• Public schema: digitized_image_schema.json
• Sample Photograph Sidecar (Single-Unit): sample_photograph_sidecar.json
• Sample Split Microfilm Sidecar (Multi-Unit): sample_split_microfilm_sidecar.json

The public URL must return the raw JSON Schema document rather than an HTML page, redirect, or error response.

2. Core Schema Requirements

Every delivered JSON sidecar must contain a valid JSON object that conforms to JSON Schema Draft 7.

The schema uses additionalProperties: false where applicable. Undocumented properties added by vendor software will therefore cause validation to fail.

Property names and controlled-vocabulary values are case-sensitive.

2.1 Required Root Objects

Every sidecar must contain the following six top-level objects:

1. administrative
   Grant, contract, or project-level administrative data, such as projectCode, fundingSource, and schemaVersion.

2. asset
   File-level tracking data, including the file role (pres, serv, access, or ocr) and exact referenceFilename.

3. identifiers
   Bibliographic and archival identifiers mapped from the Library-supplied inventory spreadsheet, such as barcode, divisionCode, bnumber, and shelf_locator (or classmark).

4. source
   Physical-source characteristics observed at capture, including the controlled object type and format pairing and sequential capture data such as sequenceNumber, unitIndex, unitTitle, side, and sequenceLabel.

5. digitizationProcess
   Technical metadata describing the capture device, lens, capture software, equipment identifiers, and calibration targets.

6. digitizer
   Administrative information identifying the operator and vendor organization responsible for capture.

3. Strict Naming and Controlled Vocabularies

3.1 Filename Pattern

To support reliable automated ingest and validation, the referenceFilename property in the asset object must follow one of the Library’s approved naming structures.

The base filename pattern is:

^33433\d{9}_(?:(?:negative|positive)_)?(?:[rv]|\d{5})_(?:pres|serv|access|ocr)\.[a-z0-9]+$

The pattern supports the following two filename structures:

[barcode]_[side-or-sequence]_[role].[extension]

[barcode]_[negative-or-positive]_[side-or-sequence]_[role].[extension]

Multi-unit identifiers such as u001 and u002 are not included in filenames. Multi-unit membership is represented by the asset’s enclosing directory and by the source.sequence.unitIndex property in its JSON sidecar.

Pattern components

• Barcode — 33433\d{9}
  A 14-digit NYPL barcode beginning with 33433.

• Negative-source component — negative|positive
  Used only for parallel files derived from a physical film or glass negative:

  • negative identifies the faithful, non-inverted representation of the source negative;
  • positive identifies the digitally inverted positive representation derived from that negative.

  These components do not describe all materials that happen to display as positive images. They must not be used for photographic prints, slides, transparencies, reflective materials, or other natively positive source objects.

• Side or sequence — [rv]|\d{5}
  Either:

  • r for recto;
  • v for verso; or
  • a five-digit, zero-padded sequence number such as 00001 or 00981.

• File role — pres|serv|access|ocr
  One of the four approved delivery roles:

  • pres;
  • serv;
  • access;
  • ocr.

• Extension — [a-z0-9]+
  A lowercase file extension such as .tif, .jp2, .jpg, .pdf, .xml, or .txt.

Valid examples

33433123456789_r_pres.tif
33433123456789_v_access.jpg
33433123456789_00001_serv.tif
33433987654321_negative_00001_pres.tif
33433987654321_positive_00001_access.jpg
33433555554444_00981_ocr.xml

Invalid examples

33433123456789_positive_r_pres.tif
33433123456789_R_pres.tif
33433123456789_1_pres.tif
33433555554444_u001_00001_ocr.xml
33433987654321_negative_u001_00001_pres.tif
nypl_33433123456789_00001_pres.tif
33433123456789_00001_master.tif
33433123456789_00001_pres.TIF

The filename pattern must be encoded with doubled backslashes when stored as a JSON string inside the schema:

{
  "pattern": "^33433\\d{9}_(?:(?:negative|positive)_)?(?:[rv]|\\d{5})_(?:pres|serv|access|ocr)\\.[a-z0-9]+$"
}

The schema applies additional conditional validation based on source.object.format:

• When the physical source is a black-and-white negative, color negative, or glass plate negative, referenceFilename must include either negative or positive.
• For all other source formats, referenceFilename must not include either component.
• The suffix in referenceFilename must agree with asset.fileRole. For example, an asset with "fileRole": "pres" must have a filename ending in _pres followed by its extension.

3.2 Multi-Unit Directory and Sequence Rules

When one physical item contains multiple distinct intellectual units, the Library still requires one BagIt bag for the complete physical object. The bag root is named with the physical item’s barcode.

Each intellectual unit is represented by a zero-padded unit subdirectory inside the bag’s data/ directory:

33433555554444/
└── data/
    ├── u001/
    └── u002/

The unit identifier is recorded in two places:

1. the enclosing directory, such as data/u002/; and
2. the JSON sidecar property source.sequence.unitIndex, where the integer 2 corresponds to u002.

A human-readable intellectual-unit title is recorded in source.sequence.unitTitle.

The unit identifier must not appear in referenceFilename.

Sequence numbering must remain absolute and uninterrupted across the entire physical object. It must not restart when a new intellectual unit begins.

For example, when unit 1 contains frames 1 through 980 and unit 2 begins at frame 981, the files are structured as follows:

33433555554444/
└── data/
    ├── u001/
    │   ├── 33433555554444_00001_pres.tif
    │   └── 33433555554444_00980_pres.tif
    └── u002/
        ├── 33433555554444_00981_pres.tif
        └── 33433555554444_01650_pres.tif

The JSON sidecar for the first frame in u002 would therefore include:

{
  "administrative": {
    "schemaVersion": "1.0",
    "projectCode": "NEH-2024-001",
    "fundingSource": "NEH_GRANT_1"
  },
  "asset": {
    "fileRole": "pres",
    "referenceFilename": "33433555554444_00981_pres.tif"
  },
  "source": {
    "object": {
      "type": "microform",
      "format": "35mm roll microfilm"
    },
    "sequence": {
      "unitIndex": 2,
      "unitTitle": "Pamphlet 2",
      "sequenceNumber": 981,
      "sequenceLabel": "Frame 981"
    }
  }
}

The integer sequenceNumber is stored as 981 in JSON. Its filename representation is zero-padded to five digits as 00981.

3.3 Controlled-Vocabulary Validation

The schema uses conditional JSON Schema logic, including oneOf, to enforce approved pairings between source.object.type and source.object.format.

For example, when a sidecar declares:

{
  "type": "reflective photographic"
}

the corresponding format value must be one of the formats explicitly permitted for that object type. An unsupported value will cause schema validation to fail.

Vendors must use the exact spelling, capitalization, and punctuation defined by the schema. Local synonyms, abbreviations, or alternate labels are not permitted unless the Library approves a schema revision.

4. Automated Validation Workflows

Vendors must perform automated schema validation on 100% of generated JSON sidecars before packaging and delivering files to the Library.

Validation must occur against a controlled local copy of the authoritative schema. The schema copy should be downloaded at the beginning of the project or build process rather than fetched separately for every sidecar.

4.1 Node.js Command-Line Validation with AJV

AJV provides command-line JSON Schema validation through the ajv-cli package.

1. Install AJV CLI

npm install --global ajv-cli

2. Download the authoritative schema

curl --fail --location \
  --output digitized_image_schema.json \
  https://nypl-research.github.io/digital-imaging-resources/schemas/digitized_image_schema.json

The --fail option causes the command to return an error for an unsuccessful HTTP response instead of saving an error page as though it were a schema.

3. Validate one sidecar

ajv validate \
  --spec=draft7 \
  --all-errors \
  -s ./digitized_image_schema.json \
  -d /path/to/delivery/33433123456789_r_pres.json

4. Validate an entire directory

ajv validate \
  --spec=draft7 \
  --all-errors \
  -s ./digitized_image_schema.json \
  -d "/path/to/delivery/**/*.json"

AJV returns an exit status of 0 when all supplied files are valid and an exit status of 1 when one or more files fail validation.

Validation errors identify the affected instance path, schema path, failed keyword, and associated error message. For example, an error may identify:

• a missing required property;
• an unapproved additional property;
• a controlled-vocabulary mismatch;
• an incorrect data type;
• a malformed filename;
• a filename that does not match referenceFilename;
• a mismatch between asset.fileRole and the filename suffix;
• an invalid use or omission of the negative or positive filename component.

For more readable output, add:

--errors=text

Example:

ajv validate \
  --spec=draft7 \
  --all-errors \
  --errors=text \
  -s ./digitized_image_schema.json \
  -d "/path/to/delivery/**/*.json"

4.2 Python Validation with jsonschema

Vendors integrating validation into Python processing pipelines may use the jsonschema package.

1. Install the package

python -m pip install jsonschema

2. Validate a sidecar

#!/usr/bin/env python3

import json
import sys
import urllib.error
import urllib.request
from json import JSONDecodeError
from pathlib import Path
from typing import Any, Iterable

from jsonschema import Draft7Validator
from jsonschema.exceptions import SchemaError, ValidationError


SCHEMA_URL = (
    "https://nypl-research.github.io/digital-imaging-resources/"
    "schemas/digitized_image_schema.json"
)

SIDECAR_PATH = Path("33433123456789_r_pres.json")


def load_remote_json(url: str) -> Any:
    """Download and parse a JSON document."""

    request = urllib.request.Request(
        url,
        headers={"User-Agent": "NYPL-sidecar-validator/1.0"},
    )

    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read().decode("utf-8"))


def load_local_json(path: Path) -> Any:
    """Read and parse a local UTF-8 JSON file."""

    with path.open("r", encoding="utf-8") as file_handle:
        return json.load(file_handle)


def format_instance_path(path: Iterable[Any]) -> str:
    """Convert a jsonschema error path into a readable JSON path."""

    formatted = "$"

    for component in path:
        if isinstance(component, int):
            formatted += f"[{component}]"
        else:
            formatted += f".{component}"

    return formatted


def main() -> int:
    try:
        schema = load_remote_json(SCHEMA_URL)
        Draft7Validator.check_schema(schema)

        sidecar_data = load_local_json(SIDECAR_PATH)

    except urllib.error.URLError as error:
        print(
            f"SCHEMA DOWNLOAD ERROR: {error}",
            file=sys.stderr,
        )
        return 2

    except FileNotFoundError:
        print(
            f"FILE ERROR: Sidecar not found: {SIDECAR_PATH}",
            file=sys.stderr,
        )
        return 2

    except JSONDecodeError as error:
        print(
            f"JSON SYNTAX ERROR: {error}",
            file=sys.stderr,
        )
        return 2

    except SchemaError as error:
        print(
            f"SCHEMA ERROR: {error.message}",
            file=sys.stderr,
        )
        return 2

    validator = Draft7Validator(schema)

    errors: list[ValidationError] = sorted(
        validator.iter_errors(sidecar_data),
        key=lambda error: list(error.absolute_path),
    )

    if not errors:
        print(
            f"VALID: {SIDECAR_PATH} is ready for delivery."
        )
        return 0

    print(
        f"INVALID: {SIDECAR_PATH} contains "
        f"{len(errors)} schema validation error(s).",
        file=sys.stderr,
    )

    for number, error in enumerate(errors, start=1):
        instance_path = format_instance_path(error.absolute_path)
        schema_path = "/".join(str(part) for part in error.absolute_schema_path)

        print(
            f"\nError {number}:",
            file=sys.stderr,
        )
        print(
            f"  Instance path: {instance_path}",
            file=sys.stderr,
        )
        print(
            f"  Schema path:   {schema_path}",
            file=sys.stderr,
        )
        print(
            f"  Message:       {error.message}",
            file=sys.stderr,
        )

    return 1


if __name__ == "__main__":
    raise SystemExit(main())

This example uses three exit statuses:

• 0: the sidecar is valid;
• 1: the sidecar is valid JSON but fails schema validation;
• 2: the schema or sidecar could not be loaded or parsed.

For high-volume validation, vendors should download the schema once, instantiate one Draft7Validator, and reuse that validator for every sidecar in the batch.

4.3 Validation Beyond JSON Schema

JSON Schema validation confirms that an individual sidecar follows the required structure and controlled rules. It does not inspect the filesystem or compare values across multiple files.

The vendor’s delivery-validation workflow must therefore perform additional checks that cannot be enforced by the Draft 7 schema alone, including:

• confirming that the barcode at the beginning of asset.referenceFilename matches identifiers.barcode;
• confirming that the asset file and its JSON sidecar have matching basenames;
• confirming that a sidecar with unitIndex: 2 resides in data/u002/;
• confirming that sequence numbers remain continuous across unit-directory boundaries;
• confirming that sequence numbering does not restart within each unit;
• confirming that every file listed in the BagIt payload is included in the applicable checksum manifest;
• confirming that no asset is duplicated or omitted when files are grouped into unit directories.

5. Schema Access and Download

The authoritative schema is maintained as an independent JSON artifact rather than duplicated within this documentation.

5.1 Browser Access

Open the following resource to view or save the schema:

• digitized_image_schema.json

5.2 Download with cURL

curl --fail --location \
  --output digitized_image_schema.json \
  https://nypl-research.github.io/digital-imaging-resources/schemas/digitized_image_schema.json

5.3 Download with Wget

wget \
  --output-document=digitized_image_schema.json \
  https://nypl-research.github.io/digital-imaging-resources/schemas/digitized_image_schema.json

5.4 Confirm That the Downloaded File Is JSON

A successful HTTP download does not by itself prove that the downloaded document contains valid JSON. Vendors should parse or validate the schema before using it.

Using Python:

python -m json.tool digitized_image_schema.json > /dev/null

Using AJV:

ajv compile \
  --spec=draft7 \
  -s ./digitized_image_schema.json

A failed parse or compile must stop the validation workflow. An HTML error page, empty file, malformed JSON document, or invalid schema must never be treated as an authoritative schema copy.