WiP: Branch with auto converted linkml model by yarikoptic · Pull Request #381 · dandi/dandi-schema

yarikoptic · 2026-03-10T19:46:27Z

NEVER TO BE MERGED

To preview and test results, actual changes should go to

Linkml conversion tooling #387

yarikoptic

initial review pointers

dandischema/models.yaml

yarikoptic · 2026-03-10T20:31:38Z

dandischema/models.yaml

+        name: identifier
+        range: string
+        required: true
+        pattern: ^(?:urn:uuid:)?[0-9a-fA-F]{8}-?[0-9a-fA-F]{4}-?4[0-9a-fA-F]{3}-?[89abAB][0-9a-fA-F]{3}-?[0-9a-fA-F]{12}$


per discussion with @candleindark and claude -- it seems we should be able to use types here. Here is an example excerpt from https://claude.ai/share/66fe0f31-8cf1-40e6-b9fd-47c6719f2006

types: ORCID: uri: xsd:string base: str pattern: "^https://orcid\\.org/\\d{4}-\\d{4}-\\d{4}-\\d{3}[0-9X]$" description: An ORCID identifier ISSN: uri: xsd:string base: str pattern: "^\\d{4}-\\d{3}[0-9X]$" description: An ISSN identifier UUID: uri: xsd:string base: str pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" description: A UUID slots: contributor_id: range: ORCID publication_id: range: ISSN asset_id: range: UUID

The definitions in dandischema.models doesn't define those types with patterns information attached. Those types are actually defines as alias of str

dandi-schema/dandischema/models.py

Lines 714 to 718 in 1deb319

Identifier = str

ORCID = str

RORID = str

DANDI = str

RRID = str

.

Because of that, I don't think these type definitions should be part of the result of auto translation but added manually something to add to the result of the auto translation.

dandischema/models.yaml

yarikoptic · 2026-03-10T20:58:07Z

dandischema/models.yaml

+    notes:
+    - 'pydantic2linkml: Unable to translate the logic contained in the wrap validation
+      function, <function _BaseUrl.__get_pydantic_core_schema__.<locals>.wrap_val
+      at 0x7f8c96c36f80>.'


this is for List[AnyHttpUrl] and @candleindark says that there is custom validator attached. May be we could just provide custom type for validation here.

codecov · 2026-03-10T22:07:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.92%. Comparing base (4b89e4f) to head (bd38142).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #381   +/-   ##
=======================================
  Coverage   97.92%   97.92%           
=======================================
  Files          18       18           
  Lines        2405     2405           
=======================================
  Hits         2355     2355           
  Misses         50       50

Flag	Coverage Δ
unittests	`97.92% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dandischema/models.yaml

pyproject.toml

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

=== Do not change lines below === { "chain": [], "cmd": "hatch run linkml-auto-converted:2linkml", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

…_orig for now

…tions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

… models.yaml The previous BRE pattern used `\+` (GNU sed extension) which silently fails on macOS BSD sed. Switch to `-E` (extended regex) with POSIX character class `[^[:space:]]` instead of `\S` (also unsupported by BSD sed), making the normalization work on both macOS and Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

The previous BRE pattern used `\+` (GNU sed extension) which silently fails on macOS BSD sed. Switch to `-E` (extended regex) with POSIX character class `[^[:space:]]` instead of `\S` (also unsupported by BSD sed), making the normalization work on both macOS and Linux. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Expand comment for linkml-auto-converted hatch env with usage instructions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

…ed and some symbols from _orig for now we do it so it does not overlay models.py since then git is unable to track renames

…into linkml-auto-converted

… flake8 issues

we had to maintain original filename for models.py to apply patches easily

for more information, see https://pre-commit.ci

Provide a partial schema to be merged with the generated schema

candleindark · 2026-03-31T01:26:48Z

I reset the merge target to the master branch. Not doing so will cause whatever changes newly introduced in master to show up in the diff as well. However, github doesn't see to be working correctly to get the correct diff in files.

yarikoptic commented Mar 10, 2026

View reviewed changes

candleindark force-pushed the linkml-auto-converted branch from fa28737 to c78f9d3 Compare March 10, 2026 23:08

yarikoptic commented Mar 12, 2026

View reviewed changes

dandischema/models.yaml Show resolved Hide resolved

This was referenced Mar 12, 2026

Duplicate Publishable attributes in CommonModel + DOI in Dandiset as Optional #285

Closed

Migration to LinkML from Pydantic #382

Open

yarikoptic commented Mar 13, 2026

View reviewed changes

pyproject.toml Show resolved Hide resolved

candleindark and others added 5 commits March 13, 2026 17:01

build(hatch-env): specify env for conversion to LinkML and back

cffee39

Specify Hatch-managed env for auto converting `dandischema.models` to LinkML schema and back to Pydantic models

[DATALAD RUNCMD] Auto-convert the model

21e7e9c

=== Do not change lines below === { "chain": [], "cmd": "hatch run linkml-auto-converted:2linkml", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [], "pwd": "." } ^^^ Do not change lines above ^^^

build(hatch-env): provide script to translate dandischema.models

b5644c1

Provide script to translate `dandischema.models` in to a LinkML schema and overly it with definition provided by an overlay file.

Now we will sanitize the converted model addresses

f109ff4

build(hatch-env): provide script to translate dandischema/models.yaml

cfe3bea

Provide script to translate `dandischema/models.yaml` back to Pydantic models and store them in `dandischema/models.py`

yarikoptic force-pushed the linkml-auto-converted branch from 28c5b44 to bd38142 Compare March 13, 2026 21:17

yarikoptic and others added 9 commits March 13, 2026 18:32

more sanitization

7580f80

More Sanitization applied

f130f4f

Reextracted model (now less clutter and with spaces)

0a8ef6b

feat: provide prefixes definition in the overlay file

63dd301

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

Added linkml: prefix and sorted all the entries

f8125d0

Provide default prefix to be custom dandi_default

7c5077a

Reextracted model (now with prefixes and deterministic order of notes)

3485245

rename original models into models_orig

bff037c

Adding models.py back to import from converted and some symbols from …

6bb4b52

…_orig for now

yarikoptic force-pushed the linkml-auto-converted branch from bd38142 to 6bb4b52 Compare March 13, 2026 22:33

candleindark and others added 7 commits March 15, 2026 22:40

Expand comment for linkml-auto-converted hatch env with usage instruc…

725e957

…tions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix: correct default_prefix to dandi

a2324fa

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

Now we will sanitize the converted model addresses

73c46b6

more sanitization

c79b1e2

feat: provide prefixes definition in the overlay file

0b2b51d

These prefixes are copied from https://github.com/dandi/schema/blob/master/releases/0.7.0/context.json

Added linkml: prefix and sorted all the entries

6cb2f82

yarikoptic and others added 10 commits March 20, 2026 17:01

Provide default prefix to be custom dandi_default

76526a9

fix: correct default_prefix to dandi

fad6879

There is no prefix defined as `dandi_default`. The intended default prefix is `dandi`

Adding models_importstab.py so we could use it to import from convert…

5f78b87

…ed and some symbols from _orig for now we do it so it does not overlay models.py since then git is unable to track renames

Adding the tool to assist in establishing "merges" with regeneration …

0208c9c

…into linkml-auto-converted

We are doomed to use --no-verify for now as converted python code has…

488e2e9

… flake8 issues

BF the commit version specification

0ba5ad7

Implement poor man patch queue

2faf932

reflect that I rewritten how we handle patching

c177479

we had to maintain original filename for models.py to apply patches easily

Inform on what we are doing

54cf276

yarikoptic mentioned this pull request Mar 20, 2026

Linkml conversion tooling #387

Draft

11 tasks

yarikoptic and others added 4 commits March 20, 2026 18:10

Now do patch diff of master in as well for good measure

5eb4a91

Removing reporting branches - did not work

76a9d77

'Merged' with linkml-conversion 0.12.1-55-g76a9d77 and 2 branches

7ab9964

[pre-commit.ci] auto fixes from pre-commit.com hooks

78d6899

for more information, see https://pre-commit.ci

yarikoptic changed the base branch from master to linkml-conversion March 20, 2026 22:22

yarikoptic and others added 6 commits March 20, 2026 18:25

ENH: patch converted to pydantic model to strip dciteCOLO

c6172c4

do pre-commit

e2d124c

'Merged' with linkml-conversion 0.12.1-57-ge2d124c and 2 branches

3938e26

feat: provide partial schema

d8b1bea

Provide a partial schema to be merged with the generated schema

feat: enable 2linkml to merge partial schema in models_merge.yaml

c0fbd02

'Merged' with linkml-conversion 0.12.1-59-gc0fbd02 and 2 branches

4e95f1e

candleindark force-pushed the linkml-conversion branch 2 times, most recently from 59b0587 to c0fbd02 Compare March 31, 2026 00:59

'Merged' with linkml-conversion 0.12.1-59-gc0fbd02 and 2 branches

991790f

candleindark changed the base branch from linkml-conversion to master March 31, 2026 01:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WiP: Branch with auto converted linkml model#381

WiP: Branch with auto converted linkml model#381
yarikoptic wants to merge 42 commits intomasterfrom
linkml-auto-converted

yarikoptic commented Mar 10, 2026 •

edited

Loading

Uh oh!

yarikoptic left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

candleindark Mar 12, 2026

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Uh oh!

codecov bot commented Mar 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

candleindark commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yarikoptic commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

NEVER TO BE MERGED

To preview and test results, actual changes should go to

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

candleindark Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yarikoptic Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

candleindark commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yarikoptic commented Mar 10, 2026 •

edited

Loading

codecov bot commented Mar 10, 2026 •

edited

Loading