WIP: Source Links Fix by MaximilianSoerenPollak · Pull Request #440 · eclipse-score/docs-as-code

MaximilianSoerenPollak · 2026-03-11T21:07:55Z

Still in progress, do not merge.
It works but need to fix tests and clean it up a lot.

Still TODO:

Achieve same or better test coverage as currently (see last comment)
- Adapt all current tests to work with new structure
- Add new tests to enable testing of new functionality
Update Documentation if needed
Clean up implementation (naming / debug prints etc. )
Proof it works in ref-integration & locally (docs-as-code & other local repos)

📌 Description

🚨 Impact Analysis

This change does not violate any tool requirements and is covered by existing tool requirements
This change does not violate any design decisions
Otherwise I have created a ticket for new tool qualification

✅ Checklist

Added/updated documentation for new or changed features
Added/updated tests to cover the changes
Followed project coding standards and guidelines

github-actions · 2026-03-11T21:08:33Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //src:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.3.0) and connecting to it...
INFO: Invocation ID: a498a856-a238-4879-b4fc-4995b788a549
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: src
Loading: 0 packages loaded
    currently loading: src
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //src:license-check (57 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (68 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (76 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (128 packages loaded, 1672 targets configured)

Analyzing: target //src:license-check (129 packages loaded, 2146 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2564 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2576 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2576 targets configured)

Analyzing: target //src:license-check (143 packages loaded, 4465 targets configured)

Analyzing: target //src:license-check (144 packages loaded, 4589 targets configured)

INFO: Analyzed target //src:license-check (145 packages loaded, 4715 targets configured).
[12 / 16] [Sched] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes
[13 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 1s disk-cache, processwrapper-sandbox
[14 / 16] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar; 0s disk-cache, processwrapper-sandbox
INFO: Found 1 target...
Target //src:license.check.license_check up-to-date:
  bazel-bin/src/license.check.license_check
  bazel-bin/src/license.check.license_check.jar
INFO: Elapsed time: 23.343s, Critical Path: 2.84s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/src/license.check.license_check src/formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

github-actions · 2026-03-11T21:13:18Z

The created documentation from the pull request is available at: docu-html

src/extensions/score_source_code_linker/metadata.py

Copilot

Pull request overview

This PR reworks how “source code links” are generated and injected into Sphinx needs, aiming to support both local builds (derive Git hash from the current repo) and combo/ref-integration builds (use per-module metadata such as repo URL + commit hash).

Changes:

Add metadata support (module name / repo URL / commit hash) to NeedLink JSON handling and Bazel CLI generation/merge scripts.
Introduce grouping of needs by module and a new module-grouped cache JSON.
Update Sphinx extension flow to read the new caches and generate GitHub links using either git-derived or metadata-derived refs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
src/helper_lib/additional_functions.py	Changes GitHub link generation API to require module metadata.
src/extensions/score_source_code_linker/needlinks.py	Extends NeedLink with metadata fields; adds new metadata-aware JSON format/load path.
src/extensions/score_source_code_linker/need_source_links.py	Moves `group_by_need` into this module.
src/extensions/score_source_code_linker/module_source_links.py	New module-grouped cache format and grouping logic.
src/extensions/score_source_code_linker/metadata.py	New TypedDict + TypeGuard for metadata records.
src/extensions/score_source_code_linker/generate_source_code_links_json.py	Refactors extraction helper signature and logging (currently inconsistent call sites).
src/extensions/score_source_code_linker/init.py	Adds module-linker stage and changes injection to use module-grouped cache + metadata-based link generation.
scripts_bazel/merge_sourcelinks.py	Merges per-module sourcelinks and enriches with known-good repo/hash metadata.
scripts_bazel/generate_sourcelinks_cli.py	Emits sourcelinks JSON with a leading metadata dict and uses updated extraction helper signature.
docs.bzl	Adds optional `known_good` wiring into the sourcelinks merge genrule and the public `docs()` macro.

You can also share your feedback on Copilot code review. Take the survey.

scripts_bazel/merge_sourcelinks.py

scripts_bazel/generate_sourcelinks_cli.py

Copilot · 2026-03-12T00:15:37Z

scripts_bazel/generate_sourcelinks_cli.py

+    store_source_code_links_with_metadata_json(
+        file=args.output, metadata=metadata, needlist=all_need_references
+    )


This switches the generated JSON format from a plain list of NeedLinks to a list whose first element is a metadata dict. Any existing consumers/tests that expect the old schema will now fail. Consider either keeping the old format as default (with an opt-in flag for metadata), or updating all in-repo consumers and tests in the same PR to avoid a partially-migrated state.

@a-zw Valid issue here .

Do you think it would be better to rename this specific cache that comes from here a bit so it is clear it is with metadata?
Like 'scl_metadata_cache.json' or whatever? SO that the name makes it clear to use the metadata reader?

src/extensions/score_source_code_linker/__init__.py

Copilot · 2026-03-12T00:15:39Z

src/helper_lib/additional_functions.py

+    metadata: moduleInfo,
+    link: NeedLink | DataForTestLink | DataOfTestCase | None = None,
+) -> str:
+    if link is None:
+        link = DefaultNeedLink()
+    if not metadata.hash:
+        # Local path (//:docs)
+        return get_github_link_from_git(link)
+    # Ref-Integration path (//:docs_combo..)
+    return get_github_link_from_json(metadata, link)


get_github_link now requires a metadata argument, but there are existing call sites in the repo that still call get_github_link(link) (e.g. in tests and xml_parser.py). This is a breaking API change that will raise TypeError. Consider keeping backwards compatibility by making metadata optional with a sensible default (e.g. infer local/git behavior when metadata is omitted) or providing a new function name for the metadata-based behavior.

Suggested change

metadata: moduleInfo,

link: NeedLink | DataForTestLink | DataOfTestCase | None = None,

) -> str:

if link is None:

link = DefaultNeedLink()

if not metadata.hash:

# Local path (//:docs)

return get_github_link_from_git(link)

# Ref-Integration path (//:docs_combo..)

return get_github_link_from_json(metadata, link)

metadata: moduleInfo

| NeedLink

| DataForTestLink

| DataOfTestCase

| None = None,

link: NeedLink | DataForTestLink | DataOfTestCase | None = None,

) -> str:

"""

Return a GitHub link for the given location.

Backwards compatible calling conventions:

- New style: get_github_link(metadata, link)

- Old style: get_github_link(link) # metadata omitted

"""

# Distinguish between new-style and old-style calls.

if isinstance(metadata, moduleInfo):

actual_metadata: moduleInfo | None = metadata

actual_link = link

else:

# Old-style: first argument is actually the link (or None)

actual_metadata = None

# If both metadata and link are provided but metadata is not a moduleInfo,

# prefer the explicitly provided link argument.

actual_link = link if link is not None else metadata

if actual_link is None:

actual_link = DefaultNeedLink()

# If no metadata is available or metadata.hash is falsy, fall back to git-based link.

if actual_metadata is None or not getattr(actual_metadata, "hash", None):

# Local path (//:docs)

return get_github_link_from_git(actual_link)

# Ref-Integration path (//:docs_combo..)

return get_github_link_from_json(actual_metadata, actual_link)

src/extensions/score_source_code_linker/need_source_links.py

src/extensions/score_source_code_linker/needlinks.py

src/extensions/score_source_code_linker/repo_source_links.py

src/extensions/score_source_code_linker/needlinks.py

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

You can also share your feedback on Copilot code review. Take the survey.

src/extensions/score_source_code_linker/helpers.py

src/extensions/score_source_code_linker/generate_source_code_links_json.py

src/extensions/score_source_code_linker/module_source_links.py

src/extensions/score_source_code_linker/__init__.py

scripts_bazel/merge_sourcelinks.py

docs.bzl

src/extensions/score_source_code_linker/need_source_links.py

src/extensions/score_source_code_linker/module_source_links.py

MaximilianSoerenPollak · 2026-03-12T15:54:48Z

Just realized that testlinks also need the module name earlier than source_code_linker extension end state.
As well as the edgecase that it could be that source_code_links are empty but testlinks exists. Then we have 0 idea about the module state.

I need to re-think this approach a bit and see what I can adapt to make this more convenient. I think I have an idea, but unsure.

MaximilianSoerenPollak · 2026-03-13T15:54:20Z

Still a bit to do here, but the architecture is 'working', though not pretty.
Will work on fixing the tests soon, and fixing the finaly bug in the testlinks that the actuall github link is put together wrong

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 14 comments.

You can also share your feedback on Copilot code review. Take the survey.

src/extensions/score_source_code_linker/generate_source_code_links_json.py

src/extensions/score_source_code_linker/helpers.py

src/extensions/score_source_code_linker/needlinks.py

src/extensions/score_source_code_linker/generate_source_code_links_json.py

src/extensions/score_source_code_linker/module_source_links.py

src/extensions/score_source_code_linker/BUILD

src/extensions/score_source_code_linker/helpers.py

scripts_bazel/merge_sourcelinks.py

src/extensions/score_source_code_linker/need_source_links.py

src/extensions/score_source_code_linker/xml_parser.py

src/extensions/score_source_code_linker/tests/test_codelink.py

MaximilianSoerenPollak · 2026-03-17T14:27:08Z

src/extensions/score_source_code_linker/tests/test_codelink.py

+#            ─────────────────────[ Roundtrip Tests ]───────────────────
+
+
+def test_roundtrip_standard_format(tmp_path: Path):


Wait, How is this different than this function:

test_store_and_load_source_code_links

=> Double check, this might have been copy paste errors (copying from older version where top tests was not yet in)

They indeed were duplicates .Deleted test_store_and_load...

src/extensions/score_source_code_linker/tests/test_codelink.py

src/extensions/score_source_code_linker/tests/test_helpers.py

MaximilianSoerenPollak · 2026-03-17T14:30:27Z

src/extensions/score_source_code_linker/tests/test_helpers.py

+
+
+def test_parse_info_from_known_good_no_module_in_json(tmp_path: Path):
+    """Test that assertion works when module not in top level keys"""


Suggested change

"""Test that assertion works when module not in top level keys"""

"""Test that assertion works when 'module' not in top level keys"""

src/extensions/score_source_code_linker/tests/test_helpers.py

MaximilianSoerenPollak · 2026-03-17T14:35:37Z

src/extensions/score_source_code_linker/__init__.py

@@ -21,7 +21,6 @@
 # This whole directory implements the above mentioned tool requirements


I don't like the namings in this file anymore.
As it was build with 'SourceCodeLiniks' as the final step before.
Now that isn't the case anymore so most likely can make this nicer / more viewable again.

src/extensions/score_source_code_linker/__init__.py

MaximilianSoerenPollak · 2026-03-17T14:44:15Z

scripts_bazel/generate_sourcelinks_cli.py

 logger = logging.getLogger(__name__)


+def clean_external_prefix(path: Path) -> Path:


Wonder if this can be used in the testlink cleaning too, or if they are too different.

MaximilianSoerenPollak · 2026-03-17T14:47:43Z

scripts_bazel/generate_sourcelinks_cli.py

+    metadata: MetaData = {
+        "module_name": "",
+        "hash": "",
+        "url": "",
+    }


I think it makes sense to make a 'default metadata' function or something.

This setup seems to be called a lot

MaximilianSoerenPollak · 2026-03-17T14:48:31Z

scripts_bazel/generate_sourcelinks_cli.py

+        if "known_good.json" not in str(file_path) and not metadata_set:
+            metadata["module_name"] = parse_module_name_from_path(file_path)
+            metadata_set = True


Suggested change

if "known_good.json" not in str(file_path) and not metadata_set:

metadata["module_name"] = parse_module_name_from_path(file_path)

metadata_set = True

if "known_good.json" not in str(file_path) and not metadata_set:

# Doing this so we only have to parse the module name once, not every file.

metadata["module_name"] = parse_module_name_from_path(file_path)

metadata_set = True

MaximilianSoerenPollak · 2026-03-17T14:49:51Z

scripts_bazel/merge_sourcelinks.py

+"""
+if bazel-out/k8-fastbuild/bin/external/ in file_path => module is external
+otherwise it's local
+if local => module_name & hash == empty
+if external => parse thing for module_name => look up known_good json for hash & url
+"""


Remove debug strings here.

MaximilianSoerenPollak · 2026-03-17T14:51:15Z

scripts_bazel/merge_sourcelinks.py

    )

    args = parser.parse_args()
+    all_files = [x for x in args.files if "known_good.json" not in str(x)]


Add a comment:
#Known_good.json is getting passed in 'args.files' as well.
#Therefore we filter it out here in order to make following logic cleaner

MaximilianSoerenPollak · 2026-03-17T14:51:54Z

scripts_bazel/merge_sourcelinks.py

+            # If the file is empty e.g. '[]' there is nothing to parse, we continue
+            if not data:
+                continue
+            metadata = data[0]


Do we need a worldview assertion here?
e.g. check if data is a list?

MaximilianSoerenPollak · 2026-03-17T14:52:33Z

scripts_bazel/merge_sourcelinks.py

+                logger.warning(
+                    f"Unexpected schema in sourcelinks file '{json_file}': "
+                    "expected first element to be a metadata dict "
+                    "with a 'module_name' key. "
+                )


Should this be an error instead?

MaximilianSoerenPollak · 2026-03-17T14:53:16Z

scripts_bazel/merge_sourcelinks.py

+            for d in data[1:]:
+                d.update(metadata)
+            assert isinstance(data, list), repr(data)
+            merged.extend(data[1:])


Add comment:
# Taking only the needlinks. As data[0] = metadata we start from 1

MaximilianSoerenPollak · 2026-03-17T15:39:17Z

src/extensions/score_source_code_linker/helpers.py

+
+    # COMBO BUILD
+
+    if str(path).startswith("external/"):


MaximilianSoerenPollak · 2026-03-17T15:41:46Z

src/extensions/score_source_code_linker/module_source_links.py

+
+
+@dataclass
+class ModuleInfo:


Module => Repo

MaximilianSoerenPollak · 2026-03-17T15:52:42Z

src/extensions/score_source_code_linker/module_source_links.py

+# # Pouplate Metadata
+# # Since all metadata inside the Codelinks is the same
+# # we can just arbitrarily grab the first one
+# module_name=need_links.CodeLinks[0].module_name,
+# hash=need_links.CodeLinks[0].hash,
+# url=need_links.CodeLinks[0].url,


src/extensions/score_source_code_linker/needlinks.py

a-zw

Looks good enough to merge to me.

Documentation is still missing but we can address that in a separate PR.

MaximilianSoerenPollak · 2026-03-20T11:45:59Z

Looks good enough to merge to me.

Documentation is still missing but we can address that in a separate PR.

Let me just make sure all the consumer tests & ref-integration pass the test now.

MaximilianSoerenPollak added 2 commits March 11, 2026 20:48

WIP: Local Works somewhat

7dece30

WIP: seems to work in ref & local

ebdb40f

github-project-automation bot added this to INF - Infrastructure Community Mar 11, 2026

github-project-automation bot moved this to Backlog in INF - Infrastructure Community Mar 11, 2026

MaximilianSoerenPollak commented Mar 11, 2026

View reviewed changes

src/extensions/score_source_code_linker/metadata.py Outdated Show resolved Hide resolved

MaximilianSoerenPollak mentioned this pull request Mar 11, 2026

docs testlinks should point to version from bazel_dep #412

Open

MaximilianSoerenPollak requested a review from Copilot March 12, 2026 00:08

Copilot started reviewing on behalf of MaximilianSoerenPollak March 12, 2026 00:08 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

MaximilianSoerenPollak added 7 commits March 12, 2026 13:24

Remove metadata file

7f9f8bb

Formatting & Linting

403b9e9

Known_good required in merge script

bb73a0f

Remove debug print statements

703cc71

Copilot Findings

59f7602

Formatting & Linting

7155441

Basepyright linting

2355aa3

MaximilianSoerenPollak requested a review from Copilot March 12, 2026 13:26

Copilot started reviewing on behalf of MaximilianSoerenPollak March 12, 2026 13:26 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

MaximilianSoerenPollak added 2 commits March 13, 2026 16:51

WIP: Testlinks endlink still wrong

bb475e8

Formatting

996a51e

Formating & Linting & Comments

9a9ebab

MaximilianSoerenPollak requested a review from Copilot March 16, 2026 10:01

Copilot started reviewing on behalf of MaximilianSoerenPollak March 16, 2026 10:01 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Fixing copilot & other findings

0edad55