feat: [BE] Add pagination to published components endpoint

[Mbeaulne]

Description

The GET /api/published_components/ endpoint currently returns all components with no pagination. There are existing TODO comments in the code acknowledging this:

# TODO: Implement paging
# TODO: Implement filtering/search
# TODO: Implement visibility/access control

As the component library grows and a Components tab is added to the homepage, this needs pagination support.

Implementation

Follow the same pagination pattern used by GET /api/pipeline_runs/:

• Add page_size query param (default 20)
• Add page_token query param for cursor-based pagination (base64-encoded offset)
• Add next_page_token to response
• Add total_count to response (component count is much smaller than runs, so a full count is fine here)

Response change

@dataclasses.dataclass(kw_only=True)
class ListPublishedComponentsResponse:
    published_components: list[PublishedComponentResponse]
    next_page_token: str | None = None  # NEW
    total_count: int | None = None      # NEW

Relevant file

• component_library_api_server.py — list() method

[yuechao-qin]

Just trying to get a sense of priority / urgency:

• how many components are being returned now without pagination?
• do we expect the growth rate to be N times in the upcoming weeks?
• can the UI mitigate this in other ways while waiting for this feature?

[Ark-kun]

this needs pagination support

Agreed.

Add total_count to response (component count is much smaller than runs, so a full count is fine here)

I'm a bit skeptical here.