Skip to content

Unexpected OpenTelemetry logging from weather-service #199

@esnible

Description

@esnible

The weather service is coded to log its instrumentation startup:

    logger.info("Setting up OpenTelemetry observability")
    logger.info(f"  Service: {service_name}")

However, those logs to not appear in kubectl logs ...

When it receives an A2A query from Kagenti, the weather service logs

DEBUG:opentelemetry.instrumentation.openai.shared:OpenLLMetry failed to trace in _set_response_attributes, error: Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/openai/utils.py", line 147, in sync_wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/openai/shared/__init__.py", line 232, in _set_response_attributes
    prompt_tokens_details = dict(usage.get("prompt_tokens_details", {}))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable

This error is unexpected. It has no stack trace and is thus hard to troubleshoot.

Perhaps related to the above, in Phoenix the root spans show 0 cumulative tokens, but if a root span is expanded its inner ChatOpenAI span shows token usage.

cc @Ladas

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions