Skip to content

fix(llmobs): use startup or forced error logs instead of throwing for invalid LLMObs initializations #7885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sabrenner wants to merge 3 commits into
base: master
Choose a base branch
from
+59 −24
Open

fix(llmobs): use startup or forced error logs instead of throwing for invalid LLMObs initializations #7885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1e0ab19
use startup logs for v6 and startup logs enabled, else console.error
sabrenner Mar 30, 2026
a93b518
Merge branch 'master' into sabrenner/do-not-throw-llmobs-agent-connec…
sabrenner Mar 30, 2026
b3569ef
Merge branch 'master' into sabrenner/do-not-throw-llmobs-agent-connec…
sabrenner Apr 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions packages/dd-trace/src/llmobs/constants/text.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,7 @@
module.exports = {
DROPPED_VALUE_TEXT: "[This value has been dropped because this span's size exceeds the 5MB size limit.]",
UNSERIALIZABLE_VALUE_TEXT: 'Unserializable value',
INCOMPATIBLE_INITIALIZATION:
'Cannot send LLM Observability data without a running agent or without both a Datadog API key and site. ' +
'Ensure these configurations are set before running your application.',
}
13 changes: 9 additions & 4 deletions packages/dd-trace/src/llmobs/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
const { channel } = require('dc-polyfill')

const log = require('../log')
const { DD_MAJOR } = require('../../../../version')
const startupLogs = require('../startup-log')
const {
ML_APP,
PROPAGATED_ML_APP_KEY,
Expand All @@ -15,6 +17,7 @@ const LLMObsEvalMetricsWriter = require('./writers/evaluations')
const LLMObsTagger = require('./tagger')
const LLMObsSpanWriter = require('./writers/spans')
const { setAgentStrategy } = require('./writers/util')
const { INCOMPATIBLE_INITIALIZATION } = require('./constants/text')

const spanFinishCh = channel('dd-trace:span:finish')
const evalMetricAppendCh = channel('llmobs:eval-metric:append')
Expand Down Expand Up @@ -66,10 +69,12 @@ function enable (config) {

setAgentStrategy(config, useAgentless => {
if (useAgentless && !(config.apiKey && config.site)) {
throw new Error(
'Cannot send LLM Observability data without a running agent or without both a Datadog API key and site.\n' +
'Ensure these configurations are set before running your application.'
)
if (DD_MAJOR < 6 || !config?.startupLogs) {
Copy link
Copy Markdown
Member

@BridgeAR BridgeAR Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am unsure if I would like to log if startupLogs is deactivated when we activate that by default in the future. I think in that case we should not log.

// eslint-disable-next-line no-console
console.error(INCOMPATIBLE_INITIALIZATION)
} else {
startupLogs.logGenericError(INCOMPATIBLE_INITIALIZATION)
}
}

evalWriter?.setAgentless(useAgentless)
Expand Down
9 changes: 9 additions & 0 deletions packages/dd-trace/src/startup-log.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,14 @@ function logAgentError (agentError) {
}
}

function logGenericError (message) {
if (!config?.startupLogs) {
return
}

warn('DATADOG TRACER DIAGNOSTIC - Generic Error: ' + message)
Copy link
Copy Markdown
Member

@BridgeAR BridgeAR Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Startup log is meant to be a single line message and having two separate entries is changing that contract. I guess what we need is a state to wait for the agent strategy to be ready. That might conflict with other expectations though. @rochdev what would be your take on that?

Copy link
Copy Markdown
Collaborator Author

@sabrenner sabrenner Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooh ok, i saw we had something for an agent error log and was just trying something similar.

Copy link
Copy Markdown
Member

@rochdev rochdev Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that multiple startup logs are now a thing, but I'm not sure if they have to respect a specific structure. I think @bm1549 has the most recent context.

Copy link
Copy Markdown
Contributor

@bm1549 bm1549 Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rochdev is correct - multiple startup logs are a thing now and are used to de-couple different concerns, e.g. an initial I'm alive from integration/agent diagnostics. Since this only applies in an Agentless case, it cannot really be combined with the "failed to reach agent" so imo it's okay

That being said, this seems to point to more of an invalid configuration state since this simplifies to

DD_LLMOBS_AGENTLESS_ENABLED=true
DD_SITE=null
DD_API_KEY=null

@BridgeAR have we considered having invalid state as part of the config system so that we warn users if they provide something like this?

}

/**
* Returns config info without integrations (used by startupLog).
* @returns {Record<string, unknown>}
Expand Down Expand Up @@ -143,4 +151,5 @@ module.exports = {
setSamplingRules,
tracerInfo,
errors,
logGenericError,
}
58 changes: 38 additions & 20 deletions packages/dd-trace/test/llmobs/index.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ const { after, afterEach, beforeEach, describe, it } = require('mocha')
const proxyquire = require('proxyquire')
const sinon = require('sinon')

const { DD_MAJOR } = require('../../../../version')
const { INCOMPATIBLE_INITIALIZATION } = require('../../src/llmobs/constants/text')
const { removeDestroyHandler } = require('./util')

const spanFinishCh = channel('dd-trace:span:finish')
Expand All @@ -23,6 +25,9 @@ describe('module', () => {
let LLMObsEvalMetricsWriterSpy
let fetchAgentInfoStub

/** @type {import('sinon').SinonStub} */
let startupLogStub

beforeEach(() => {
store = {}
logger = { debug: sinon.stub() }
Expand All @@ -40,7 +45,7 @@ describe('module', () => {

fetchAgentInfoStub = sinon.stub()

llmobsModule = proxyquire('../../../dd-trace/src/llmobs', {
const llmobsModuleProxyRequireMeta = {
'./writers/spans': LLMObsSpanWriterSpy,
'./writers/evaluations': LLMObsEvalMetricsWriterSpy,
'../log': logger,
Expand All @@ -56,7 +61,19 @@ describe('module', () => {
fetchAgentInfo: fetchAgentInfoStub,
},
}),
})
}

if (DD_MAJOR < 6) {
startupLogStub = sinon.stub(console, 'error')
} else {
startupLogStub = sinon.stub()

llmobsModuleProxyRequireMeta['../startup-log'] = {
logGenericError: startupLogStub,
}
}

llmobsModule = proxyquire('../../../dd-trace/src/llmobs', llmobsModuleProxyRequireMeta)

removeDestroyHandler()
})
Expand Down Expand Up @@ -117,22 +134,22 @@ describe('module', () => {
describe('with agentlessEnabled set to `true`', () => {
describe('when no api key is provided', () => {
it('throws an error', () => {
assert.throws(() => llmobsModule.enable({
llmobsModule.enable({
llmobs: {
agentlessEnabled: true,
},
}),
{
message: 'Cannot send LLM Observability data without a running agent ' +
'or without both a Datadog API key and site.\n' +
'Ensure these configurations are set before running your application.',
startupLogs: true,
})

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

describe('when no site is provided', () => {
it('throws an error', () => {
assert.throws(() => llmobsModule.enable({ llmobs: { agentlessEnabled: true, apiKey: 'test' } }))
llmobsModule.enable({ llmobs: { agentlessEnabled: true, apiKey: 'test' }, startupLogs: true })

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

Expand Down Expand Up @@ -180,13 +197,17 @@ describe('module', () => {

describe('when no API key is provided', () => {
it('throws an error', () => {
assert.throws(() => llmobsModule.enable({ llmobs: { mlApp: 'test', site: 'datadoghq.com' } }))
llmobsModule.enable({ llmobs: { mlApp: 'test', site: 'datadoghq.com' }, startupLogs: true })

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

describe('when no site is provided', () => {
it('throws an error', () => {
assert.throws(() => llmobsModule.enable({ llmobs: { mlApp: 'test', apiKey: 'test' } }))
llmobsModule.enable({ llmobs: { mlApp: 'test', apiKey: 'test' }, startupLogs: true })

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

Expand Down Expand Up @@ -227,20 +248,17 @@ describe('module', () => {

describe('when no API key is provided', () => {
it('throws an error', () => {
assert.throws(
() => llmobsModule.enable({ llmobs: { mlApp: 'test', site: 'datadoghq.com' } }),
{
message: 'Cannot send LLM Observability data without a running agent ' +
'or without both a Datadog API key and site.\n' +
'Ensure these configurations are set before running your application.',
}
)
llmobsModule.enable({ llmobs: { mlApp: 'test', site: 'datadoghq.com' }, startupLogs: true })

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

describe('when no site is provided', () => {
it('throws an error', () => {
assert.throws(() => llmobsModule.enable({ llmobs: {}, apiKey: 'test' }))
llmobsModule.enable({ llmobs: {}, apiKey: 'test', startupLogs: true })

sinon.assert.calledWith(startupLogStub, INCOMPATIBLE_INITIALIZATION)
})
})

Expand Down
Loading