Rename to NER

[iceljc]

PR Type

Enhancement, Refactoring

---

Description

• Rename tokenizer functionality to NER (Named Entity Recognition) terminology

  • Update API endpoints from /knowledge/tokenize to /knowledge/NER/analyze
  • Rename functions: getTokenizers → getNERAnalyzers, tokenize → nerAnalyze
  • Update type definitions: TokenizeRequest → NERAnalyzeRequest, TokenizeResult → NERResult

• Add Tab key support in textarea inputs for better text editing experience

  • Implement onKeyDown handler to insert tab character instead of focus change
  • Apply to both agent instruction and template textarea components

• Rename document processors API endpoint and function names

  • Change endpoint from /knowledge/document/processors to /knowledge/processors
  • Rename function getKnowledgeDocumentProcessors → getKnowledgeProcessors

• Update UI labels and placeholders to reflect NER terminology

  • Change "Tokenizer" to "Analyzer", "Data Loaders" to "Data Providers"
  • Update select component tags and placeholders accordingly

---

Diagram Walkthrough

flowchart LR
  A["Tokenizer API<br/>Endpoints"] -->|Rename to NER| B["NER API<br/>Endpoints"]
  C["Textarea<br/>Components"] -->|Add Tab Support| D["Enhanced Text<br/>Editing"]
  E["Document Processors<br/>Endpoint"] -->|Simplify Path| F["Unified Processors<br/>Endpoint"]
  B --> G["Updated Service<br/>Functions"]
  D --> H["Agent Instruction &<br/>Template Pages"]
  F --> G

Loading

File Walkthrough

	Relevant files
Enhancement	agent-instruction.svelte Add Tab key support to instruction textarea                            src/routes/page/agent/[agentId]/agent-components/agent-instruction.svelte Add onKeyDown event handler to support Tab key insertion in textarea Insert tab character at cursor position when Tab is pressed Bind on:keydown event to textarea input element +17/-0    agent-template.svelte Add Tab key support to template textarea                                  src/routes/page/agent/[agentId]/agent-components/agent-template.svelte Add onKeyDown event handler for Tab key support in textarea Implement tab character insertion at cursor position Bind on:keydown event to textarea input element Reorder placeholder attribute in Input component +18/-1
Refactoring	+page.svelte Rename tokenizer to NER analyzer terminology                          src/routes/page/knowledge-base/dictionary/+page.svelte Rename all tokenizer-related variables to analyzer terminology Update function names: getTokenizerProviders → getAnalyzerProviders, getTokenizeResult → getAnalysisResult Change UI labels from "Tokenizer" to "Analyzer" and "Data Loaders" to "Data Providers" Update select component tags and placeholder text Rename imported functions from getTokenizers, getTokenizerDataLoaders, tokenize to NER equivalents +38/-38  +page.svelte Rename document processors function                                            src/routes/page/knowledge-base/documents/+page.svelte Rename imported function getKnowledgeDocumentProcessors to getKnowledgeProcessors Update function call in getDocProcessors method +2/-2      knowledgeTypes.js Rename tokenize type definitions to NER                                    src/lib/helpers/types/knowledgeTypes.js Rename typedef TokenizeRequest to NERAnalyzeRequest Rename typedef TokenizeOptions to NERAnalyzeOptions Rename typedef TokenizeResponse to NERResponse Rename typedef TokenizeResult to NERResult +6/-6      knowledge-base-service.js Rename service functions to NER terminology                            src/lib/services/knowledge-base-service.js Rename function getKnowledgeDocumentProcessors to getKnowledgeProcessors Rename function getTokenizers to getNERAnalyzers Rename function getTokenizerDataLoaders to getNERDataLoaders Rename function tokenize to nerAnalyze Update JSDoc type references to use new NER type names +10/-10
Configuration changes	http.js Update API endpoint patterns for NER                                          src/lib/helpers/http.js Update POST request regex pattern from /knowledge/tokenize to /knowledge/NER/analyze Update GET request regex patterns for NER endpoints Change /knowledge/document/processors to /knowledge/processors Change /knowledge/tokenizer/providers to /knowledge/NER/analyzers Change /knowledge/tokenizer/data-providers to /knowledge/NER/data-providers +4/-4      api-endpoints.js Update API endpoint definitions for NER                                    src/lib/services/api-endpoints.js Rename endpoint knowledgeDocumentProcessorsUrl to knowledgeProcessorsUrl Rename endpoint tokenizersUrl to nerAnalyzersUrl Rename endpoint tokenizerDataLoadersUrl to nerDataLoadersUrl Rename endpoint tokenizeUrl to nerAnalyzeUrl Update endpoint paths to reflect NER terminology +4/-4

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
⚪	🎫 No ticket provided Create ticket/issue
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: State desync risk: The new Tab key handler mutates e.target.value directly without updating selected_instruction.instruction (or triggering the same update path as on:input), which can lead to UI/state desynchronization and lost edits. Referred Code /** @param {any} e */ function onKeyDown(e) { if (e.key === 'Tab') { e.preventDefault(); if (e.target) { const start = e.target.selectionStart || 0; const end = e.target.selectionEnd || 0; const value = e.target.value || ''; e.target.value = value.substring(0, start) + "\t" + value.substring(end); e.target.selectionStart = start + 1; e.target.selectionEnd = start + 1; } } } Learn more about managing compliance generic rules or creating your own custom rules
⚪	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Weak target validation: The new onKeyDown handler assumes e.target is a textarea-like element with selectionStart, selectionEnd, and value but only checks that e.target is truthy, so unexpected event targets could cause runtime errors without proper type/shape validation. Referred Code /** @param {any} e */ function onKeyDown(e) { if (e.key === 'Tab') { e.preventDefault(); if (e.target) { const start = e.target.selectionStart || 0; const end = e.target.selectionEnd || 0; const value = e.target.value || ''; e.target.value = value.substring(0, start) + "\t" + value.substring(end); e.target.selectionStart = start + 1; e.target.selectionEnd = start + 1; } } } Learn more about managing compliance generic rules or creating your own custom rules
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Sync state after tab insertion Call the changePrompt handler within the onKeyDown function after inserting a tab to ensure the component's state is synchronized with the textarea's value. src/routes/page/agent/[agentId]/agent-components/agent-instruction.svelte [91-104] function onKeyDown(e) { if (e.key === 'Tab') { e.preventDefault(); if (e.target) { const start = e.target.selectionStart || 0; const end = e.target.selectionEnd || 0; const value = e.target.value || ''; e.target.value = value.substring(0, start) + "\t" + value.substring(end); e.target.selectionStart = start + 1; e.target.selectionEnd = start + 1; + // Sync state after insertion + changePrompt(e); } } } Apply / Chat Suggestion importance[1-10]: 8 __ Why: This suggestion correctly identifies a bug where programmatically changing the textarea's value does not trigger the on:input event, leading to state desynchronization. Calling changePrompt(e) fixes this critical issue.	Medium
	Sync template state on tab Call the changePrompt handler within the onKeyDown function after inserting a tab to ensure the component's state is synchronized with the textarea's value. src/routes/page/agent/[agentId]/agent-components/agent-template.svelte [84-97] function onKeyDown(e) { if (e.key === 'Tab') { e.preventDefault(); if (e.target) { const start = e.target.selectionStart || 0; const end = e.target.selectionEnd || 0; const value = e.target.value || ''; e.target.value = value.substring(0, start) + "\t" + value.substring(end); e.target.selectionStart = start + 1; e.target.selectionEnd = start + 1; + // Sync state after insertion + changePrompt(e); } } } Apply / Chat Suggestion importance[1-10]: 8 __ Why: This suggestion correctly identifies a bug where programmatically changing the textarea's value does not trigger the on:input event, leading to state desynchronization. Calling changePrompt(e) fixes this critical issue.	Medium
Possible issue	Make NER regex case-insensitive Make the regex for the /knowledge/NER/analyze path case-insensitive by adding the i flag to ensure it matches variations like /knowledge/ner/analyze. src/lib/helpers/http.js [180-192] const postRegexes = [ ..., - new RegExp('http(s*)://(.*?)/knowledge/NER/analyze', 'g'), + new RegExp('http(s*)://(.*?)/knowledge/NER/analyze', 'gi'), ..., ]; [To ensure code accuracy, apply this suggestion manually] Suggestion importance[1-10]: 5 __ Why: The suggestion correctly points out that the regex for the /knowledge/NER/analyze path is case-sensitive. Adding the 'i' flag makes the matching more robust, which is a good practice for URL path matching.	Low
Update