Skip to content

v0.241.001 - GA#835

Merged
paullizer merged 163 commits intomainfrom
Staging
Apr 8, 2026
Merged

v0.241.001 - GA#835
paullizer merged 163 commits intomainfrom
Staging

Conversation

@paullizer
Copy link
Copy Markdown
Contributor

v0.241.001

New Features

  • Fact Memory Instructions and Facts

    • Added a clearer Fact Memory experience that distinguishes always-on Instructions from relevance-based Facts on the profile page and in chat-time recall.
    • Chat responses now surface saved-memory usage more clearly through separate Instruction Memory and Fact Memory Recall thoughts and citations.
    • Admin Settings Latest Features and the user-facing Support > Latest Features page now include Fact Memory guidance and screenshots, and admins can show or hide that announcement from General > User-Facing Latest Features.
    • (Ref: semantic_kernel_fact_memory_store.py, route_backend_chats.py, route_frontend_profile.py, profile.html, support_menu_config.py, admin_settings.html, latest_features.html, fact memory guidance and latest-features coverage)
  • Support Menu and User-Facing Latest Features

    • Added a configurable Support menu for signed-in app users so teams can expose Latest Features and Send Feedback directly in everyday navigation.
    • Admins can rename the Support menu, control the internal feedback-recipient email address, and choose exactly which latest-feature cards are shared with end users from the General tab.
    • The user-facing Latest Features page now mirrors the available admin screenshots more closely, includes clearer guidance about why each feature matters, and adds direct links into Chat, Personal Workspace, or Support destinations where users can try the feature.
    • The Admin Settings Latest Features tab now also calls out the General-tab User-Facing Latest Features checklist so admins can see where feature sharing is configured.
    • (Ref: support_menu_config.py, route_frontend_support.py, latest_features.html, support_send_feedback.html, admin_settings.html, test_support_menu_user_feature.py, support menu configuration and user-facing latest features)
  • MultiGPT Endpoint Management

    • Added multi-endpoint model management so admins can define multiple global model endpoints and users can add personal or group-scoped endpoints when those workspace features are enabled.
    • Personal Workspace and Group Workspace now surface dedicated model endpoint management cards, and agent/model selection can use combined global plus workspace endpoint lists instead of relying on a single shared deployment.
    • The endpoint workflow supports Azure OpenAI and Azure AI Foundry discovery flows, including model fetch/test operations and endpoint-based Foundry agent import.
    • (Ref: route_backend_models.py, route_frontend_admin_settings.py, workspace_model_endpoints.js, admin_model_endpoints.js, workspace.html, group_workspaces.html, test_workspace_multi_endpoints.py)
  • Guided Chat Tutorial

    • Expanded the in-app chat tutorial into a fuller guided walkthrough of the current chat experience so new users can learn the live interface in context.
    • The tutorial now walks through the main chat toolbar, workspace and scope controls, conversation search, advanced search, selection mode, bulk actions, export-related flows, and message-level actions such as retry, edit, feedback, thoughts, and citations.
    • The walkthrough also includes reliability improvements for dynamic chat UI elements, including sidebar expansion, popup alignment, and tutorial-owned surfaces for steps that depend on transient menus.
    • (Ref: chat-tutorial.js, chats.html, chat-sidebar-conversations.js, test_chat_tutorial_selector_coverage.py, chat tutorial walkthrough)
  • Personal Workspace Guided Tutorial

    • Added a dedicated in-app tutorial for Personal Workspace so users can learn document, prompt, agent, action, and tag workflows directly inside the workspace page.
    • The walkthrough covers uploads, search and filters, list and grid views, document details, row actions, bulk selection flows, tag management, prompt management, agent management, and action management.
    • It also includes layout-aware positioning and state-restoration behavior so the overlay remains aligned while tabs, filters, menus, and collapsible sections change during the walkthrough.
    • (Ref: workspace.html, workspace-tutorial.js, test_personal_workspace_tutorial_selector_coverage.py, test_personal_workspace_tutorial_document_flow.py, test_workspace_tutorial_reposition_fix.py, test_workspace_tutorial_layer_order_fix.py)
  • Conversation Completion Notifications

    • Added personal chat completion notifications so users who leave a conversation before the assistant finishes can still see that a response is ready.
    • Notification clicks deep-link back into the completed conversation, and personal conversations now show a green unread dot until the assistant response is opened.
    • The unread state and notification lifecycle are wired into the chat conversation list, sidebar list, and mark-read flow so the indicator clears once the conversation is actually viewed.
    • (Ref: conversation notifications, unread assistant responses, route_backend_chats.py, route_backend_conversations.py, functions_notifications.py, functions_conversation_unread.py, chat-conversations.js, chat-sidebar-conversations.js)
  • Background Chat Completion Away From Chat Page

    • Updated streaming chat execution so assistant responses can continue running after the user leaves the chat page instead of stopping when the browser disconnects from the stream.
    • This keeps final assistant persistence, unread markers, and completion notifications reachable even when users navigate into Personal, Group, or other pages while a reply is still generating.
    • (Ref: background stream execution, BackgroundStreamBridge, route_backend_chats.py, test_chat_stream_background_execution.py, test_streaming_only_chat_path.py)
  • SimpleChat Startup and Scheduler Separation

    • Added deployment guidance for local development, Azure App Service native Python startup, and container runtimes so administrators can choose between direct Gunicorn startup and optional python app.py handoff behavior with clear environment-variable guidance.
    • Extracted the scheduler-style logging timer, approval expiration, and retention loops into a shared background task module and added a dedicated simplechat_scheduler.py entrypoint so scheduled work can run in a separate process or job.
    • This allows the web app to use Gunicorn with workers=2 without duplicating scheduler loops inside every worker process, while keeping a legacy override available for single-process environments.
    • (Ref: app.py, background_tasks.py, simplechat_scheduler.py, SIMPLECHAT_STARTUP.md, test_startup_scheduler_support.py)
  • Deployment, Setup, and Upgrade Documentation Refresh

    • Expanded the deployment guidance so teams can more quickly choose between manual deployment, Azure CLI, Bicep, Terraform, and special-environment setup paths from the main setup documentation.
    • Added a dedicated upgrade guide for existing deployments that separates native Python App Service upgrades from container-based App Service upgrades, including when to use VS Code deployment, ZIP deploy, deployment slots, azd deploy, azd provision, or azd up.
    • Clarified developer and production runtime documentation with explicit local-development guidance, Azure production startup expectations, Gunicorn startup rules, container entrypoint behavior, and scheduler-separation recommendations.
    • (Ref: setup_instructions.md, setup_instructions_manual.md, how-to/upgrade_paths.md, running_simplechat_azure_production.md, running_simplechat_locally.md, SIMPLECHAT_STARTUP.md, deployment and developer documentation)
  • Chat Completion Notifications

    • Added personal chat completion notifications so users who leave a streaming conversation before the assistant finishes now receive a notification when the AI response is ready.
    • Notification clicks deep-link directly back to the completed conversation, and personal conversations now show a green unread dot in both chat conversation lists until that response is opened.
    • The unread state is cleared automatically when the conversation is opened or when the user stays on the chat page through stream completion, keeping the active-view experience clean without adding heartbeat tracking.
    • (Ref: route_backend_chats.py, route_backend_conversations.py, functions_notifications.py, functions_conversation_unread.py, chat-conversations.js, chat-sidebar-conversations.js, chat-streaming.js, test_chat_completion_notifications.py)
  • Configurable Tabular Preview Blob Size Limit

    • Added an admin-configurable maximum blob size for tabular file previews, replacing the previous hardcoded limit. Default is 200 MB.
    • New Tabular Preview Limits card in the Enhanced Citations section of Admin Settings (Citations tab) lets admins increase or decrease the limit based on their compute resources and user population.
    • Setting is stored as tabular_preview_max_blob_size_mb and accepts values from 1 to 1024 MB.
    • (Ref: route_enhanced_citations.py, functions_settings.py, admin_settings.html)
  • Tabular Preview Memory Optimization

    • The /api/enhanced_citations/tabular_preview endpoint no longer loads entire files into a DataFrame. It now uses nrows limits in pandas.read_csv/read_excel to read only the rows needed for the preview, and checks blob size before downloading to reject oversized files early.
    • (Ref: route_enhanced_citations.py)
  • Persistent Conversation Summaries

    • Summaries generated during conversation export are now saved to the conversation document in Cosmos DB for future reuse.
    • Cached summaries include message_time_start and message_time_end — when a conversation has new messages beyond the cached range, a fresh summary is generated automatically.
    • The conversation details modal now shows a Summary card at the top. If a summary exists it displays the content, generation date, and model used. If no summary exists a Generate Summary button with model selector lets users create one on demand.
    • A Regenerate button is available on existing summaries to force a refresh with the currently selected model.
    • New POST /api/conversations/<id>/summary endpoint accepts an optional model_deployment and returns the generated summary.
    • The GET /api/conversations/<id>/metadata response now includes a summary field.
    • Extracted generate_conversation_summary() as a shared helper used by both the export pipeline and the new API endpoint.
    • (Ref: route_backend_conversation_export.py, route_backend_conversations.py, chat-conversation-details.js, functions_conversation_metadata.py)
  • PDF Conversation Export

    • Added PDF as a third export format option alongside JSON and Markdown, giving users a print-ready, visually styled conversation archive.
    • PDF output renders chat messages with colored bubbles that mirror the live chat UI: blue for user messages, gray for assistant messages, green for file messages, and amber for system messages.
    • Message content is converted from Markdown to HTML for rich formatting (bold, italic, code blocks, lists, tables) inside the PDF.
    • Full appendix structure is included (metadata, message details, references, processing thoughts, supplemental messages), matching the Markdown export layout.
    • Rendering uses PyMuPDF's Story API on US Letter paper with 0.5-inch margins and automatic multi-page overflow.
    • Works with both single-file and ZIP packaging; intro summaries are supported in PDF as well.
    • Frontend format step updated to a 3-column card grid with a new PDF card using the bi-filetype-pdf icon.
    • (Ref: route_backend_conversation_export.py, chat-export.js, PyMuPDF Story API, conversation export workflow)
  • Conversation Export Intro Summaries

    • Added an optional AI-generated intro summary step to the conversation export workflow, so each exported chat can begin with a short abstract before the full transcript.
    • Summary model selection now reuses the same model list shown in the chat composer, keeping the export flow aligned with the main chat experience.
    • Works for both JSON and Markdown exports, including ZIP exports where each conversation keeps its own summary metadata.
    • (Ref: route_backend_conversation_export.py, chat-export.js, conversation export workflow)
  • Agent & Action User Tracking (created_by / modified_by)

    • All agent and action documents (personal, group, and global) now include created_by, created_at, modified_by, and modified_at fields that track which user created or last modified the entity.
    • On updates, the original created_by and created_at values are preserved while modified_by and modified_at are refreshed with the current user and timestamp.
    • New optional user_id parameter added to save_group_agent, save_global_agent, save_group_action, and save_global_action for caller-supplied user tracking (backward-compatible, defaults to None).
    • (Ref: functions_personal_agents.py, functions_group_agents.py, functions_global_agents.py, functions_personal_actions.py, functions_group_actions.py, functions_global_actions.py)
  • Activity Logging for Agent & Action CRUD Operations

    • Every create, update, and delete operation on agents and actions now generates an activity log record in the activity_logs Cosmos DB container and Application Insights.
    • Six new logging functions: log_agent_creation, log_agent_update, log_agent_deletion, log_action_creation, log_action_update, log_action_deletion.
    • Activity records include: user_id, activity_type, entity_type (agent/action), operation (create/update/delete), workspace_type (personal/group/global), and workspace_context (group_id when applicable).
    • Logging is fire-and-forget — failures never break the CRUD operation.
    • All personal, group, and admin routes for both agents and actions are wired up.
    • (Ref: functions_activity_logging.py, route_backend_agents.py, route_backend_plugins.py)
  • Tabular Data Analysis — SK Mini-Agent for Normal Chat

    • Tabular files (CSV, XLSX, XLS, XLSM) detected in search results now trigger a lightweight Semantic Kernel mini-agent that pre-computes data analysis before the main LLM response. This brings the same analytical depth previously only available in full agent mode to every normal chat conversation.
    • Automatic Detection: When AI Search results include tabular files from any workspace (personal, group, or public) or chat-uploaded documents, the system automatically identifies them via the TABULAR_EXTENSIONS configuration and routes the query through the SK mini-agent pipeline.
    • Unified Workspace and Chat Handling: Tabular files are processed identically regardless of their storage location. The plugin resolves blob paths across all four container types (user-documents, group-documents, public-documents, personal-chat) with automatic fallback resolution if the primary source lookup fails. A user asking about an Excel file in their personal workspace gets the same analytical treatment as one asking about a CSV uploaded directly to a chat.
    • Six Data Analysis Functions: The TabularProcessingPlugin exposes describe_tabular_file, aggregate_column (sum, mean, count, min, max, median, std, nunique, value_counts), filter_rows (==, !=, >, <, >=, <=, contains, startswith, endswith), query_tabular_data (pandas query syntax), group_by_aggregate, and list_tabular_files — all registered as Semantic Kernel functions that the mini-agent orchestrates autonomously.
    • Pre-Computed Results Injected as Context: The mini-agent's computed analysis (exact numerical results, aggregations, filtered data) is injected into the main LLM's system context so it can present accurate, citation-backed answers without hallucinating numbers.
    • Graceful Degradation: If the mini-agent analysis fails for any reason, the system falls back to instructing the main LLM to use the tabular processing plugin functions directly, preserving full functionality.
    • Non-Streaming and Streaming Support: Both chat modes are supported. The mini-agent runs synchronously before the main LLM call in both paths.
    • Requires Enhanced Citations: The tabular processing plugin depends on the blob storage client initialized by the enhanced citations system. The enable_enhanced_citations admin setting must be enabled for tabular data analysis to activate.
    • (Ref: run_tabular_sk_analysis(), TabularProcessingPlugin, collect_tabular_sk_citations(), TABULAR_EXTENSIONS)
  • Tabular Tool Execution Citations

    • Every tool call made by the SK mini-agent during tabular analysis is captured and surfaced as an agent citation, providing full transparency into the data analysis pipeline.
    • Automatic Capture: The existing @plugin_function_logger decorator on all TabularProcessingPlugin functions records each invocation including function name, input parameters, returned results, execution duration, and success/failure status.
    • Citation Format: Tool execution citations appear in the same "Agent Tool Execution" modal used by full agent mode, showing tool_name (e.g., TabularProcessingPlugin.aggregate_column), function_arguments (the exact parameters passed), and function_result (the computed data returned).
    • End-to-End Auditability: Users can verify exactly which aggregations, filters, or queries were run against their data, what parameters were used, and what raw results were returned — before the LLM summarized them into the final response.
    • (Ref: collect_tabular_sk_citations(), plugin_invocation_logger.py)
  • Assistant Citation Artifact Storage for Large Tabular Payloads

    • Moved large raw tabular and tool citation payloads off the main assistant message document and into linked child artifact records so tool-heavy answers stay compact in primary chat storage.
    • Added helper flows in functions_message_artifacts.py to keep a compact citation summary on the assistant message, externalize the full raw citation payload into assistant_artifact records with assistant_artifact_chunk support for larger payloads, and rehydrate those raw payloads later for exports or deeper inspection.
    • Assistant messages now keep compact summaries such as tool name, reduced arguments, counts, and a few sample rows while the heavy raw citation payload is referenced through artifact_id and raw_payload_externalized=True.
    • Updated chat persistence to store the linked artifact records during message save, excluded those artifact records from normal chat history and conversation views, and updated export flows to stitch the preserved raw payloads back together when needed.
    • This reduced primary assistant message size, lowered the risk of hitting Cosmos DB per-item limits on large tabular responses, reduced heavy citation data carried through normal chat reads, and preserved the full raw evidence for export and debugging.
    • Additional size reductions in the same phase compacted stored citation summaries, dropped noisy tabular citation arguments such as user_id, conversation_id, and source, and removed the duplicate user_message field from assistant message documents.
    • (Ref: functions_message_artifacts.py, route_backend_chats.py, route_backend_conversations.py, route_frontend_conversations.py, route_backend_conversation_export.py, test_assistant_citation_artifact_storage.py, ASSISTANT_CITATION_ARTIFACT_STORAGE_FIX.md)
  • SK Mini-Agent Performance Optimization

    • Reduced typical tabular analysis time from ~74 seconds to an estimated ~30-33 seconds (55-60% reduction) through three complementary optimizations.
    • DataFrame Caching: Per-request in-memory cache eliminates redundant blob downloads. Previously, each of the ~8 tool calls in a typical analysis downloaded and parsed the same file independently. Now the file is downloaded once and subsequent calls read from cache. Cache is automatically scoped to the request (new plugin instance per analysis) and garbage-collected afterward.
    • Pre-Dispatch Schema Injection: File schemas (columns, data types, row counts, and a 3-row preview) are pre-loaded and injected into the SK mini-agent's system prompt before execution begins. This eliminates 2 LLM round-trips that were previously spent on file discovery (list_tabular_files) and schema inspection (describe_tabular_file), allowing the model to jump directly to analysis tool calls.
    • Async Plugin Functions: All six @kernel_function methods converted to async def using asyncio.to_thread(). This enables Semantic Kernel's built-in asyncio.gather() to truly parallelize batched tool calls (e.g., 3 simultaneous aggregate_column calls) instead of executing them serially on the event loop.
    • Batching Instructions: The system prompt now instructs the model to batch multiple independent function calls in a single response, reducing LLM round-trips further.
    • (Ref: _df_cache, asyncio.to_thread, pre-dispatch schema injection in run_tabular_sk_analysis())
  • SQL Test Connection Button

    • Added a "Test Connection" button to the SQL Database Configuration section (Step 3) of the action wizard, allowing users to validate database connectivity before saving.
    • Supports all database types: SQL Server, Azure SQL (with managed identity), PostgreSQL, MySQL, and SQLite.
    • Shows inline success/failure alerts with a 15-second timeout cap and sanitized error messages.
    • New backend endpoint: POST /api/plugins/test-sql-connection.
    • (Ref: route_backend_plugins.py, plugin_modal_stepper.js, _plugin_modal.html)
  • Per-Message Export

    • Added export and action options to the three-dots dropdown menu on individual chat messages (both AI and user messages).
    • Export to Markdown: Downloads the message as a .md file with a role header. Entirely client-side.
    • Export to Word: Generates a styled .docx document via a new backend endpoint (POST /api/message/export-word). Includes Markdown-to-Word formatting (headings, bold, italic, code blocks, lists) and a citations section when present.
    • Use as Prompt: Inserts the raw message content directly into the chat input box for reuse — no clipboard, one click and it's ready to edit and send.
    • Open in Email: Opens the user's default email client with the message pre-filled in the subject and body via mailto:.
    • New options appear below a divider in the dropdown, preserving existing actions (Delete, Retry, Edit, Feedback).
    • (Ref: chat-message-export.js, chat-messages.js, route_backend_conversation_export.py, per-message export)
  • Custom Azure Environment Support in Bicep Deployment

    • Added custom as a supported cloudEnvironment value alongside public and usgovernment, enabling deployment to sovereign or custom Azure environments via Bicep.
    • New Bicep parameters for custom environments: customBlobStorageSuffix, customGraphUrl, customIdentityUrl, customResourceManagerUrl, customCognitiveServicesScope, and customSearchResourceUrl. All of these are automatically populated from az.environment() defaults except customGraphUrl, which must be explicitly provided for custom cloud environments and can be overridden as needed.
    • The cloudEnvironment parameter now defaults intelligently based on az.environment().name, and legacy values (AzureCloud, AzureUSGovernment) are mapped to SimpleChat's expected values (public, usgovernment).
    • Custom environment app settings (CUSTOM_GRAPH_URL_VALUE, CUSTOM_IDENTITY_URL_VALUE, CUSTOM_RESOURCE_MANAGER_URL_VALUE, etc.) are conditionally injected only when azurePlatform == 'custom'.
    • Replaced hardcoded ACR domain logic and auth issuer URLs with dynamic az.environment() lookups for better cross-cloud compatibility.
    • Fixed trailing slash handling in AUTHORITY URL construction in config.py using rstrip('/').
    • (Ref: deployers/bicep/main.bicep, deployers/bicep/modules/appService.bicep, config.py, sovereign cloud support)
  • Redis Key Vault Authentication

    • Added a new key_vault authentication type for Redis, allowing the Redis access key to be retrieved securely from Azure Key Vault at runtime rather than stored directly in settings.
    • Applies across all Redis usage paths: app settings cache (app_settings_cache.py), session management (app.py), and the Redis test connection flow (route_backend_settings.py).
    • Uses retrieve_secret_direct() from functions_keyvault.py to fetch the Redis key by its Key Vault secret name. Respects key_vault_identity for a user-assigned managed identity on the Key Vault client.
    • New admin setting fields: redis_auth_type (values: key, managed_identity, key_vault) and redis_key (used as the Key Vault secret name when key_vault auth type is selected).
    • Files Modified: app_settings_cache.py, app.py configure_sessions, route_backend_settings.py _test_redis_connection, functions_keyvault.py retrieve_secret_direct
  • Cross-Cloud Deployment Improvements

    • Updated the Azure CLI, AZD, Bicep, and Terraform deployment paths to better align with the current SimpleChat runtime configuration and reduce post-deployment manual fixes.
    • Added optional Azure Video Indexer deployment support with cloud-aware defaults, including the correct endpoint and ARM API version handling for Azure Commercial, Azure Government, and registered custom clouds.
    • (Ref: deployers/azure.yaml, deployers/azurecli/deploy-simplechat.ps1, deployers/bicep/main.bicep, deployers/bicep/modules/videoIndexer.bicep, deployers/terraform/main.tf, application/single_app/functions_settings.py)
  • Idle Session Timeout Feature

    • Added a new idle timer that automatically clears the user session after a configurable set time and redirects to the main chat login page.
    • Added a frontend idle warning modal that pops up after a configurable set time, but disappears if the user moves the mouse over the chat window or interacts with the app in any way.
    • Default values are used if the idle logout and warning values are not set.
    • Idle logout and idle warning values are validated and auto-fixed as needed.
    • Added a new admin switch to enable or disable idle session timeout and warning behavior.
    • Timeout and warning inputs are grouped under a toggleable section in General > System Settings.
    • (Ref: application/single_app/templates/admin_settings.html, application/single_app/static/js/admin/admin_settings.js, application/single_app/route_frontend_admin_settings.py, application/single_app/functions_settings.py, application/single_app/app.py, application/single_app/templates/base.html, application/single_app/static/js/idle-logout-warning.js, application/single_app/config.py, functional_tests/test_idle_logout_timeout.py, application/single_app/route_frontend_authentication.py)

User Interface Enhancements

  • Agent Responded Thought — Seconds & Total Duration

    • The "responded" thought now shows time in seconds instead of milliseconds, and clarifies it is the total time from the initial user message (e.g., 'gpt-5-nano' responded (16.3s from initial message)).
    • A request_start_time is now captured at the top of both the non-streaming and streaming chat handlers, so the duration reflects the full request lifecycle — including content safety, hybrid search, and agent invocation — not just the model response time.
    • Applies to all three agent paths: local SK agents (non-streaming), Azure AI Foundry agents, and streaming SK agents.
    • (Ref: route_backend_chats.py, request_start_time, agent responded thoughts)
  • Enhanced Agent Execution Thoughts

    • Added detailed model-level status messages during agent execution, giving users full visibility into each stage of the AI pipeline.
    • Model Identification: A new "Sending to '{deployment_name}'" thought appears immediately after "Sending to agent", showing the exact model deployment being used (e.g., gpt-5-nano).
    • Generating Response: A "Generating response..." thought now appears before the agent begins its invocation loop, matching the existing behavior for non-agent GPT calls.
    • Model Responded with Duration: A "'{deployment_name}' responded ({duration}ms)" thought appears after the agent completes, showing total wall-clock execution time.
    • Applies to all three agent paths: local SK agents (streaming and non-streaming) and Azure AI Foundry agents.
    • Uses the existing generation step type (lightning bolt icon) — no frontend changes required.
    • (Ref: route_backend_chats.py, ThoughtTracker, agent execution pipeline)
  • List/Grid View Toggle for Agents and Actions

    • Added a list/grid view toggle to all four workspace areas: personal agents, personal actions, group agents, and group actions.
    • Grid View: Large cards with type icon, humanized name, truncated description, and action buttons (Chat, View, Edit, Delete as applicable).
    • List View: Improved table layout with fixed column widths (28%/47%/25%), humanized display names, and truncated descriptions with hover tooltips for full text.
    • View Button: New eye-icon button on every agent and action that opens a read-only detail modal with gradient-header summary cards (Basic Information, Model Configuration, Instructions for agents; Basic Information, Configuration for actions).
    • Name Humanization: Display names are now automatically parsed — underscores and camelCase/PascalCase boundaries are converted to properly spaced, title-cased words (e.g., myCustomAgentMy Custom Agent).
    • Persistent Preference: View mode selection (list/grid) is saved per area in localStorage and restored on page load.
    • New shared utility module view-utils.js provides reusable functions for all four workspace areas.
    • (Ref: view-utils.js, workspace_agents.js, workspace_plugins.js, plugin_common.js, group_agents.js, group_plugins.js, workspace.html, group_workspaces.html, styles.css)
  • Chat with Agent Button for Group Agents

    • Added a "Chat" button to each group agent row, allowing users to quickly select a group agent and navigate to the chat page.
    • (Ref: group_agents.js, group_workspaces.html)
  • Hidden Deprecated Action Types

    • Deprecated action types (sql_schema, ui_test, queue_storage, blob_storage, embedding_model) are now hidden from the action creation wizard type selector. Existing actions of these types remain functional.
    • (Ref: plugin_modal_stepper.js)
  • Advanced Settings Collapse Toggle

    • Step 4 (Advanced) content is now hidden behind a collapsible toggle button ("Show Advanced Settings") instead of being displayed by default. Reduces visual noise for most users.
    • For SQL action types, the redundant additional fields UI in Step 4 is hidden entirely since all SQL configuration is already handled in Step 3.
    • Step 5 (Summary) no longer shows the raw additional fields JSON dump for SQL types, since that data is already shown in the SQL Database Configuration summary card.
    • (Ref: _plugin_modal.html, plugin_modal_stepper.js)

Bug Fixes

  • Chat History Citation Replay Improvements

    • Fixed follow-up prompts so prior assistant turns can reuse stored citation results, including tabular tool outputs, instead of relying only on the visible assistant message text.
    • Assistant history replay now hydrates stored citation artifacts and deduplicates repeated cross-sheet tabular calls so later file results, such as Licensing workbook values, remain available to the next turn.
    • History-context diagnostics remain available in message metadata and optional debug citations, while the thoughts timeline stays compact.
    • (Ref: route_backend_chats.py, functions_message_artifacts.py, chat-thoughts.js, chat-messages.js, test_chat_stream_history_context_fix.py, CHAT_STREAM_HISTORY_CONTEXT_FIX.md)
  • Document Revision Visibility and Storage Preservation

    • Fixed same-name document uploads so new revisions now inherit the previous document's editable metadata, including classification, tags, title, abstract, keywords, publication date, authors, and sharing state.
    • Workspace lists and chat search now only use the current revision, while older revisions remain retained for future comparison work instead of staying active in normal workspace flows.
    • Document deletion now offers a choice between deleting only the current revision or deleting all stored revisions for that document family.
    • Blob storage now preserves older source files by keeping the active document at the existing alias path and archiving prior current revisions into a revision-family hierarchy before the alias path is overwritten.
    • (Ref: document revision families, current-only workspace visibility, hybrid blob alias plus archived revision storage, functions_documents.py, functions_search.py, route_enhanced_citations.py, workspace/group/public document flows)
  • Python Runtime Dependency Refresh and Supply-Chain Hardening

    • Continued the requirements hardening work from v0.240.014 by tightening the main application runtime to exact package pins, reducing dependency drift across local development, CI, and Azure deployments to help mitigate supply-chain exposure.
    • Upgraded the Flask runtime stack to Flask==3.1.3 and Werkzeug==3.1.6, and updated the shared Markup import path to markupsafe so the app starts correctly with Flask 3's package boundary changes.
    • Refreshed key runtime dependencies including gunicorn, requests, openai, Markdown, markdown2, azure-ai-projects, azure-ai-agents, pyjwt, pypdf, semantic-kernel, protobuf, redis, pyodbc, PyMySQL, cython, and aiohttp to pick up current security, compatibility, and capability improvements while keeping builds reproducible.
    • (Ref: application/single_app/requirements.txt, application/single_app/config.py, functional_tests/test_flask_markup_import_fix.py, docs/explanation/fixes/FLASK_31_MARKUP_IMPORT_FIX.md)
  • Dependency Pinning and Requirements Hardening

    • Pinned previously floating Python package requirements to exact versions across the main app, UI test, deployer, and external app requirement files to reduce unexpected dependency drift and tighten supply-chain control.
    • Corrected stale external app dependency entries by replacing dotenv with python-dotenv, removing the stdlib-only logging package, removing an unused Flask requirement from the databaseseeder utility, and adding pytest-playwright so the UI test dependency set matches the pytest fixture usage in the test suite.
    • (Ref: application/single_app/requirements.txt, ui_tests/requirements.txt, deployers/bicep/requirements.txt, application/external_apps/databaseseeder/requirements.txt, application/external_apps/bulkloader/requirements.txt)
  • Settings Default Merge Persistence Fix

    • Fixed app settings merge detection in get_settings() where deep_merge_dicts() mutates the existing settings object in place, causing change detection to always evaluate as unchanged.
    • Updated deep_merge_dicts() to return a boolean changed flag and wired get_settings() to call upsert_item() when settings_changed is True, so missing default keys correctly trigger persistence back to Cosmos DB.
    • Added a functional regression test to validate the merge detection and persistence markers.
    • (Ref: application/single_app/functions_settings.py, application/single_app/config.py, functional_tests/test_settings_deep_merge_persistence_fix.py)
  • Pillow PSD Upload Hardening

    • Updated the application to use pillow==12.1.1, moving the app off the vulnerable Pillow range for specially crafted PSD image parsing.
    • Hardened admin logo and favicon uploads so Pillow now only opens the PNG and JPEG formats already allowed by the route, preventing disguised PSD content from being decoded during upload processing.
    • (Ref: application/single_app/requirements.txt, application/single_app/route_frontend_admin_settings.py, functional_tests/test_pillow_psd_upload_hardening.py)
  • Changed-Files GitHub Action Supply Chain Remediation

    • Updated the release-notes pull request workflow to use the patched tj-actions/changed-files@v46.0.1 release after the March 2025 supply chain compromise affecting older tag families.
    • Added a functional regression check to ensure the workflow does not drift back to the known malicious commit or an older vulnerable action reference.
    • (Ref: release-notes-check.yml, test_changed_files_action_version.py, GitHub Actions workflow security, CI dependency pinning)
  • Personal Conversation Notification Scope Detection

    • Fixed a scope-detection bug where personal chat completions could save successfully without creating a completion notification or unread dot when unrelated active workspace state was still present in session.
    • Personal completion-side effects are now determined from the saved conversation type instead of active workspace session values.
    • (Ref: personal chat scope gating, route_backend_chats.py, test_chat_completion_notifications.py)
  • Distributed Background Task Locks

    • Added Cosmos-backed distributed lock documents for approval expiry and retention policy background jobs so duplicate execution is reduced across multiple Gunicorn workers and App Service instances.
    • Kept the current web-app-hosted scheduler model intact so teams can continue running these jobs from the existing App Service while improving cross-worker coordination.
    • Updated the startup documentation and added functional validation for the distributed lock wiring.
    • (Ref: background_tasks.py, SIMPLECHAT_STARTUP.md, test_background_task_distributed_locks.py, test_startup_scheduler_support.py)
  • Background Task Default-On Gating

    • Updated the web runtime background task gate so scheduler loops now start by default even when SIMPLECHAT_RUN_BACKGROUND_TASKS is unset.
    • Only explicit false-like values such as 0, false, no, or off now disable the background loops, which matches the requested deployment behavior.
    • Updated the startup guide and Gunicorn runtime validation test to reflect the new default-on behavior.
    • (Ref: app.py, SIMPLECHAT_STARTUP.md, test_gunicorn_startup_support.py)
  • Gunicorn Production Startup Support

    • Updated the app bootstrap so production deployments can run cleanly under Gunicorn instead of relying on Flask's built-in server, which is a poor fit for long-lived streaming chat requests on App Service.
    • Added a shared Gunicorn config, switched the container entrypoint to Gunicorn, and made application initialization idempotent so startup logic can run safely in multi-worker web processes.
    • Background timer and retention loops are now disabled by default under Gunicorn workers to avoid duplicating scheduler-style threads across workers, while local debug startup continues to use the Flask development server.
    • (Ref: app.py, gunicorn.conf.py, Dockerfile, test_gunicorn_startup_support.py)
  • Streaming-Only Chat Path

    • Updated the first-party chat experience so normal sends, retries, and message edits now use the streaming chat path instead of maintaining a separate non-streaming UI path.
    • Preserved parity-sensitive behavior by extending the streaming flow to finalize image-generation responses correctly and by adding a backend compatibility bridge for retry, edit, and image-generation requests while the legacy /api/chat route remains in transition.
    • Removed the chat-page streaming toggle, updated the UI to treat streaming as required behavior, and added regression coverage to prevent first-party chat modules from drifting back to direct /api/chat calls.
    • (Ref: route_backend_chats.py, chat-messages.js, chat-streaming.js, chat-retry.js, chat-edit.js, chats.html, test_streaming_only_chat_path.py)
  • Embedding Retry-After Wait Time Handling

    • Fixed embedding retries so 429 Too Many Requests responses now honor server-provided wait times from Retry-After style headers instead of always using local backoff timing.
    • This reduces avoidable repeat throttling during document processing, batched embedding generation, and search embedding requests when Azure OpenAI asks the client to wait.
    • The existing exponential backoff behavior remains in place as a fallback when the service does not provide a usable retry delay.
    • (Ref: functions_content.py, embedding retry logic, test_embedding_rate_limit_wait_time.py)
  • SQL Plugin Key Vault Secret Storage

    • New and updated SQL Query and SQL Schema actions now store sensitive values such as connection strings and passwords in Azure Key Vault when Key Vault secret storage is enabled.
    • Editing an existing SQL action now preserves stored Key Vault-backed credentials, including the SQL test connection flow, so users do not need to re-enter unchanged secrets just to validate or save the action.
    • Personal, group, and global action flows now preserve existing secret references during updates, clean them up correctly on delete, and redact secret-bearing plugin values from logs.
    • Existing plaintext SQL action credentials are not backfilled automatically; they move to Key Vault the next time the action is saved while Key Vault storage is enabled.
    • (Ref: functions_keyvault.py, route_backend_plugins.py, plugin_modal_stepper.js, workspace_plugins.js, SQL action configuration)
  • Group/Public Expanded Document Tags

    • Fixed group and public workspace list views so expanding a document now shows its tags, matching the personal workspace experience.
    • The fix adds color-coded tag badges with a No tags fallback in expanded document details without changing the existing backend document APIs.
    • (Ref: group_workspaces.html, public_workspace.js, expanded document details, workspace tag rendering)
  • Agent Save Validation for Round-Tripped Metadata

    • Fixed agent saves failing when an existing personal, group, or global agent was edited and the browser sent back backend-managed audit fields such as created_at, created_by, modified_at, and modified_by.
    • Agent payload sanitization now strips backend-managed audit and Cosmos metadata before schema validation, while preserving server-side tracking during persistence.
    • (Ref: functions_agent_payload.py, route_backend_agents.py, agent schema validation, functional test coverage)
  • Live Tool Invocation Thoughts During Streaming

    • Updated plugin thought handling so the chat can surface an immediate Invoking Plugin.Function thought as soon as a tool starts, instead of waiting until the tool completes.
    • Streaming chat now polls pending thoughts while the response is still in flight, allowing the active status badge to switch from model-sending text to the currently executing plugin call during long-running tools such as WaitPlugin.wait.
    • Completed plugin thoughts still include the richer human-readable summaries for wait, math, and generic plugin executions, and broader plugin coverage remains enabled through auto-wrapping for manifest-loaded plugins.
    • (Ref: plugin_invocation_logger.py, plugin_invocation_thoughts.py, chat-thoughts.js, chat-streaming.js, logged_plugin_loader.py, test_logged_core_plugins.py)
  • Multi-Sheet Workbook Tabular Analysis

    • Fixed multi-sheet Excel workbooks being analyzed from the wrong worksheet during tabular chat responses. Questions that clearly target a specific tab, such as asset values in a workbook with Assets, Balance, and Income sheets, no longer silently default to the first sheet.
    • Tabular runtime analysis now requires explicit sheet_name or sheet_index selection for analytical calls on multi-sheet workbooks, and the SK mini-agent preload now includes workbook sheet inventory and per-sheet schemas so the model can choose the correct worksheet before computing results.
    • Enhanced citations and tabular previews now preserve worksheet context, using Sheet: <name> for sheet-specific references and Location: Workbook Schema for workbook-level schema citations instead of generic Page 1 labels. The tabular preview modal also supports switching between workbook sheets.
    • (Ref: tabular_processing_plugin.py, route_backend_chats.py, route_enhanced_citations.py, chat-enhanced-citations.js, chat-citations.js, chat-messages.js)
  • Tabular Citation Conversation Ownership Check

    • Fixed an IDOR vulnerability on /api/enhanced_citations/tabular where any authenticated user who could guess a conversation_id and file_id could download another user's chat-uploaded tabular files.
    • The endpoint now reads the conversation document from Cosmos DB and verifies that conversation.user_id matches the current user before serving the blob. Returns 403 Forbidden on mismatch and 404 if the conversation does not exist.
    • (Ref: route_enhanced_citations.py, cosmos_conversations_container)
  • Tabular Preview max_rows Parameter Validation

    • The max_rows query parameter on /api/enhanced_citations/tabular_preview was parsed with bare int(), causing a 500 error on non-integer input. Switched to Flask's request.args.get(..., type=int) which silently falls back to the default on invalid input, matching the pattern used by other endpoints.
    • (Ref: route_enhanced_citations.py)
  • Streaming Chat Post-Finalization JSON Sanitization

    • Fixed a repeatable late-stream failure where assistant responses could appear nearly complete and then end with a Stream interrupted warning during final persistence.
    • Normalized non-finite numeric values from citation payloads before assistant messages, assistant artifacts, and terminal chat payloads are written, preventing Cosmos DB from rejecting invalid JSON.
    • This improves reliability for streaming chat, compatibility streaming, and the standard JSON response path when tool or search citations include sparse or tabular numeric values.
    • (Ref: functions_message_artifacts.py, route_backend_chats.py, test_chat_post_stream_json_sanitization.py, post-stream citation sanitization)
  • On-Demand Summary Generation — Content Normalization Fix

    • Fixed the POST /api/conversations/<id>/summary endpoint failing with an error when generating summaries from the conversation details modal.
    • Root cause: message content in Cosmos DB can be a list of content parts (e.g., [{type: "text", text: "..."}]) rather than a plain string. The endpoint was passing the raw list as content_text, which either stringified incorrectly or produced empty transcript text.
    • Now uses _normalize_content() to properly flatten list/dict content into plain text, matching the export pipeline's behavior.
    • (Ref: route_backend_conversations.py, _normalize_content, generate_conversation_summary)
  • Export Summary Reasoning-Model Compatibility

    • Fixed export intro summary generation failing or returning empty content with reasoning-series models (gpt-5, o1, o3) through a series of incremental fixes: using developer role instead of system for instruction messages, removing all max_tokens / max_completion_tokens caps so the model decides output length naturally, and adding null-safe content extraction for None responses.
    • Summary now includes ALL messages (user, assistant, system, file, image analysis) for full context, with a simplified prompt producing 1-2 factual paragraphs.
    • Added detailed debug logging showing message count, character count, model name, role, and finish reason.
    • (Ref: route_backend_conversation_export.py, _build_summary_intro, generate_conversation_summary)
  • Conversation Export Schema and Markdown Refresh

    • Fixed conversation exports lagging behind the live chat schema. JSON exports now include processing thoughts, normalized citations, and the raw document/web/tool citation buckets stored with assistant messages.
    • Fixed Markdown exports being too flat and text-heavy by reorganizing them into a transcript-first layout with appendices for metadata, message details, references, thoughts, and supplemental records.
    • Fixed exported conversations including content that no longer matched the visible chat by filtering deleted messages and inactive-thread retries, then reapplying thread-aware ordering before export.
    • (Ref: route_backend_conversation_export.py, test_conversation_export.py, conversation export rendering)
  • Export Tag/Classification Rendering Fix

    • Fixed conversation tags and classifications rendering as raw Python dicts (e.g., {'category': 'model', 'value': 'gpt-5'}) in both Markdown and PDF exports.
    • Tags now display as readable category: value strings, with smart handling for participant names, document titles, and generic category/value pairs.
    • (Ref: route_backend_conversation_export.py, _format_tag helper, Markdown/PDF metadata rendering)
  • Export Summary Error Visibility

    • Added debug_print and log_event logging to all summary generation error paths, including the empty-response path that previously failed silently.
    • The actual error detail is now shown in both Markdown and PDF exports when summary generation fails, replacing the generic "could not be generated" message.
    • (Ref: route_backend_conversation_export.py, _build_summary_intro, export error rendering)
  • Content Safety for Streaming Chat Path

    • Added full Azure AI Content Safety checking to the streaming (/api/chat/stream) SSE path, matching the existing non-streaming (/api/chat) implementation.
    • Previously, only the non-streaming path performed content safety analysis; streaming conversations bypassed safety checks entirely.
    • Implementation includes: AnalyzeTextOptions analysis, severity threshold checking (severity ≥ 4 blocks the message), blocklist matching, persistence of blocked messages to cosmos_safety_container, creation of safety-role message documents, and proper SSE event delivery of blocked status to the client.
    • On block, the streaming generator yields the safety message and [DONE] event, then stops — preventing any further LLM invocation.
    • Errors in the content safety call are caught and logged without breaking the chat flow, consistent with the non-streaming behavior.
    • (Ref: route_backend_chats.py, streaming SSE generator, AnalyzeTextOptions, cosmos_safety_container)
  • SQL Schema Plugin — Eliminate Redundant Schema Calls

    • Fixed agent calling get_database_schema twice per query even though the full schema was already injected into the agent's instructions at load time.
    • Root cause: The @kernel_function descriptions in sql_schema_plugin.py said "ALWAYS call this function FIRST," which overrode the schema context already available in the instructions.
    • Updated all four function descriptions (get_database_schema, get_table_schema, get_table_list, get_relationships) to use the resilient pattern: "If the database schema is already provided in your instructions, use that directly and do NOT call this function."
    • This eliminates ~400ms+ of unnecessary database round trips per query and aligns with the same pattern already used in sql_query_plugin.py.
    • (Ref: sql_schema_plugin.py, @kernel_function descriptions, schema injection)
  • SQL Schema Plugin — Empty Tables from INFORMATION_SCHEMA

    • Fixed get_database_schema returning 'tables': {} (empty) despite the database having tables, while relationships were returned correctly.
    • Root cause: SQL Server table/column enumeration used INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.COLUMNS views, which returned empty results in the Azure SQL environment. Meanwhile, the relationships query used sys.foreign_keys/sys.tables/sys.columns catalog views which worked perfectly.
    • Migrated all SQL Server schema queries to use sys.* catalog views consistently: sys.tables/sys.schemas for table enumeration, sys.columns with TYPE_NAME() for column details, and sys.indexes/sys.index_columns for primary key detection.
    • Fixed pyodbc.Row handling throughout the plugin — removed all isinstance(table, tuple) checks that could fail with pyodbc Row objects, replaced with robust try/except indexing.
    • This enables the full schema (tables, columns, types, PKs, FKs) to be injected into agent instructions, allowing agents to construct complex multi-table JOINs for analytical queries.
    • (Ref: sql_schema_plugin.py, sys.tables, sys.columns, sys.indexes, pyodbc.Row handling)
  • SQL Query Plugin — Auto-Create Companion Schema Plugin

    • Fixed the remaining issue where SQL-connected agents still asked for clarification instead of querying the database, even after description improvements.
    • Root cause: Agents configured with only a sql_query action never had a SQLSchemaPlugin loaded in the kernel. The descriptions demanded calling get_database_schema — a function that didn't exist — creating an impossible dependency that caused the LLM to ask for clarification.
    • LoggedPluginLoader now automatically creates a companion SQLSchemaPlugin whenever a SQLQueryPlugin is loaded, using the same connection details. This ensures schema discovery is always available.
    • Updated @kernel_function descriptions to be resilient: "If the database schema is provided in your instructions, use it directly. Otherwise, call get_database_schema." This dual-path approach works whether schema is injected via instructions or available via plugin functions.
    • Added fallback in _extract_sql_schema_for_instructions() to also detect SQLQueryPlugin instances and create a temporary schema extractor if no SQLSchemaPlugin is found.
    • (Ref: logged_plugin_loader.py, sql_query_plugin.py, semantic_kernel_loader.py)
  • SQL Query Plugin Schema Awareness

    • Fixed agents connected to SQL databases asking users for clarification about table/column names instead of querying the database directly.
    • Root cause: SQL Query and SQL Schema plugin @kernel_function descriptions were generic with no workflow guidance, agent instructions had no database schema context, and the two plugins operated independently with no linkage.
    • Rewrote all @kernel_function descriptions in both SQL plugins to be prescriptive workflow guides (modeled after the working LogAnalyticsPlugin), explicitly instructing the LLM to discover schema first before generating queries.
    • Added auto-injection of database schema into agent instructions at load time — when SQL Schema plugins are detected, the full schema (tables, columns, types, relationships) is fetched and appended to the agent's system prompt.
    • Added new query_database(question, query) convenience function to SQLQueryPlugin for intent-aligned tool calling.
    • Enabled the SQL-specific plugin creation path in logged_plugin_loader.py (was previously commented out).
    • (Ref: sql_query_plugin.py, sql_schema_plugin.py, semantic_kernel_loader.py, logged_plugin_loader.py)
  • Chat-Uploaded Tabular Files Now Trigger SK Mini-Agent in Model-Only Mode

    • Fixed an issue where tabular files (CSV, XLSX, XLS, XLSM) uploaded directly to a chat conversation were not analyzed by the SK mini-agent when no agent was selected. The model would describe what analysis it would perform instead of returning actual computed results.
    • Root Cause: The mini SK agent only triggered from search results, but chat-uploaded files are stored in blob storage and not indexed in Azure AI Search. Additionally, the streaming path completely ignored file role messages in conversation history.
    • Fix: Both streaming and non-streaming chat paths now detect chat-uploaded tabular files during conversation history building and trigger run_tabular_sk_analysis(source_hint="chat") to pre-compute results. The streaming path also now properly handles file role messages (tabular and non-tabular) matching the non-streaming path's behavior.
    • (Ref: route_backend_chats.py, run_tabular_sk_analysis(), collect_tabular_sk_citations())
  • Group SQL Action/Plugin Save Failure

    • Fixed group SQL actions (sql_query and sql_schema types) failing to save correctly due to missing endpoint placeholder. Group routes now apply the same sql://sql_query / sql://sql_schema endpoint logic as personal action routes.
    • Fixed Step 4 (Advanced) dynamic fields overwriting Step 3 (Configuration) SQL values with empty strings during form data collection. SQL types now skip the dynamic field merge entirely since Step 3 already provides all necessary configuration.
    • Fixed auth type definition schemas (sql_query.definition.json, sql_schema.definition.json) only allowing connection_string auth type, blocking user, identity, and servicePrincipal types that the UI and runtime support.
    • Fixed __Secret key suffix mismatch in additional settings schemas where connection_string__Secret and password__Secret didn't match the runtime's expected connection_string and password field names. Also removed duplicate azuresql enum value.
    • (Ref: route_backend_plugins.py, plugin_modal_stepper.js, sql_query.definition.json, sql_schema.definition.json, sql_query_plugin.additional_settings.schema.json, sql_schema_plugin.additional_settings.schema.json)
  • Workspace Model Endpoint Scope Gate Enforcement

    • Fixed personal and group workspace model discovery and model test routes so they now enforce the same custom-endpoint feature gates as the corresponding endpoint management routes.
    • Restored the intended endpoint modal workflow so users can still fetch and test models before saving a new personal or group endpoint when those scope features are enabled.
    • Requests that reference a saved endpoint now resolve against the caller's authorized persisted endpoint configuration instead of allowing raw request payloads to override stored settings.
    • (Ref: route_backend_models.py, workspace_model_endpoints.js, test_model_endpoint_scope_gate_enforcement.py, model endpoint scope gating)
  • Workspace Agent View Consistency

    • Fixed personal and group workspace agent lists so table-view actions now use the same button order, making agent management behavior more predictable across both workspaces.
    • Fixed group workspace agent grid cards so editable group agents once again show Edit and Delete actions when the current user has permission to manage them.
    • Fixed personal workspace agent table layout so action buttons stay inside the table instead of overflowing past the Actions column.
    • (Ref: workspace.html, workspace_agents.js, group_agents.js, view-utils.js, test_workspace_agent_views_consistency.py)
  • MultiGPT Endpoint Key Vault Secret Storage and Foundry Fetch Reliability

    • MultiGPT endpoint secrets such as API keys and service principal client secrets now move into Azure Key Vault when Key Vault secret storage is enabled, instead of remaining in saved endpoint payloads.
    • Endpoint fetch, test, Foundry listing, and runtime execution now resolve stored secrets server-side by endpoint ID, so reopening an endpoint no longer depends on the browser still holding plaintext credentials.
    • Fixed a follow-up regression in Foundry model discovery where sync fetch routes could fail with 'coroutine' object has no attribute 'token' because async credentials were being reused in a synchronous token acquisition path.
    • (Ref: functions_keyvault.py, functions_settings.py, route_backend_models.py, route_frontend_admin_settings.py, semantic_kernel_loader.py, foundry_agent_runtime.py, admin_model_endpoints.js, workspace_model_endpoints.js, test_model_endpoints_key_vault_secret_storage.py, test_foundry_model_fetch_sync_credentials.py)
  • Legacy Office Binary Upload Support

    • Added native OLE-based support for older Word .doc and PowerPoint .ppt files instead of relying on OOXML-only assumptions during processing.
    • Legacy .doc uploads now extract available metadata and follow the same shared document-processing workflow used for richer Office files, so enhanced citations and final metadata extraction stay consistent when those features are enabled.
    • Legacy .ppt uploads now extract slide text and available summary metadata from the OLE presentation streams while keeping the same enhanced-citation and final-metadata workflow used by .pptx uploads.
    • .pptx uploads now also populate presentation metadata such as title, author, subject, and keywords during the initial metadata update when metadata extraction is enabled.
    • (Ref: functions_content.py, functions_documents.py, test_legacy_doc_ole_extraction.py, test_legacy_ppt_ole_extraction.py, legacy Office OLE support and metadata parity)

paullizer and others added 30 commits September 22, 2025 20:20
* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features
Updated else if for AUTHORITY
* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>
Bionic711 and others added 29 commits January 23, 2026 17:03
* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)
* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)
* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

---------

Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

---------

Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* fixed retention policy runtime bug and sidebar bug (#672)

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* Update release_notes.md

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* Update release_notes.md

* fixed sidebar race condition (#679)

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustio…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
@paullizer paullizer merged commit e556be3 into main Apr 8, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants