Feature Request: Automatic Model Selection Based on Language Detection

[erklu]

Description

In the current proof of concept, the language model is selected manually by the user—either one of the default Whisper models or a KB-Whisper variant.
To streamline the user experience, we propose an automated default setting where the system selects the most suitable model based on language detection.

⸻

Expected Behavior
• If the user does not select a specific model, the system should automatically detect the language of the input audio.
• Based on the detected language, the system selects the optimal model:
• If the language is Swedish → Use KB-Whisper Large
• If another language is detected → Use Whisper Large (OpenAI)
• If language is unknown or confidence is low → Fallback to Whisper Large

⸻

Technical Guidelines

1. Perform fast language detection on the first 30 seconds of the audio
   • Whisper has a built-in method (language field in the result dict) for this.
   • Alternatively, faster detection tools like langdetect or fastText can be considered.
2. Select the model based on the detected language
   • If sv (Swedish) is detected with high confidence (e.g., >90%), load KB-Whisper Large.
   • Otherwise, default to Whisper Large (OpenAI).
3. Log the model selection decision
   • Store this information in a sidecar file for traceability and to allow analysis/improvement of the fallback logic over time.

[erklu]

Comment: Auto language with code-switching

When introducing auto language we should also support model-switching within the same recording.
This means splitting the audio into smaller chunks and running language detection per chunk.

Each detected language block can then be transcribed with the language locked for higher accuracy.
In low-confidence cases, the system should allow fallback to multiple analyses to reduce errors.

The goal is that the pipeline does not only pick an initial language, but can also handle language switches over time in a stable way.

---

Expected Behavior:

• Audio is segmented into speech chunks.
• Each chunk is language-detected individually.
• Consecutive chunks with the same language are merged into stable blocks.
• Transcription runs per block with the language fixed.
• If language confidence is low, run multiple passes (top-N candidate languages) and select the best match.
• Output should clearly annotate language per segment in the transcript metadata.
• User should explicitly select this mode (e.g. Auto (multi-language))