Spark-tts-salt deployemnt

[huwenjie333]

This PR deploys the Sunbird/spark-tts-salt model to the Modal platform.

• Deployment dashboard: https://modal.com/apps/sb-modal-ws/main/deployed/spark-tts-salt
• With enforce_eager=False, it takes ~2 min for cold startup, and ~1s for generating a sentence.

Usage Example:

 curl -X POST --get "https://sb-modal-ws--spark-tts-salt-sparktts-generate.modal.run" \
   --data-urlencode "text=I am a nurse who takes care of many people who have cancer." \
   --data-urlencode "speaker_id=248" \
   --output output.wav

Speaker ID:
241: Acholi (female)
242: Ateso (female)
243: Runyankore (female)
245: Lugbara (female)
246: Swahili (male)
248: Luganda (female)

This PR also inlcudes other codes for testing the latency of Sunflower Ultravox model.

[jqug]

Thanks @huwenjie333 this is great. You mention that generating a sentence takes 3-4 seconds, is it possible to find out which part of the processing is taking this time? It should be a few hundred milliseconds, max 1 second I think. At least that's what I was getting when testing on an RTX4090 GPU.

[huwenjie333]

Thanks @huwenjie333 this is great. You mention that generating a sentence takes 3-4 seconds, is it possible to find out which part of the processing is taking this time? It should be a few hundred milliseconds, max 1 second I think. At least that's what I was getting when testing on an RTX4090 GPU.

I can get ~1s when enforce_eager=False and cuda graph is built during the model initialization. However, it increases the cold startup time from 1 min to 1.5-2 mins.

Text chunking time: 0.00s 
Prompt preparation time: 0.00s
Model generation time: 1.05s
Audio decoding time: 0.04s 
Audio saving time: 0.00s 
Total generation time: 1.10s

When enforce_eager=True, the inference becomes ~3s for one sentence.

Text chunking time: 0.00s 
Prompt preparation time: 0.00s
Model generation time: 3.13s
Audio decoding time: 0.06s 
Audio saving time: 0.00s 
Total generation time: 3.19s

[jqug]

Ah OK that makes sense. I think we would want to use enforce_eager=False in practice, as otherwise the latency is too high to make it feel like a natural response. There could be some tricks like to already warm up the endpoint as soon as someone loads the frontend (i.e. it could be starting up in parallel with the LLM).

[jqug]

Does Modal have any persistent storage that could be used to cache the compiled model? Much of that increased cold-start time is model setup, which perhaps is possible to avoid doing every time... (Pointer here on vllm caching)