fix(extract_audio) correct extraction to WAV

[ekuleshov]

Description

Related Issue: follow up fixes for #65 to correctly handle extraction to the wav format

• ios and macos implementations returning m4a instead of wav, the code was also crashing on some missing config keys in native code

• android implementation also returned some non-wav format. Unfortunately I couldn't think of an easy way to encode wav on android other than creating a wav file writer to save converted PCM audio

• added mime lookup into the example app to verify what audio format is returned

On a side note - the video used in the example has a 6-track audio. It is probably not the most representative example.

Type of Change

• ✨ New feature (non-breaking change which adds functionality)
• 🛠️ Bug fix (non-breaking change which fixes an issue)
• ❌ Breaking change (fix or feature that would cause existing functionality to change)
• 🧹 Code refactor
• ✅ Build configuration change
• 📝 Documentation
• 🗑️ Chore

[hm21]

Thanks for working on the PR. I left a few review comments, and I noticed that some integration tests in audio_extract_test.dart are failing on android. Once those are fixed, we can merge it.

Let me know if you need any help.

[ekuleshov]

noticed that some integration tests in audio_extract_test.dart are failing on android. Once those are fixed, we can merge it.

So you aren't against this approach then. I'll work through your comments.

While having export to wav is useful, most audio processing tools need raw pcm data downsampled to 16-bit and certain sample rate to make it work with. I ended up using the Wav package to decode it back to pcm.

Adding support for the raw format and allowing to specify required sample rate and bit depth would make things simpler and more optimal. Perhaps in a separate PR?

[ekuleshov]

I also having some odd issue (at least on macos, have to verify other platforms), and unsure what is the root cause is...

Download some video from instagram, load it into Audacity audio editor and view waveform and spectrogram. Then extract audio to m4a (I think to wav does the same) and load extracted audio into Audacity. Note some time marker in the original audio and find it in the extracted audio - the marker times don't match, by 0.3..0.7 seconds (different between different videos). 🤷🏻

Extracting wav audio with ffmpeg producing matching time markers.

Do you have any thoughts on this? Or should I create an issue for the backlog? Could it be related to some sample rate rounding/miscalculation or something like that?

[hm21]

So you aren't against this approach then. I'll work through your comments.

Yes, I like it, and wav is definitely a useful format.

Adding support for the raw format and allowing to specify required sample rate and bit depth would make things simpler and more optimal. Perhaps in a separate PR?

Yeah, definitely helpful, and I agree that doing it in a separate PR would be great.

Do you have any thoughts on this? Or should I create an issue for the backlog? Could it be related to some sample rate rounding/miscalculation or something like that?

That’s honestly a very interesting issue. I also think it could be a rounding or miscalculation, like you mentioned, but I’d be really interested to know if you’re seeing the same problem on Android.

[ekuleshov]

some integration tests in audio_extract_test.dart are failing on android...

It is also failing for me on macos:

         Expected: audio/aac
            Actual: audio/mp4

Probably due to mime lookup assertion I've added to make sure returned format is correct. Could use some help to align the correct mime types in the audio_format_model.dart (or in the platform code that handles them).

Another misalignment is that Flutter's mime package reports wav file mime types as audio/x-wav instead of audio/wav. I wonder if we should change wav mime type to x-wav one to align... Or may have to submit a but to mime package on that.

BTW, is the AudioFormat enum not using recent enhanced enums for historical reasons? Something like:

enum AudioFormat {
  /// MP3 format - widely supported, good compression.
  /// Supported on: Android only (not supported on iOS/macOS)
  mp3('mp3', 'audio/mpeg'),

  /// AAC format - high quality, modern codec.
  /// Supported on: Android, iOS, macOS
  aac('aac', 'audio/aac'),
  ...
  
  const AudioFormat(this._extension, this.mimeType);
  
  final String _extension;
  String get extension =>
      this == .aac && !kIsWeb && (Platform.isIOS || Platform.isMacOS) ? 'm4a' : extension;
  
  final String mimeType;
  ...
  
    

[hm21]

Could use some help to align the correct mime types in the

Sure, I just fixed all the review comments and fixed all the failing tests, so it's ready to merge :)

BTW, is the AudioFormat enum not using recent

yeah when building a package, the main goal is usually to keep it as compatible as possible with older versions. But enhanced enums have already been around for quite a while, probably 3 or 4 years. So I’m not planning to make the editor more backward compatible and will go with dart 3.4, which means we can go ahead and update it.

[ekuleshov]

should probably add com.microsoft.waveform-audio for wav

[ekuleshov]

probably should add com.microsoft.waveform-audio for wav

[ekuleshov]

@hm21 btw, I pushed some audio alignment fixes yesterday to make sure start times declared on the audio tracks are used.

Also, for the AudioFormat.mimeType it might be a good idea to change wav mimetype to audio/x-wav to align with the standard Dart's mime package. Mainly because the audio/wav is still not a standard iana mimetype - https://www.iana.org/assignments/media-types/media-types.xhtml

[ekuleshov]

@hm21 anyhow, please merge it when you see fit and thank you for resolving these issues.

[hm21]

@ekuleshov yeah I think the things you mentioned above would be great to implement, so feel free to add them. After that, we can merge it

[ekuleshov]

@ekuleshov yeah I think the things you mentioned above would be great to implement, so feel free to add them. After that, we can merge it

I've pushed those changes in. Please take a look.

BTW, are there instructions somewhere how to run integration tests from example/integration_test on different platforms?

[hm21]

Great, thanks for updating it. I’ll merge it now.

BTW, are there instructions somewhere how to run integration tests from example/integration_test on different platforms?

Yes i recommend you the flutter documentation here