I’m building real-time audio description: a spoken description track generated at playback time and played over video, so the viewer hears description layered on the film’s own audio. USAGE_ACCESSIBILITY looks like exactly the usage type this is meant for, but I can’t find a worked example anywhere, so I’d rather confirm the approach than build against an assumption.
Four questions:
Is playing a second audio stream via AudioPlaybackStreamBuilder concurrently with W3C media playback (MSE mode, Shaka) a supported pattern? Or is there a different intended approach for layering narration over media?
With AudioUsageType.USAGE_ACCESSIBILITY and StreamDuckingPolicy.SYSTEM, does the media stream duck automatically while the accessibility stream is playing? Or does the app have to call duckVolume on the media stream explicitly? The AudioUsageType docs say USAGE_ACCESSIBILITY “ducks most other audio”, and the accessibility FAQ describes VoiceView ducking video during playback, so I want to check whether an app-created stream gets that same treatment.
Does the Vega Virtual Device emulate audio focus and ducking faithfully, or is that behaviour only correct on physical hardware? This one matters most to me — if ducking only works on a real device, I need to know now rather than at demo time.
Is there any sample using @amazon-devices/keplerscript-audio-lib? Neither vega-video-sample nor vega-audio-sample does — both use react-native-w3cmedia — so I have the API reference but no reference implementation.
Happy to write up whatever I learn if it’d be useful to others.
Follow-up on Q3, with what I found trying to answer it myself.
I wanted to observe the reference behaviour first — the accessibility
FAQ’s line that “when VoiceView speaks during video playback, VoiceView
speech should get priority and video audio should duck” — before testing
my own stream. I can’t, because I can’t enable VoiceView on the VVD by
any of the three methods the FAQ documents:
“Navigate to Accessibility Settings → Toggle VoiceView ON” — there is
no Accessibility section in the VVD’s Settings. Settings contains only
Account Settings, which contains only Amazon Account and Parental
Controls.
“Hold Back and Menu for 2 seconds” — ESC + F2 per the documented VVD
keyboard mapping, window focused. Nothing happens; the config value
stays DISABLED when polled.
Shell: vdcm set “com.amazon.devconf/system/accessibility/
VoiceViewEnabled” “ENABLED” — fails with “No permission for
operation”, via both vega device run-cmd and an interactive vega device shell. vdcm get on the same key works fine and returns
‘DISABLED’, so vdcm itself is working; it’s specifically the write
that’s refused. Both shells run as uid=5000(app_user).
I also tried vsm developer-mode enable (from the run-apps docs, for
full inputd-cli functionality). It returns “enabled” and the VVD reboots,
but the shell’s user context is unchanged and vdcm set is still refused.
The stack does appear to be on the image — vega device installed-packages lists com.amazon.accessibility.tts.service and
com.amazon.a11y_tutorial — so this looks like a missing switch rather
than a missing feature.
So Q3 splits in two:
(a) Does the VVD emulate audio focus and ducking faithfully at all?
(b) Is VoiceView enableable on the VVD by any means? If not, is
accessibility audio behaviour testable on the virtual device, or
does it require physical hardware?
(b) is the one that changes my plan. If accessibility audio can only be
verified on real hardware, I need to source a device rather than find out
at demo time.
One refinement on Q2 as well: the FAQ’s troubleshooting entry says if
ducking isn’t happening, “verify your media player is properly integrated
with the platform audio focus system.” Does the W3C media player
(react-native-w3cmedia, MSE mode via Shaka) register with platform audio
focus automatically, or is there something the app has to do?
Happy to move the VoiceView part to Bug Reports if that’s the better home
— I kept it here since it’s what’s blocking my answer to Q3.
What we can confirm:
The audio-focus/ducking model is documented and works the way you’d expect at the platform level. Per AudioUsageType, a USAGE_ACCESSIBILITY stream is high priority and “ducks most other audio,” while USAGE_MEDIA “can be ducked by higher priority.” So a concurrent accessibility stream ducking your media playback is the intended, documented behavior.
Reference: AudioUsageType - Vega Audio Library
Also - the three VoiceView enable methods you tried come from the WebView / Web App Accessibility guides, which don’t state they apply to the Virtual Device specifically. So your finding that they don’t work on the VVD lines up with the fact that VVD support for those paths isn’t documented.
What we’re checking internally and will confirm:
Whether the VVD faithfully emulates audio focus + ducking at all.
Whether VoiceView can be enabled on the VVD by any supported means - or whether accessibility-audio behavior can only be verified on physical hardware.
Whether the W3C media player (react-native-w3cmedia, MSE/Shaka) registers with platform audio focus automatically, or if the app has to do something explicitly.
We’ll follow up here once we have confirmation from the internal team.
Thank you Ivy, that’s exactly what I needed on the ducking model — good to know a concurrent USAGE_ACCESSIBILITY stream ducking media is the intended behaviour rather than something I’d be relying on by accident.
One small correction on the VoiceView methods, since it affects how they’re classified. The accessibility FAQ does state VVD applicability explicitly, under “How do I test on the device?”:
“VoiceView can also be enabled on the Vega Virtual Device via shell, but for the most accurate results, test on physical hardware.”
The shell method is the one that fails with “No permission for operation”. So it reads as a documented path that doesn’t work, rather than an undocumented one — which I think makes it worth a doc or permissions fix either way.
Also worth connecting to your items (1) and (3): I’ve since found that no media plays on the VVD at all. Your own documented audio-only URL Mode example, run verbatim on a clean helloWorld project, gives loadstart → MEDIA_ERR_SRC_NOT_SUPPORTED with no HTTP request ever leaving the device, and native logs show the app never connects to com.amazon.media.server. Both official media samples also fail on the same device (vega-video-sample SIGSEGV with two RCT-folly versions loaded, vega-audio-sample ANR on launch).
I’m filing that separately in Bug Reports with full repro and tombstones and will link it here. But it may bear directly on (1) and (3): if media playback doesn’t work on the VVD, audio focus and ducking can’t be verified there regardless of whether they’re emulated.
Which leaves my practical question unchanged, and it’s the one that drives a purchase decision: should I be developing audio behaviour on physical hardware? If so I’d rather order a Fire TV stick this week.
Yes, please plan to develop and verify the audio-description/ducking behaviour on physical hardware, that’s the reliable path for a demo.
Please do link the media-playback Bug Report here - that one may be the blocker underneath the VVD audio testing, and we’re still confirming the VVD audio-focus/ducking emulation regardless.