Introduction to Android Audio Effects
Audio effects are used to enhance audibility of sound or to improve a specific audio feature. Android source code comes with few default audio effects like bassboost, equalizer, loudness enhancer which are then applied to the track using AudioFlinger while playing audio.
Android framework uses a vast variety of algorithms to provide the enhanced user experience in video and audio use cases. Often it is required to add custom audio effects to achieve additional benefits over the default audio effects. If we concentrate on audio effects in Android, we are having different options to come up with our own audio processing algorithms which could be applied to audio streams using the newly added effect library. Using audio effect framework, audio processing can be done either by using different android effect libraries or offloading audio streams to a third-party library.
This document explains ways of effect integration and how we can integrate effects into android.
New Audio Effect Integration
Here, we are discussing how a new android effect can be integrated into the existing android framework. Android effect integration can be done in two steps:
1. Implementing a new effect library and integrating it into android effect framework
2. Integrate with JNI and Java Implementations to connect the applications with native systems
We will be concentrating here on the effect integration in android native side.
Effect Flow from Upper Java Layer to Native Effect Library
Flow of effect handle creation from upper layer can be listed as follows:
1. From application code: MainActivity.java, Object of audio effects need to be created that invoke constructor of particular effect’s derived class.
2. From constructor of derived class, constructor of base class from AudioEffect.java will get called.
3. AudioEffect.java will take the flow to native framework through JNI layer: android_media_AudioEffect.cpp
4. Then the call reaches AndroidEffect.cpp ‘s set API. This is the entry point to native effect framework.
5. AudioEffect.cpp communicates to AudioFlinger.cpp
6. Audioflinger service will be calling create_effect_l of Threads.cpp implementation
7. Then EffectCreate of Effects.cpp will get called
8. From Effects.cpp, flow goes to EffectCreate of EffectFactory.cpp
9. Particular effect details - sampleEffect will be loaded from configuration files and particular effect handler will get created
Figure 1: Effect Flow Diagram
New Effect Implementation – Native Layer
UUID and Type
As a first step to start effect library implementation, we need to generate a UUID (universally unique identifier) and a type. Here UUID will be particular for effect implementation and the type will be another UUID to the OpenSL ES interface implemented by this effect. This can be kept even if there is no OpenSL ES interface implementation for the particular library. UUIDs can be generated online using the link.
Effect Library Implementation
To start with the effect library implementation, a new directory is created in frameworks/av/media/libeffects. If the effect name is ‘sampleEffect’, directory can be created with the name sampleEffect. Effect directory will include implementation specific c/cpp files, corresponding header files and Android compilation file– Android.bp/Android.mk. Let’s assume the file name as EffectSample.c, EffectSample.h and Android.mk. Following are the main components in each file.
- Header file:
- Definition of a structure to store effect attributes
- Declarations of effect interface APIs
- Declarations of effect interface control APIs
- c/cpp file:
- Definitions of effect interface APIs
- Definitions of effect control APIs
- Android.bp/Android.mk:
- Compiling options
Effect Attributes Structure
Figure 2: Effects Attributes Structure
This structure will contain a pointer to another structure containing the addresses of all effect interface control APIs(itfe), a configuration structure which will get stored with all the configuration details of the effect flow, a context pointer particularly for the effect and an id of type integer. We could use this id field to store the session id.
Library Implementation
All android native effect libraries follow a basic method of implementation. Following structure definitions and API implementations come under this library implementation design.
AUDIO_EFFECT_LIBRARY_INFO_SYM
All effect libraries need to implemented as audio_effect_library_t structure named AUDIO_EFFECT_LIBRARY_INFO_SYM
Figure 3: AUDIO_EFFECT_LIBRARY_INFO_SYM
Here macro definitions of tag and versions will be defined in /hardware/libhardware/include/hardware/audio_effect. Apart from this, we could add name of new effect library and implementer details. Effect interface API pointers will also get added here.
Effect Descriptor Structure
There will be an effect_descriptor_t structure specific to the new effect. This structure will be having a unique uuid, type , effect flags and variables for cpu load and memory usage.
Figure 4: Effect Descriptor Structure
Effect Interface APIs
There are three interface APIs:
Create
Release
Get descriptor
Create function:
This function will create an instance of the effect of the particular uuid. And it will allocate the resources for the instance and will provide a handle of effect_handle_t.
I/O parameters:
Uuid : Pointer to the effect uuid
sessionId : Audio session id to which effect will be attached
ioId : Output or input stream id the effect is directed to in audio HAL
pHandle : Address where to return the effect interface handle
Figure 5: Create Function
Release function:
This function releases all the resources related to the particular effect handle.
I/O parameters:
handle: Effect interface handle
Figure 6: Release Function
Get descriptor function:
This function gets the effect descriptor structure corresponding to the given uuid.
I/O parameters:
uuid : Pointer to the effect uuid
pDescriptor : Address where to return the effect descriptor
Figure 7: Get Descriptor Interface Function
Effect Control APIs
Get descriptor function:
This function returns the effect descriptor. It will return the effect handle to the address location which will get passed as input parameter to the function.
I/O parameters:
Self : Handle to effect interface
Descriptor : Address to return the effect descriptor
Figure 8: Get Descriptor Control Function
Command function:
This function helps to send commands to effect engine and receives to and fro responses from effect.
I/O parameters:
Self handle to effect interface
cmdCode : Command code
cmdSize : Size of command in bytes
pCmdData : Pointer to command data
replySize : Maximum size of reply data
pReplyData : Pointer to reply data
Few of the commands defined in the effect framework are:
EFFECT_CMD_INIT : initialize effect engine
EFFECT_CMD_SET_CONFIG : configure effect engine
EFFECT_CMD_RESET : reset effect engine
EFFECT_CMD_ENABLE : enable effect process
EFFECT_CMD_DISABLE : disable effect process
EFFECT_CMD_SET_PARAM : set parameter immediately
EFFECT_CMD_GET_PARAM : get parameter
Figure 9: Command Function
Process function:
Process function as the name indicates processes the audio data. Function will get input audio samples of particular specification and will provide the processed data of specifications corresponding to output buffer. We could add our processing algorithm implementation here. There will an input and output buffer descriptors of type audio_buffer_t as function parameters. If these buffer descriptors are not specified by the function, buffer or buffer provider function installed by the command EFFECT_CMD_SET_CONFIG is to be used. Once EFFECT_CMD_ENABLE is received, effect framework will start to call the process function until EFFECT_CMD_DISABLE is received. When this command is received –ENODATA is to be returned so that frame can stop the calling of process function.
I/O parameters:
self : Handle to the effect interface
inBuffer : Input audio buffer
outBuffer : Output audio buffer
Figure 10: Process Function
Effect library compilation
Effect libraries are compiled to create a shared library of .so extension. This compiled library is to be stored to the built path: /system/lib/soundfx.
Figure 11: Compilation File - Android.mk
Adding Effect Details to Configuration File
When we add a new effect library to make effect visible to the framework, we need to add the effect details in the configuration file: audio_effects.conf. This configuration file will give the details of audio effects supported by platform. We could find details such as uuid, library name and library path here. Similarly new effect details are to be added to the file which can be found in the path: /frameworks/av/media/libeffects/data/audio_effects.conf.
Filed: libraries will point to the effect library .so file path and the field: effects will give the mapping details of .so file name, effect name and the corresponding uuid. In the android device this configuration file can be found either in /system/etc/ or in /vendor/etc.
Figure 12: audio_effects.conf Entries
In some platform builds configuration file can be an xml file in the name of audio_effcts.xml in the path: /frameworks/av/media/libeffects/data.
Similar to audio_effects.conf, in audio_effcts.xml, there will be a ‘libraries’ field and an ‘effects’ field.Name and path of the effect library will be added in . And effect name, library name and uuid mapping will get added as a new entry in field.
Figure 13: audio_effects.xml Entries
Custom Audio Format for Audio Effects
If our audio effect doesn’t support all the audio formats as input and output, there is a way to configure desired audio format. When the audioflinger tries to configure the audio effect, it will try to set one format. This will come as EFFECT_CMD_SET_CONFIG into the effect’s command function. If the effect is not our desired format, we can return error. Then AudioFlinger will try to set a different format. AudioFlinger will take care of resampling of audio if required.
Conclusion
We could integrate new audio effect libraries to android audio effects framework by following the generic implementation structure. We are able to configure the audio stream properties and to apply our audio processing algorithm. This gives an option to try out different audio enhancement algorithms. Since all effect library implementations keep the basic structure, effect framework becomes more open and always gets available with reliable references.
References
Interesting post, thanks for sharing.