Integration of Audio Effects in Android

Date: October 29, 2021

Author: Aneeshya Rose K M

Introduction to Android Audio Effects

Audio effects are used to enhance audibility of sound or to improve a specific audio feature. Android source code comes with few default audio effects like bassboost, equalizer, loudness enhancer which are then applied to the track using AudioFlinger while playing audio.

Android framework uses a vast variety of algorithms to provide the enhanced user experience in video and audio use cases. Often it is required to add custom audio effects to achieve additional benefits over the default audio effects. If we concentrate on audio effects in Android, we are having different options to come up with our own audio processing algorithms which could be applied to audio streams using the newly added effect library. Using audio effect framework, audio processing can be done either by using different android effect libraries or offloading audio streams to a third-party library.

This document explains ways of effect integration and how we can integrate effects into android.

New Audio Effect Integration

Here, we are discussing how a new android effect can be integrated into the existing android framework. Android effect integration can be done in two steps:

1. Implementing a new effect library and integrating it into android effect framework

2. Integrate with JNI and Java Implementations to connect the applications with native systems

We will be concentrating here on the effect integration in android native side.

Effect Flow from Upper Java Layer to Native Effect Library

Flow of effect handle creation from upper layer can be listed as follows:

1. From application code: MainActivity.java, Object of audio effects need to be created that invoke constructor of particular effect’s derived class.

2. From constructor of derived class, constructor of base class from AudioEffect.java will get called.

3. AudioEffect.java will take the flow to native framework through JNI layer: android_media_AudioEffect.cpp

4. Then the call reaches AndroidEffect.cpp ‘s set API. This is the entry point to native effect framework.

5. AudioEffect.cpp communicates to AudioFlinger.cpp

6. Audioflinger service will be calling create_effect_l of Threads.cpp implementation

7. Then EffectCreate of Effects.cpp will get called

8. From Effects.cpp, flow goes to EffectCreate of EffectFactory.cpp

9. Particular effect details - sampleEffect will be loaded from configuration files and particular effect handler will get created

Figure 1: Effect Flow Diagram

New Effect Implementation – Native Layer

UUID and Type

As a first step to start effect library implementation, we need to generate a UUID (universally unique identifier) and a type. Here UUID will be particular for effect implementation and the type will be another UUID to the OpenSL ES interface implemented by this effect. This can be kept even if there is no OpenSL ES interface implementation for the particular library. UUIDs can be generated online using the link.

Effect Library Implementation

To start with the effect library implementation, a new directory is created in frameworks/av/media/libeffects. If the effect name is ‘sampleEffect’, directory can be created with the name sampleEffect. Effect directory will include implementation specific c/cpp files, corresponding header files and Android compilation file– Android.bp/Android.mk. Let’s assume the file name as EffectSample.c, EffectSample.h and Android.mk. Following are the main components in each file.

  • Header file:
    • Definition of a structure to store effect attributes
    • Declarations of effect interface APIs
    • Declarations of effect interface control APIs
  • c/cpp file:
    • Definitions of effect interface APIs
    • Definitions of effect control APIs
  • Android.bp/Android.mk:
    • Compiling options

Effect Attributes Structure

Figure 2: Effects Attributes Structure

This structure will contain a pointer to another structure containing the addresses of all effect interface control APIs(itfe), a configuration structure which will get stored with all the configuration details of the effect flow, a context pointer particularly for the effect and an id of type integer. We could use this id field to store the session id.

Library Implementation

All android native effect libraries follow a basic method of implementation. Following structure definitions and API implementations come under this library implementation design.

AUDIO_EFFECT_LIBRARY_INFO_SYM

All effect libraries need to implemented as audio_effect_library_t structure named AUDIO_EFFECT_LIBRARY_INFO_SYM

Figure 3: AUDIO_EFFECT_LIBRARY_INFO_SYM

Here macro definitions of tag and versions will be defined in /hardware/libhardware/include/hardware/audio_effect. Apart from this, we could add name of new effect library and implementer details. Effect interface API pointers will also get added here.

Effect Descriptor Structure

There will be an effect_descriptor_t structure specific to the new effect. This structure will be having a unique uuid, type , effect flags and variables for cpu load and memory usage.

Figure 4: Effect Descriptor Structure

Effect Interface APIs

There are three interface APIs:

Create

Release

Get descriptor

Create function:

This function will create an instance of the effect of the particular uuid. And it will allocate the resources for the instance and will provide a handle of effect_handle_t.

I/O parameters:

Uuid : Pointer to the effect uuid

sessionId : Audio session id to which effect will be attached

ioId : Output or input stream id the effect is directed to in audio HAL

pHandle : Address where to return the effect interface handle

Figure 5: Create Function

Release function:

This function releases all the resources related to the particular effect handle.

I/O parameters:

handle: Effect interface handle

Figure 6: Release Function

Get descriptor function:

This function gets the effect descriptor structure corresponding to the given uuid.

I/O parameters:

uuid : Pointer to the effect uuid

pDescriptor : Address where to return the effect descriptor

Figure 7: Get Descriptor Interface Function

Effect Control APIs

Get descriptor function:

This function returns the effect descriptor. It will return the effect handle to the address location which will get passed as input parameter to the function.

I/O parameters:

Self : Handle to effect interface

Descriptor : Address to return the effect descriptor

Figure 8: Get Descriptor Control Function

Command function:

This function helps to send commands to effect engine and receives to and fro responses from effect.

I/O parameters:

Self handle to effect interface

cmdCode : Command code

cmdSize : Size of command in bytes

pCmdData : Pointer to command data

replySize : Maximum size of reply data

pReplyData : Pointer to reply data

Few of the commands defined in the effect framework are:

EFFECT_CMD_INIT : initialize effect engine

EFFECT_CMD_SET_CONFIG : configure effect engine

EFFECT_CMD_RESET : reset effect engine

EFFECT_CMD_ENABLE : enable effect process

EFFECT_CMD_DISABLE : disable effect process

EFFECT_CMD_SET_PARAM : set parameter immediately

EFFECT_CMD_GET_PARAM : get parameter

Figure 9: Command Function

Process function:

Process function as the name indicates processes the audio data. Function will get input audio samples of particular specification and will provide the processed data of specifications corresponding to output buffer. We could add our processing algorithm implementation here. There will an input and output buffer descriptors of type audio_buffer_t as function parameters. If these buffer descriptors are not specified by the function, buffer or buffer provider function installed by the command EFFECT_CMD_SET_CONFIG is to be used. Once EFFECT_CMD_ENABLE is received, effect framework will start to call the process function until EFFECT_CMD_DISABLE is received. When this command is received –ENODATA is to be returned so that frame can stop the calling of process function.

I/O parameters:

self : Handle to the effect interface

inBuffer : Input audio buffer

outBuffer : Output audio buffer

Figure 10: Process Function

Effect library compilation

Effect libraries are compiled to create a shared library of .so extension. This compiled library is to be stored to the built path: /system/lib/soundfx.

Figure 11: Compilation File - Android.mk

Adding Effect Details to Configuration File

When we add a new effect library to make effect visible to the framework, we need to add the effect details in the configuration file: audio_effects.conf. This configuration file will give the details of audio effects supported by platform. We could find details such as uuid, library name and library path here. Similarly new effect details are to be added to the file which can be found in the path: /frameworks/av/media/libeffects/data/audio_effects.conf.

Filed: libraries will point to the effect library .so file path and the field: effects will give the mapping details of .so file name, effect name and the corresponding uuid. In the android device this configuration file can be found either in /system/etc/ or in /vendor/etc.

Figure 12: audio_effects.conf Entries

In some platform builds configuration file can be an xml file in the name of audio_effcts.xml in the path: /frameworks/av/media/libeffects/data.

Similar to audio_effects.conf, in audio_effcts.xml, there will be a ‘libraries’ field and an ‘effects’ field.Name and path of the effect library will be added in . And effect name, library name and uuid mapping will get added as a new entry in field.

Figure 13: audio_effects.xml Entries

Custom Audio Format for Audio Effects

If our audio effect doesn’t support all the audio formats as input and output, there is a way to configure desired audio format. When the audioflinger tries to configure the audio effect, it will try to set one format. This will come as EFFECT_CMD_SET_CONFIG into the effect’s command function. If the effect is not our desired format, we can return error. Then AudioFlinger will try to set a different format. AudioFlinger will take care of resampling of audio if required.

Conclusion

We could integrate new audio effect libraries to android audio effects framework by following the generic implementation structure. We are able to configure the audio stream properties and to apply our audio processing algorithm. This gives an option to try out different audio enhancement algorithms. Since all effect library implementations keep the basic structure, effect framework becomes more open and always gets available with reliable references.

References

By submitting this form, you authorize PathPartner to contact you with further information about our relevant content, products and services. You may unsubscribe any time. We are committed to your privacy. For more details, refer our Privacy Policy

Automotive
Camera & IoT
Multimedia

By submitting this form, you authorize PathPartner to contact you with further information about our relevant content, products and services. You may unsubscribe any time. We are committed to your privacy. For more details, refer our Privacy Policy

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
Back to Top