25Jan
By: Alex Restrepo On: January 25, 2012 In: Blog, Demos, Featured, Technical Articles Comments: 17

I recently had the opportunity to work in Motorola Mobility’s Betaworks Lab as an intern, it was a great learning experience. One of my projects there was the inspiration for this project: get to run the audio fingerprinting library on an Android device.

Echoprint (http://echoprint.me) is an open source C++ fingerprinting library that is available for desktop computers and iOS devices, however, as far as I know, there isn’t an Android-compatible version, until now 🙂

This article is divided into 3 steps, the first one is how to compile the Echoprint C++ code as an Android native library, the second one on how build a reusable Android fingerprinting library project, and finally, step 3 is how to put everything together in your own projects, so without further ado, let’s get started.

Creating the Android native library.
The codegen library does all the heavy lifting, it actually processes the audio samples and generates a set of hashes that are the actual audio fingerprint.

For this step you will need:

– Android’s NDK (r6 or better)

– Echoprint codegen source, available at https://github.com/echonest/echoprint-codegen
Once you download the project, go to the src folder and rename all .cxx files to .cpp, this is necessary as the NDK doesn’t accept files with .cxx extension. Move the downloaded folder (echonest-echoprint-codegen-v4.11-45-g7bc4764 or something similar) to a new folder (I named it jni) and rename it “codegen” or whatever makes sense to you

– Boost C++ libraries src (http://www.boost.org/) and add the boost_1_XX_X folder to the jni folder (I used version 1.47.0)

Now fire up Eclipse and create a new Android project, I used “EchoprintLib” as the name of the project, but you can use anything you like.
In order to use a native library, some jni code needs to be written, as well as glue java code to invoke the jni code. Create a new java class, I used Codegen.java and declare the following native method:

native String codegen(float data[], int numSamples);

Your class should look like this:

public class Codegen
{
	native String codegen(float data[], int numSamples);
}

Save your changes as we will need the compiled .class file.
Now, navigate to the bin folder of your project and run the javah tool, this tool will generate the required .h file needed to implement the native jni code.

alex$ javah edu.gvsu.masl.echoprint.Codegen

Once it finishes, you will see a edu_gvsu_masl_echoprint_Codegen.h file that was generated, move this file to your jni folder.

Now it’s time to write the actual jni code that will invoke the codegen library and will return the results to our java library.

Create a new C++ file AndroidCodegen.cpp and add the following code:

#include <android/log.h>
#include <string.h>
#include <jni.h>
#include "edu_gvsu_masl_echoprint_Codegen.h"
#include "codegen/src/Codegen.h"

JNIEXPORT jstring JNICALL Java_edu_gvsu_masl_echoprint_Codegen_codegen
  (JNIEnv *env, jobject thiz, jfloatArray pcmData, jint numSamples)
{
    // get the contents of the java array as native floats
	float *data = (float *)env->GetFloatArrayElements(pcmData, 0);

    // invoke the codegen
	Codegen c = Codegen(data, (unsigned int)numSamples, 0);
	const char *code = c.getCodeString().c_str();

    // release the native array as we're done with them
	env->ReleaseFloatArrayElements(pcmData, data, 0);

    // return the fingerprint string
	return env->NewStringUTF(code);
}

The actual signature of the codegen function is machine generated by the javah tool, in the edu_gvsu_masl_echoprint_Codegen.h file.

We now have all the necessary code to compile the codegen code as an Android native library, however the NDK requires two more files in order to build the code: the Android.mk and Application.mk files.

Create a new text file and name it Android.mk, this file is the equivalent of a makefile for the NDK, it contains the list of files needed as well as other parameters used by the NDK. Add the following to the Android.mk file:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE    :=echoprint-jni

LOCAL_SRC_FILES :=AndroidCodegen.cpp 
			/codegen/src/Codegen.cpp 
			/codegen/src/Whitening.cpp 
			/codegen/src/SubbandAnalysis.cpp 
			/codegen/src/MatrixUtility.cpp 
			/codegen/src/Fingerprint.cpp 
			/codegen/src/Base64.cpp 
			/codegen/src/AudioStreamInput.cpp 
			/codegen/src/AudioBufferInput.cpp

LOCAL_LDLIBS    :=-llog
		-lz
LOCAL_C_INCLUDES :=<path to your jni folder>/codegen/src 
			<path to your jni folder>/boost_1_47_0

include $(BUILD_SHARED_LIBRARY)

In the LOCAL_C_INCLUDES line, make sure you replace the <path to your jni folder> placeholder with your actual path to the jni folder.

Create a new text file, Application.mk and add the following line:

APP_STL := gnustl_static

Basically, all this file does is enable STL support in the NDK compiler.

Your jni folder should now look similar to this:

If everything looks good, fire up a terminal window and run the ndk-build tool, from within your jni folder:

$ <path to your ndk>/ndk-build

if everything goes well you should see something similar to this:

Compile++ thumb  : echoprint-jni <= AndroidCodegen.cpp
Compile++ thumb  : echoprint-jni <= Codegen.cpp
Compile++ thumb  : echoprint-jni <= Whitening.cpp
Compile++ thumb  : echoprint-jni <= SubbandAnalysis.cpp
Compile++ thumb  : echoprint-jni <= MatrixUtility.cpp
Compile++ thumb  : echoprint-jni <= Fingerprint.cpp
Compile++ thumb  : echoprint-jni <= Base64.cpp
Compile++ thumb  : echoprint-jni <= AudioStreamInput.cpp
Compile++ thumb  : echoprint-jni <= AudioBufferInput.cpp
Prebuilt       : libstdc++.a <= <NDK>/sources/cxx-stl/gnu-libstdc++/libs/armeabi/
SharedLibrary  : libechoprint-jni.so
Install        : libechoprint-jni.so => libs/armeabi/libechoprint-jni.so

The freshly built native library now resides in a “libs” folder next to your jni folder:

Go ahead and copy the libs folder into your EchoprintLib project folder.

We can now write the rest of the glue code from the java side, open your EchoprintLib project and select the Codegen.java file.

In order to use the native library we need to load it, we just need to do this one time, when the class is loaded. To do so we add a static block that loads the library:

static
{
	System.loadLibrary("echoprint-jni");
}

As it turns out, the codegen code expects the audio samples to be floats in the [-1, 1] range, however, Android records audio samples as 16 bit shorts. The solution is simply to normalize the audio before is sent to the native library, to do so we define a normalizing value:

private final float normalizingValue = Short.MAX_VALUE;

Finally, the actual generate method will normalize the audio samples and then invoke the  codegen native method:

public String generate(short data[], int numSamples)
{
	float normalizeAudioData[] = new float[numSamples];
	for (int i = 0; i < numSamples - 1; i++)
		normalizeAudioData[i] = data[i] / normalizingValue;

	return this.codegen(normalizeAudioData, numSamples);
}

For completeness, I’m adding an overloaded method that expects already normalized values as floats:

public String generate(float data[], int numSamples)
{
	return codegen(data, numSamples);
}

This is the complete Codegen class:

package edu.gvsu.masl.echoprint;

public class Codegen
{
	private final float normalizingValue = Short.MAX_VALUE;

	native String codegen(float data[], int numSamples);

	static
	{
        	System.loadLibrary("echoprint-jni");
    	}

	/**
	 * Invoke the echoprint native library and generate the fingerprint code.<br>
	 * Echoprint REQUIRES PCM encoded audio with the following parameters:<br>
	 * Frequency: 11025 khz<br>
	 * Data: MONO - PCM enconded float array
	 *
	 * @param data PCM encoded data as floats [-1, 1]
	 * @param numSamples number of PCM samples at 11025 KHz
	 * @return The generated fingerprint as a compressed - base64 string.
	 */
	public String generate(float data[], int numSamples)
	{
		return codegen(data, numSamples);
	}

	/**
	 * Invoke the echoprint native library and generate the fingerprint code.<br>
	 * Since echoprint requires the audio data to be an array of floats in the<br>
	 * range [-1, 1] this method will normalize the data array transforming the<br>
	 * 16 bit signed shorts into floats.
	 *
	 * @param data PCM encoded data as shorts
	 * @param numSamples number of PCM samples at 11025 KHz
	 * @return The generated fingerprint as a compressed - base64 string.
	 */
	public String generate(short data[], int numSamples)
	{
		float normalizeAudioData[] = new float[numSamples];
		for (int i = 0; i < numSamples - 1; i++)
			normalizeAudioData[i] = data[i] / normalizingValue;

		return this.codegen(normalizeAudioData, numSamples);
	}
}

All the code needed to invoke the codegen native library is now in place, let’s now start step 2 and turn the project into a library and add a class that records audio from the microphone and returns its fingerprint.

Turning the project into a reusable library
This is actually pretty simple, all we need to do is go to the properties of the project, and under the Android category, check the appropriate “Is Library” checkbox:

Your project can now be used as a library for other projects to use, let’s now add some code to take care of the audio recording and returning the results back to the client code.

First, let’s define an interface to communicate the results (or problems) back to whatever code is using the fingerprinter. My background is in iOS so you’ll find that the name of the methods closely resemble iOS delegate method names, but you can rename them to whatever you like:

/**
 * Interface for the fingerprinter listener<br>
 * Contains the different delegate methods for the fingerprinting process
 * @author Alex Restrepo
 *
 */
public interface AudioFingerprinterListener
{
	/**
	 * Called when the fingerprinter process loop has finished
	 */
	public void didFinishListening();

	/**
	 * Called when a single fingerprinter pass has finished
	 */
	public void didFinishListeningPass();

	/**
	 * Called when the fingerprinter is about to start
	 */
	public void willStartListening();

	/**
	 * Called when a single listening pass is about to start
	 */
	public void willStartListeningPass();

	/**
	 * Called when the codegen libary generates a fingerprint code
	 * @param code the generated fingerprint as a zcompressed, base64 string
	 */
	public void didGenerateFingerprintCode(String code);

	/**
	 * Called if the server finds a match for the submitted fingerprint code
	 * @param table a hashtable with the metadata returned from the server
	 * @param code the submited fingerprint code
	 */
	public void didFindMatchForCode(Hashtable<String, String> table, String code);

	/**
	 * Called if the server DOES NOT find a match for the submitted fingerprint code
	 * @param code the submited fingerprint code
	 */
	public void didNotFindMatchForCode(String code);

	/**
	 * Called if there is an error / exception in the fingerprinting process
	 * @param e an exception with the error
	 */
	public void didFailWithException(Exception e);
}

Now, let’s create an AudioFingerprinter class, and define a few keys to extract the metadata returned from the echoprint server:

public final static String META_SCORE_KEY = "meta_score";
public final static String SCORE_KEY = "score";
public final static String ALBUM_KEY = "release";
public final static String TITLE_KEY = "track";
public final static String TRACK_ID_KEY = "track_id";
public final static String ARTIST_KEY = "artist";

NOTE: the metadata dictionary IS NOT included by default in the echoprint server API, to add it, just replace line 66/67 in API.py with:

return json.dumps({"ok":True,"message":response.message(), "match":response.match(), "score":response.score, 
"qtime":response.qtime, "track_id":response.TRID, "total_time":response.total_time, "metadata":response.metadata})

Then let’s declare some constants for our audio settings, as the codegen requires the audio to be uncompressed PCM, mono @ 11Khz:

private final int FREQUENCY = 11025;
private final int CHANNEL = AudioFormat.CHANNEL_IN_MONO;
private final int ENCODING = AudioFormat.ENCODING_PCM_16BIT;

With these constants in place, we can now initialize the audio recorder and start listening for audio:

// create the audio buffer
// get the minimum buffer size
int minBufferSize = AudioRecord.getMinBufferSize(FREQUENCY, CHANNEL, ENCODING);

// and the actual buffer size for the audio to record
// frequency * seconds to record.
bufferSize = Math.max(minBufferSize, this.FREQUENCY * this.secondsToRecord);

audioData = new short[bufferSize];

// start recorder
mRecordInstance = new AudioRecord(MediaRecorder.AudioSource.MIC,
					FREQUENCY, CHANNEL,
					ENCODING, minBufferSize);

mRecordInstance.startRecording();

At this point the audio recording has started, in this loop we extract the audio samples,  invoke the codegen class to extract their fingerprint and submit the fingerprint to your own echoprint server in order to get the metadata of the fingerprinted audio. The results are then forwarded to the client code, via an interface, as a hashtable.

I have defined a continuous variable to allow the class to listen for audio continuously, if it’s set to true, the loop will be repeated until the stop method is called in the AudioFingerprinter instance:

do
{
	try
	{
		// fill audio buffer with mic data.
		int samplesIn = 0;
		do
		{
			samplesIn += mRecordInstance.read(audioData, samplesIn, bufferSize - samplesIn);

			if(mRecordInstance.getRecordingState() == AudioRecord.RECORDSTATE_STOPPED)
				break;
		}
		while (samplesIn < bufferSize);

		// create an echoprint codegen wrapper and get the code
		Codegen codegen = new Codegen();
		String code = codegen.generate(audioData, samplesIn);

		if(code.length() == 0)
		{
			// no code?
			// not enough audio data?
			continue;
		}

		// fetch data from echoprint server
		String urlstr = SERVER_URL + code;
		HttpClient client = new DefaultHttpClient();
		HttpGet get = new HttpGet(urlstr);

		// get response
		HttpResponse response = client.execute(get);
	    	// Examine the response status
		Log.d("Fingerprinter",response.getStatusLine().toString());

		// Get hold of the response entity
		HttpEntity entity = response.getEntity();
		// If the response does not enclose an entity, there is no need
		// to worry about connection release

		String result = "";
		if (entity != null)
		{
			// A Simple JSON Response Read
			InputStream instream = entity.getContent();
			result= convertStreamToString(instream);
			// now you have the string representation of the HTML request
			instream.close();
		}

	    	// parse JSON
		JSONObject jobj = new JSONObject(result);
		if(jobj.has("match"))
		{
			if(jobj.getBoolean("match"))
			{
				Hashtable<String, String> match = new Hashtable<String, String>();
				match.put(SCORE_KEY, jobj.getDouble(SCORE_KEY) + "");
				match.put(TRACK_ID_KEY, jobj.getString(TRACK_ID_KEY));

				if(jobj.has("metadata"))
		    		{
					JSONObject metadata = jobj.getJSONObject("metadata");
			    		if(metadata.has(SCORE_KEY)) match.put(META_SCORE_KEY, metadata.getDouble(SCORE_KEY) + "");
					if(metadata.has(TITLE_KEY)) match.put(TITLE_KEY, metadata.getString(TITLE_KEY));
					if(metadata.has(ARTIST_KEY)) match.put(ARTIST_KEY, metadata.getString(ARTIST_KEY));
					if(metadata.has(ALBUM_KEY)) match.put(ALBUM_KEY, metadata.getString(ALBUM_KEY));
		    		}

		    		didFindMatchForCode(match, code);
			}
			else
				didNotFindMatchForCode(code);
		}
		firstRun = false;

	}
	catch(Exception e)
	{
		e.printStackTrace();
		Log.e("Fingerprinter", e.getLocalizedMessage());
	}
}
while (this.continuous);

We now have a complete library that can listen to audio, invoke the codegen native library and return the computed fingerprint back to your code.

The final step is then create a sample project that uses the completed code library.

Creating a sample project that uses the library
Create a new project in eclipse, I chose EchoprintTest as its name, and in its properties, go to the Android category and add the EchoprintLib library to the project:

In this sample project, I added a button (recordButton) and a TextView (status) to the main Activity, in order to start/stop the audio fingerprinting process and to display the results from the echoprint server.

In the main activity, we need to implement the AudioFingerprinterListener interface:

public class EchoprintTestActivity extends Activity implements AudioFingerprinterListener
...
public void didFinishListening()
{
	btn.setText("Start");

	if(!resolved)
		status.setText("Idle...");

	recording = false;
}

public void didFinishListeningPass()
{}

public void willStartListening()
{
	status.setText("Listening...");
	btn.setText("Stop");
	recording = true;
	resolved = false;
}

public void willStartListeningPass()
{}

public void didGenerateFingerprintCode(String code)
{
	status.setText("Will fetch info for code starting:n" + code.substring(0, Math.min(50, code.length())));
}

public void didFindMatchForCode(final Hashtable<String, String> table,
		String code)
{
	resolved = true;
	status.setText("Match: n" + table);
}

public void didNotFindMatchForCode(String code)
{
	resolved = true;
	status.setText("No match for code starting with: n" + code.substring(0, Math.min(50, code.length())));
}

public void didFailWithException(Exception e)
{
	resolved = true;
	status.setText("Error: " + e);
}

In order to start the fingerprinting process, all we need to do is create an instance of the AudioFingerprinter class and call the fingerprint method, specifying how many seconds of audio to record per listening pass, the maximum is 30 and the minimum is 10. I’ve found out that for better results, anything above 15 should work fine:

btn = (Button) findViewById(R.id.recordButton);
btn.setOnClickListener(new View.OnClickListener()
{
    public void onClick(View v)
    {
    	if(recording)
    	{
		fingerprinter.stop();
    	}
    	else
    	{
    		if(fingerprinter == null)
    			fingerprinter = new AudioFingerprinter(EchoprintTestActivity.this);

    		fingerprinter.fingerprint(15);
    	}
    }
});

And voila, all you need to do now is do anything interesting with the results returned from the server in the didFindMatchForCode method!

One final thing though, don’t forget to add

<uses-permission android:name="android.permission.RECORD_AUDIO"/>
<uses-permission android:name="android.permission.INTERNET"/>

To your manifest file in order to enable audio recording and internet access!

Now go ahead and download the echoprint server, feed it with some audio files (check the documentation on how to do that) and start developing your own audio fingerprinting apps!

Download the complete source from GitHub repo: https://github.com/gvsumasl/EchoprintForAndroid.

Happy coding!

Alex Restrepo on GithubAlex Restrepo on InstagramAlex Restrepo on Linkedin
Alex Restrepo

Alex is an intuitive and imaginative problem solver, currently interested in developing rich experiences for mobile / tablet devices, with a particular affinity towards iOS devices, computer games and their mechanics, computer graphics, computational problems and algorithms.


17 Comments:

    • John van Derton
    • February 18, 2012
    • Reply

    ndk is supporting C++ exceptions (NDK r5+) but stills with -fno-exceptions by default. In case of exception during compiling “error: exception handling disabled, use -fexceptions to enable”, set the variable below into the .mk file :

    LOCAL_CPPFLAGS += -fexceptions

    It adds this complementary flag to the armcc compiler.

    • Eric Brynsvold
    • March 31, 2012
    • Reply

    Thank you for this comment John, it was extremely helpful in clearing up an issue I had! Just to add, the location of the setting of the LOCAL_CPPFLAGS variable is important. I added it at the end of the file, right before the “include $(BUILD_SHARED_LIBRARY)” and my project built successfully.

    • tGadget
    • March 31, 2012
    • Reply

    yeah, it’s nice tutorial dude. i want to try it now

    • Jed Gainer
    • April 08, 2012
    • Reply

    Anyone got this working on ICS?

    • Filipe Rodrigues
    • April 17, 2012
    • Reply

    Does anyone know if this is 2.2 compatible ? (sdk=8)

    thanks in advance 😉

    • earthwormjeff
    • May 17, 2012
    • Reply

    Hi, application is not working for me : I set the server’s address to : SERVER_URL = “http://developer.echonest.com/api/v4/song/identify?api_key=N6E4NIOVYMTHNDM8J&version=4.12&code=”; and server responds : Results fetched : {“response”: {“status”: {“version”: “4.2”, “code”: 0, “message”: “Success”}, “songs”: []}} althought I tried songs from database.
    What am I missing ?

    • John
    • May 31, 2012
    • Reply

    By the way how do you set up your on fingerprint data . I have a set of songs which iam pretty sure is not available . How i can generate the data . Any libraries for that ?

      • Alex Restrepo
      • June 01, 2012
      • Reply

      Download the echoprint server code, there’s instructions on how to generate and ingest data into a server.

    • John
    • June 11, 2012
    • Reply

    I was able to run the code and i did install my on instance of the server and did fingerprint some and did the ingest. But somehow the same doesnt match from the emulator – Am i missing anything..
    i changed the frequency to 8000 in Android Code..

    • hari
    • July 14, 2012
    • Reply

    how are we supposed to install boost ?? are we to use the boost that is installed for android or boost installed for the linux … sorry for my english :p

    • Ketan
    • July 18, 2012
    • Reply

    Hello,

    I am using Echonest API to identify song,

    My URL, Every time it response

    Request :
    http://developer.echonest.com/api/v4/song/identify?api_key=&code=

    Response :
    {“response”: {“status”: {“version”: “4.2”, “code”: 0, “message”: “Success”}, “songs”: []}}

    Src : http://developer.echonest.com/docs/v4/song.html#identify

    • cem
    • August 06, 2012
    • Reply

    Hallo,

    I also tried to use the Echonest API to identify song.

    URL:
    http://developer.echonest.com/api/v4/song/identify?api_key=myapikey

    but i always get

    Unknow error

    can anybody help me? sry about my bad English 😛

    thx

    • cem
    • August 06, 2012
    • Reply

    where does i need this:

    {“response”: {“status”: {“version”: “4.2″, “code”: 0, “message”: “Success”}, “songs”: []}}

    • albano
    • August 08, 2012
    • Reply

    Hello all, hello Alex,

    First of all thanks for this tuto. I’m blocked when compiling the code with ndk-build. I’m using the r8b version of android ndk.

    I get an error here :

    “Compile++ thumb : echoprint-jni <= Codegen.cpp
    //jni//codegen/src/Codegen.cpp: In constructor ‘Codegen::Codegen(float const*, unsigned int, in
    t)’:
    //jni//codegen/src/Codegen.cpp:27:54: error: exception handling disabled, use -fexceptions to enable
    make: *** [//obj/local/armeabi/objs/echoprint-jni//codegen/src/Codegen.o] Error 1

    It seems to me that since r6 they changed the way they handle exception, but i’m not expert in c++ compiling.

    I removed the line in the /jni/codegen/src/Codegen.cpp file that was handling exception and it compiles, but my nose knows that’s not a clean behaviour.

      • Alex Restrepo
      • August 11, 2012
      • Reply

      Try John’s suggestion and enable

      LOCAL_CPPFLAGS += -fexceptions

      in your .mk file.

    • Frank
    • August 12, 2012
    • Reply

    Thank you really helpfull =)

    • Cameron
    • September 14, 2012
    • Reply

    Thank you for the tutorial, but I am hitting a snag when compiling using the r8 NDK. I am getting the following error:

    Compile++ thumb : echoprint-jni <= AndroidCodegen.cpp
    cc1plus: error: unrecognized command line option "-mbionic"
    cc1plus: error: unrecognized command line option "-mthumb"
    cc1plus: error: unrecognized command line option "-mfpu=vfp"
    /Users/wingdom/Desktop/jni/AndroidCodegen.cpp:1: error: bad value (armv5te) for -march= switch
    /Users/wingdom/Desktop/jni/AndroidCodegen.cpp:1: error: bad value (xscale) for -mtune= switch
    make: *** [/Users/wingdom/Desktop/obj/local/armeabi/objs/echoprint-jni/AndroidCodegen.o] Error 1

    Any help would be appreciated.

Leave reply:

Your email address will not be published. Required fields are marked *