Panorama app on Android using stitching module OpenCV

Posted: December 21, 2012 in Technology
Tags: , , , ,

Me and a buddy of mine from class, Thomas LaBruyere (Linkedin profile here) had recently worked on a panorama app on Android using OpenCV stitching module. Here is the GUI of the app-

Panorama app GUI

We built the app on Google Nexus 7  running Android 4.1(Jelly Bean). The basic functionality of the app is as follows -

1)  It has four buttons “Start Video Capture” , “Capture Still Image”, “Stitch” and “View Stitched Images”.

2) The app is started with normal video mode where the video from the front camera is shown. There are two main modes of getting the panorama and stitching it. First one is to use the “Start Video capture” button where a video of couple of seconds is stored where you move  your camera around to capture your surroundings. Then press the “Stitch” button and the stitched panorama image is saved on to the SD card after a couple of seconds. Second  is  “Capture Still Image” button where you  click the button to store images that need to be stitched as panorama while you move the camera around. Once done with capturing the images to be stitched , click on “Stitch” button to complete the panorama stitching.  The stitched output will be saved on to  the SD card. You may change the path in the code accordingly to save it at a different path on the disk. The button ” View Stitched images”  should open up the gallery to look through the stitched images but it has a slight problem that hasn’t been fixed yet so wouldn’t work as expected. You should navigate to the SD card folder manually to see the image.

The algorithm of panorama stitching is implemented through   OpenCV stitching module.  The Android jni interface communicates with the OpenCV C/C++ native  code.

We did not have any previous experience with Android programming or  linking OpenCV . We used the following link and got started -

Getting started with Android and OpenCV

Here is the output of the  panorama that we took in the class .

Panorama output

This project might be old  gives some compilation errors. I don’t have the setup now to fix it. I would highly recommend you to compile the example projects from opencv4android and integrate the code parts from below in them. Nevertheless the project folder can be downloaded just for reference from here.

The two main codes from the project are shown below -

1) OpenCV code to stitch the images. (jni_part.cpp)

#include < jni.h >
#include <opencv2/core/core.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/stitching/stitcher.hpp>

#include < vector >
#include < iostream >
#include < stdio.h >
#include < list >
#include< sstream >
#include< string >

using namespace std;
using namespace cv;

extern "C" {
//JNIEXPORT Mat JNICALL Java_org_opencv_samples_tutorial3_Sample3Native_FindFeatures(JNIEnv*, jobject, jlong addrGray, jlong addrRgba)

JNIEXPORT void JNICALL Java_org_opencv_samples_tutorial3_Sample3Native_FindFeatures(
		JNIEnv*, jobject, jlong im1, jlong im2, jlong im3, jint no_images) {

	vector < Mat > imgs;
	bool try_use_gpu = false;
	// New testing
	Mat& temp1 = *((Mat*) im1);
	Mat& temp2 = *((Mat*) im2);
	Mat& pano = *((Mat*) im3);

	for (int k = 0; k < no_images; ++k) {
		string id;
		ostringstream convert;
		convert << k;
		id = convert.str();
		Mat img = imread("/storage/emulated/0/panoTmpImage/im" + id + ".jpeg");

		imgs.push_back(img);
	}

	Stitcher stitcher = Stitcher::createDefault(try_use_gpu);
	Stitcher::Status status = stitcher.stitch(imgs, pano);

}

}

2) Android code that has main GUI and calls the OpenCV function (Sample3Native.java)

package org.opencv.samples.tutorial3;

import java.io.File;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.highgui.Highgui;

import android.app.Activity;
import android.content.Intent;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.os.Handler;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.view.Window;
import android.view.WindowManager;
import android.widget.Button;

public class Sample3Native extends Activity implements CvCameraViewListener {
	private static final String TAG = "OCVSample::Activity";

	public static final int VIEW_MODE_RGBA = 0;
	public static final int SAVE_IMAGE_MAT = 1;
	public static final int CAPT_STILL_IM = 2;
	private static int viewMode = VIEW_MODE_RGBA;
//	public static int image_count = 0;
	private MenuItem mStitch;
	private MenuItem mItemCaptureImage;
	private Mat mRgba;
	private Mat mGrayMat;
	private Mat panorama;
	private Mat mtemp;
	private List < Mat > images_to_be_stitched = new ArrayList < Mat >();
	private CameraBridgeViewBase mOpenCvCameraView;
	private long mPrevTime = new Date().getTime();
	private static final int FRAME2GRAB = 10;
	private int mframeNum = 0;
	private static final File tempImageDir = new File(Environment.getExternalStorageDirectory() + File.separator + "panoTmpImage");
	private static final File StitchImageDir = new File(Environment.getExternalStorageDirectory()+ File.separator  + "panoStitchIm");
	private static final String mImageName = "im";
	private static final String mImageExt = ".jpeg";
	private long recordStart = new Date().getTime();
	private static final long MAX_VIDEO_INTERVAL_IN_SECONDS = 3 * 1000; // Convert milliseconds to seconds
	public final Handler mHandler = new Handler();

	// Create runnable for posting
    final Runnable mUpdateResults = new Runnable() {
        public void run() {
            updateResultsInUi();
        }
    };

    private void updateResultsInUi()
    {

    }

	private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
		@Override
		public void onManagerConnected(int status) {
			switch (status) {
			case LoaderCallbackInterface.SUCCESS: {
				Log.i(TAG, "OpenCV loaded successfully");

				// Load native library after(!) OpenCV initialization
				System.loadLibrary("native_sample");

				mOpenCvCameraView.enableView();
			}
				break;
			default: {
				super.onManagerConnected(status);
			}
				break;
			}
		}
	};

	public Sample3Native() {
		Log.i(TAG, "Instantiated new " + this.getClass());
	}

	/** Called when the activity is first created. */
	@Override
	public void onCreate(Bundle savedInstanceState) {
		Log.i(TAG, "called onCreate");
		super.onCreate(savedInstanceState);
		requestWindowFeature(Window.FEATURE_NO_TITLE);
		getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

		setContentView(R.layout.tutorial3_surface_view);

		final Button btnVidCapt = (Button) findViewById(R.id.btnVidCapt);
		btnVidCapt.setOnClickListener(new View.OnClickListener() {
            public void onClick(View v) {
            	startVidCap();
            }
        });

		final Button btnStitch = (Button) findViewById(R.id.btnStitch);
		btnStitch.setOnClickListener(new View.OnClickListener() {
            public void onClick(View v) {
            	stitchImages();
            }
        });

		final Button btnViewStitchedIm = (Button) findViewById(R.id.btnViewStitchedIm);
		btnViewStitchedIm.setOnClickListener(new View.OnClickListener() {
            public void onClick(View v) {
            	viewStitchImages();
            }
        });

		final Button btnCapStil = (Button) findViewById(R.id.btnCapStil);
		btnCapStil.setOnClickListener(new View.OnClickListener() {
            public void onClick(View v) {
            	captStillImage();
            }
        });
		mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial4_activity_surface_view);
		mOpenCvCameraView.setCvCameraViewListener(this);
	}

	@Override
	public void onPause() {
		if (mOpenCvCameraView != null)
			mOpenCvCameraView.disableView();
		super.onPause();
	}

	@Override
	public void onResume() {
		super.onResume();
		OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this,
				mLoaderCallback);
	}

	public void onDestroy() {
		super.onDestroy();
		if (mOpenCvCameraView != null)
			mOpenCvCameraView.disableView();
	}

	public void onCameraViewStarted(int width, int height) {
		mRgba = new Mat(height, width, CvType.CV_8UC3);
		mGrayMat = new Mat(height, width, CvType.CV_8UC1);
		mtemp = new Mat(height, width, CvType.CV_8UC3);
		panorama = new Mat(height, width, CvType.CV_8UC3);
	}

	public void onCameraViewStopped() {
		mRgba.release();
		mGrayMat.release();
		mtemp.release();
		panorama.release();
	}

	public Mat onCameraFrame(Mat inputFrame) {
		inputFrame.copyTo(mRgba);
		switch (Sample3Native.viewMode) {
		case Sample3Native.VIEW_MODE_RGBA: {
			Core.putText(mRgba, "Video Mode", new Point(10, 50), 3, 1, new Scalar(255, 0, 0, 255), 2);
			// Update start recordtime until starting recording
		}break;
		case Sample3Native.SAVE_IMAGE_MAT: {
			long curTime = new Date().getTime();
			Core.putText(mRgba, "Record Mode", new Point(10, 50), 3, 1, new Scalar(255, 0, 0, 255), 2);
			long timeDiff = curTime - recordStart;
			Log.i("timeDiff", Long.toString(timeDiff));

			if ( timeDiff < MAX_VIDEO_INTERVAL_IN_SECONDS) {
				if ((mframeNum % FRAME2GRAB) == 0) {
					saveImageToArray(inputFrame);
					mframeNum++;
				}
				else
					mframeNum++;
			}
			else
			{
				mframeNum = 0;
				turnOffCapture();
			}
		}break;
		case Sample3Native.CAPT_STILL_IM :
		{
			saveImageToArray(inputFrame);
			Sample3Native.viewMode = Sample3Native.VIEW_MODE_RGBA;
		}
		}
		return mRgba;
	}

	public void startVidCap() {
		if (Sample3Native.viewMode == Sample3Native.VIEW_MODE_RGBA)
		{
			turnOnCapture();
		}
		else if (Sample3Native.viewMode == Sample3Native.SAVE_IMAGE_MAT)
		{
			turnOffCapture();
		}
	}

	private void turnOffCapture()
	{

		Sample3Native.viewMode = Sample3Native.VIEW_MODE_RGBA;
	}

	private void turnOnCapture()
	{

		Sample3Native.viewMode = Sample3Native.SAVE_IMAGE_MAT;
//		startVidCapture.setText("Stop Video Capture");
		images_to_be_stitched.clear();
		recordStart = new Date().getTime();

	}

	public void stitchImages() {
		if(!images_to_be_stitched.isEmpty())
		{
			for (int j = 0; j < images_to_be_stitched.size(); j++) {
				writeImage(images_to_be_stitched.get(j), j);
			}
		Log.i("stitchImages", "Done writing 2 disk. Starting stitching " + images_to_be_stitched.size() + " images");
			FindFeatures(images_to_be_stitched.get(0).getNativeObjAddr(),
					images_to_be_stitched.get(0).getNativeObjAddr(),
					panorama.getNativeObjAddr(), images_to_be_stitched.size());
		Log.i("stitchImages", "Done stitching. Writing panarama");
			writePano(panorama);

		Log.i("stitchImages", "deleting temp files");

			deleteTmpIm();
		}
	}

    public void captStillImage()
    {
    	Sample3Native.viewMode = Sample3Native.CAPT_STILL_IM;

    }

	private String getFullFileName( int num)
	{
		return mImageName + num + mImageExt;
	}

	private void writeImage(Mat image, int imNum)
	{
		writeImage(image, getFullFileName(imNum));
	}

	private void writeImage(Mat image, String fileName) {
		File createDir = tempImageDir;
		if(!createDir.exists())
			createDir.mkdir();
		Highgui.imwrite(tempImageDir+File.separator + fileName, image);
	}

	private void writePano(Mat image)
	{
		Date dateNow = new  Date();
		SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMdd_HHmmss");
		if(!StitchImageDir.exists())
			StitchImageDir.mkdir();
		Highgui.imwrite(StitchImageDir.getPath()+ File.separator + "panoStich"+dateFormat.format(dateNow) +mImageExt, image);

	}

	private void deleteTmpIm()
    {
		File curFile;
		for (int j = 0; j < images_to_be_stitched.size(); j++) {
			curFile = new File(getFullFileName(j));
			curFile.delete();
		}
		images_to_be_stitched.clear();
    }

	public void viewStitchImages()
	{

		Intent intent = new Intent(this, GalleryActivity.class);

		startActivity(intent);
	}

	private void saveImageToArray(Mat inputFrame) {
		images_to_be_stitched.add(inputFrame.clone());
	}

	private int FPS() {
		long curTime = new Date().getTime();
		int FPS = (int) (1000 / (curTime - mPrevTime));
		mPrevTime = curTime;
		return FPS;
	}

	@Override
	public boolean onCreateOptionsMenu(Menu menu) {
		return true;

	}

	@Override
	public boolean onOptionsItemSelected(MenuItem item) {
	return true;
	}

	// public native void FindFeatures(List pano_images, Long stitch );
	public native void FindFeatures(long image1, long image2, long image3,
			int count);
}

About these ads
Comments
  1. Rushank says:

    i want to make an app with .jar extension…..so how can i do….please help

  2. Olga says:

    OpenCV Error: Assertion failed (ssize.area() > 0) in void cv::resize(cv::InputArray, cv::OutputArray, cv::Size, double, double, int), file /home/reports/ci/slave/opencv/modules/imgproc/src/imgwarp.cpp, line 1725

  3. carlosmaciel says:

    Hello Ramsri. Yes, I downloaded the code and I didn’t understood why you passed the same images to the FindFeatures.

    I changed a bit your code and it worked :) , here’s my code:

    Java Code,

    FindFeatures(Highgui.imread(path + "pan1.jpg").getNativeObjAddr(), Highgui.imread(path + "pan2.jpg").getNativeObjAddr(),panorama.getNativeObjAddr(), 2);

    C code:

    vector imgs;
    bool try_use_gpu = false;
    Mat& temp1 = *((Mat*) im1);
    Mat& temp2 = *((Mat*) im2);
    Mat& pano = *((Mat*) im3);
    imgs.push_back(temp1);
    imgs.push_back(temp2);

    Stitcher stitcher = Stitcher::createDefault(try_use_gpu);
    Stitcher::Status status = stitcher.stitch(imgs, pano);

    Thanks for answering so fast.

  4. kk says:

    Hi,I have used your code but getting the error of Assertion failed (same as above error).I am using 2.4.4 version of OpenCV. Please help me..Thanks in advance

  5. Sarah Salem says:

    Nice!

    I am trying to do this -somehow- as a part of my final project this semester; I want to use multiple images instead of Video using c++ visual studio, but what shall I use to represent the panorama, an image box ?!

    Thanks for sharing :)

  6. tonyro says:

    Hi Ramsri,

    Great tutorial, thanks! I’ve been trying to do a similar thing using two webcam feeds, but am struggling to get the code to work. Have you ever tried doing this? The code is below, and returns “Stitching Error Code: 1″:

    #include
    #include
    #include
    #include
    #include

    using namespace std;
    using namespace cv;

    int main(int argc, char *argv[])
    {
    Mat fr1, fr2, pano;
    bool try_use_gpu = false;
    vector imgs;
    VideoCapture cap(0), cap2(1);

    while (true)
    {
    cap >> fr1;
    cap2 >> fr2;

    imgs.push_back(fr1.clone());
    imgs.push_back(fr2.clone());

    Stitcher stitcher = Stitcher::createDefault(try_use_gpu);
    Stitcher::Status status = stitcher.stitch(imgs, pano);

    if (status != Stitcher::OK)
    {
    cout << "Error stitching – Code: " <<int(status)<= 0)
    break;
    }
    return 0;
    }

    • HI Tony,

      I would display the two images pushed into imgs to see if they are correct before calling the stitcher class.

      Also if I were to do 2 webcam captures and stitching I would go like this ——-

      int main()

      {

      —————————————
      —————————————

      CvCapture * capture;
      CvCapture * capture2;

      capture = cvCaptureFromCAM( 0 );
      capture2 = cvCaptureFromCAM( -1 );

      if( capture && capture2)
      {
      while( true )
      {
      frame1 = cvQueryFrame( capture );
      frame2 = cvQueryFrame( capture2 );
      % for checking if they are correct or not
      imwrite(“image1″+”.jpg”, frame1);
      imwrite(“image1″+”.jpg”, frame2);

      imgs.push_back(frame1);
      imgs.push_back(frame2);
      }

      Then use

      Stitcher stitcher = Stitcher::createDefault(try_use_gpu);
      Stitcher::Status status = stitcher.stitch(imgs, pano);

      imwrite(“result.jpg”, pano);
      }

      I haven’t compiled this. So rewrite this according to your requirements.

      Keep us posted if you’ve made progress.

      Good luck,

      Ramsri

  7. Rushank says:

    hey anyone’s code over here is working….i have choosen this as my topic for project and have demonstration on monday i.e. 9/4/13….please can anyonwe whose code is working fine can mail me the folder……side by side i m trying to make this code also working……please guys……mail me at
    rushankparmar@yahoo.com

    and also mention that u read d comment for panorama and so u mailed me…so that i will come to know its not the spam……please guuys

  8. Rushank says:

    when i tried this on my phone an error is shown.It says “The application octv3 Add Native Opencv (process org.opencvsamples.tutorial3)has stopped unexpectedly”…….please help…what should ido

  9. Rushank says:

    @carlos will u please send me ur entire package @ rushankparmar@yahoo.com asap

  10. phash says:

    i have tried this code and it is give errors in
    import org.opencv.android.BaseLoaderCallback;
    import org.opencv.android.CameraBridgeViewBase;
    import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener;
    import org.opencv.android.LoaderCallbackInterface;
    import org.opencv.android.OpenCVLoader;
    import org.opencv.core.Core;
    import org.opencv.core.CvType;
    import org.opencv.core.Mat;
    import org.opencv.core.Point;
    import org.opencv.core.Scalar;
    import org.opencv.highgui.Highgui;

    private Mat mRgba;
    private Mat mGrayMat;
    private Mat panorama;
    private Mat mtemp;
    private List images_to_be_stitched = new ArrayList();
    private CameraBridgeViewBase mOpenCvCameraView;

    can I please have an answer the reason of this issue please.
    Thank you

  11. Anonymous says:

    Hi,I have used your code but getting some errors: GalleryActivity cannot be resolved to a type.
    Can you tell me where is the definition of “GalleryActivity”.
    And There is another error that the APP will stop when I click the stitch buttoon.
    Please help me..Thanks in advance

    • Thomas says:

      I am getting the same error as the last guy. After spending several hours fixing all of the stuff to make it compatible with 2.4.6, I am down to this last problem. “GalleryActivity cannot be resolved to a type” I think there is a missing .java file or something.

    • Thomas says:

      This project is missing all kinds of crap.

      • Hi Thomas,

        The project is out of date and would give compilation errors. I apologize that I do not have the setup to fix it now. I would advise you to compile a base project from opencv4android and try to integrate the code.

        Thanks,
        Ramsri

      • Thomas says:

        Yeah that is what I was thinking. I am involved with a week long short project and we are trying to implement a sort of “Photo Sphere”. I was hoping I could find some code that I could use to build on. OpenCV looks promising but I have not been able to find a good baseline piece of code to start from. My only other option is to learn as much as I can about how OpenCV stitches photos to create a panorama and do just what you said. Start from a base sample project in OpenCV4Android. However, I noticed you used the native-activity as a base for yours. I cannot get the native-activity to do anything useful. Do you have any suggestions?

        Thanks
        Thomas

      • Hi Thomas,

        There are about 7-8 sample tutorial projects that are already available. Opencv Sample- color blob detection, Opencv Sample face detection, Opencv Tutorial1-Camera Preview etc. Lets say you compile one of them. Now modify the code to grab few frames and pass them to opencv for stitching. If you are looking to underatnd more see into opencv stitcher class and understand how it works. Basically I grabbed few images and passed them into stitcher class of opencv which returns a stitched panoramic image.

        Best,
        ramsri

  12. Jack says:

    hi ramsri.. i want to download your project and with codes and i want to learn how can i do like that and i want to learn open cv.. i cant download it can you send your project with codes to me my mail is : forever_life@mynet.com and how can i setup the open cv with java ? how did you do android application ? i dont know how can i do that i am the beginner and can you help it. i want to create android application which is takes panoroma pictures application with open cv can you help me about this ?

  13. Where should I put the .cpp file inside the Android project?

  14. Aisha says:

    Hi Ramsri..
    i tried to run your program. I had fix the project properties and renew the path of sdk library. also the ndk. but it still cannot be run.
    the program said:

    Errors occurred during the build.
    Errors running builder ‘Integrated External Tool Builder’ on project ‘tutorial-3-native’.
    The builder launch configuration could not be found.
    The builder launch configuration could not be found.

    Do you know how to fix it?

    • Hi Aisha ,

      It has been a while since I compiled the project. I would strongly advise you to compile a project from the opencv4android library and then paste these pieces of code into the respective files. In that way you know the system is setup to run the code!

      Best,
      Ramsri

      • Aisha says:

        Dear Ramsri,
        I’m a student with very minim experience in both java and openCV..
        Do you mind if I’m asking you several questions personally? May I ask your email please?
        Thank you before, for your kindness..

  15. James says:

    Dear Ramsri,

    I just checked your project and its not working. When import in to the eclipse the jni part gives error. I use opencv 2.4.5 for android and also eclipse indigo java ee.

    How can i fix this errors or what is the solution for it ? can you help this about this issue !

    Kindly;

    • Anonymous says:

      Hi Ramsri,

      I tried to run your code but it always says “galleryactivity” cannot be resolved to a type.
      I am using eclispe and opencv2.4.5 and I can run some basic code with opencv in my envoriment. Can you point me how can I fix this error?

      Thanks

  16. Aisha says:

    hi Ramsri..
    I tried to use your attached program, and it caused an error like this:

    OpenCV Error: Assertion failed (ssize.area() > 0) in void cv::resize(cv::InputArray, cv::OutputArray, cv:: Size, double, double, int), file /home/reports/ci/slave_desktop/50-SDK/opencv/modules/improc/scr/imgwrap.cpp line 1723
    Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1)

    Why is it? And how to solve that?

    Thank you :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s