File: tutorial-android-getting-started.dox

package info (click to toggle)
visp 3.7.0-5
  • links: PTS, VCS
  • area: main
  • in suites: sid
  • size: 166,332 kB
  • sloc: cpp: 392,705; ansic: 224,448; xml: 23,444; python: 13,701; java: 4,792; sh: 206; objc: 145; makefile: 60
file content (488 lines) | stat: -rw-r--r-- 21,012 bytes parent folder | download | duplicates (3)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
/**

\page tutorial-android-getting-started Tutorial: Creating a simple Android App with ViSP
\tableofcontents

\section android-getting-started 1. Getting Started
In this tutorial, we'll create an Android app demonstrating AprilTag detection using the ViSP SDK. We assume that you've
already created the SDK using this tutorial: \ref tutorial-create-android-sdk.

This tutorial assumes you have the following software installed and configured:
- <a href="https://developer.android.com/studio">Android Studio</a>
- <a href="https://developer.android.com/studio/run/emulator">Android Emulator</a> - If you want to test your apps on
an emulator first

\section android-getting-started-build 2. Building the project

\subsection android-getting-started-build-recent 2.1. On recent Android Studio

This section presents how to either create from scratch or import the sample project that is located in `samples/android`
folder.

For this section, Android Studio `Ladybug Feature Drop | 2024....` has been used.

We highly recommend to follow the steps presented in \ref android-gs-build-recent-import if you want to have the example
located in `samples/android` quickly built and running.

\subsubsection android-gs-build-recent-import 2.1.1. Importing sample project

If you carefully followed \ref tutorial-create-android-sdk, when you ran `build.py` script, a folder named
`visp-android-sdk` has been created in `$VISP_WS/visp-build-android`.

In `$VISP_WS/visp-build-android/visp-android-sdk` folder, you should find as many folders as ABI targets you have
compiled ViSP for. To work with Android emulator, you will need to use either `x86_64` or `arm64-v8a` folder depending on
your arch.

Open Android Studio.

- In the welcoming window of Android Studio, enter `"File > New > Import Project..."` menu.

- Then, in the explorator that opens, navigate until you find the `"samples"` folder in the ABI folder you want
  to use.
\image html 02-import.jpg Android sample project imported on a Macbook Pro M1.

The project is now ready to build and run. It will be explained in the \ref android-gs-build-recent-running section.

\subsubsection android-gs-build-recent-scratch 2.1.2. Creating a project from scratch

Open Android Studio.
- Enter `"File > New > New project"` menu, select in the `"Templates"` section `"Phone and Tablet"`.
 Then, choose `"Empty Views Activity"`:
\image html 01-applichoice.jpg

- Press `Next` button. Give a name to the application. We suggest to choose `ApriltagDetection` as the Java files of
  the sample application use it for the Java package naming.
  Be sure to select `"Java"` as the language and `"Groovy DSL (build.gradle)"` for the `"Build configuration language"`.
  Select the `"Minimum SDK"` as `"API 24"`:
\image html 02-language-choice.jpg
- The source code of this new `ApriltagDetection` project is located by default in `$HOME/AndroidStudioProjects/` folder.

- Edit the `build.gradle` file of the application in order to add `implementation project("visp")` in the
  `"dependencies"` section.
\image html 03-editing-build.jpg

- Copy the `sdk` folder located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/sdk` at the root of the
  project folder, at the same level than `app`.
\image html 04-copy-location.jpg

- Edit the `settings.gradle` file at the root of the project in order to add the `":visp"` project, indicating that its
  sources are located in the `sdk` folder. As shown in the next image, you should add the following lines:
\code
include ':visp'
project(':visp').projectDir = new File('sdk')
\endcode
\image html 05-editing-global.jpg

- Copy the shared library `libvisp_java3.so` located in the `sdk/native/libs/${YOUR_ABI}` folder in the destination folder
  `app/src/main/jniLibs/${YOUR_ABI}`. If `app/src/main/jniLibs/${YOUR_ABI}` folder doesn't exists, create it. We
  recall that `${YOUR_ABI}` should be set to `arm64-v8a` or `x86_64` depending on your arch.

- Copy also all `libvisp_*.a` static libraries located in the `sdk/native/staticlibs/${YOUR_ABI}` folder in the
  destination folder `app/src/main/jniLibs/${YOUR_ABI}`.

\image html 06-copying-libs.jpg Content of the app/src/main/jniLibs/${YOUR_ABI} folder after copying shared and static libraries from ViSP Android sdk.

- Finally, copy the files located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/java/com/example/apriltagdetection/`
  in the `app/src/main/java/com/example/apriltagdetection/` folder of the project.

- Copy also the folders located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/res`
  in the `app/src/main/res` folder.

\subsubsection android-gs-build-recent-running 2.1.3. Building and running the app in a simulator

Before running the app, we need to edit some settings to be able to emulate the camera.

- To this end, in Android Studio enter `"Run > Edit Configurations..."` menu.

- A configuration window should appear. In the `"Launch Options"` section, select `"Specified Activity"` in the
  `"Launch"` section. Then, in the `"Activity"` section, click on the three dots and then choose `Camera Preview Activity`.
\image html 01-camera-preview-activity.jpg

- Finally, click on `"Apply"` and then `"Run"` buttons. If you use a simulator, a panel should appear on the right-hand side
of Android Studio and display a simulated Android device.

- Our sample app indicates in the bottom how many AprilTags are detected. As shown in the next image, when presenting
  a 36h11 tag in front of the camera you should see that the tag gets detected.
\image html 02-android-app-with-tag-detected.jpg ViSP sample app emulated with the embedded camera. It indicates that one 36h11 AprilTag is detected.
  \note At this point, if your app doesn't show the camera stream, follow \ref android-gs-troubleshoutings-permissions
  section to modify app permissions to access the camera.

- After few seconds the app should close and you should see the home screen like in the following image.

\image html 03-app-home-screen.jpg Home screen that shows the app green icon.

\subsection android-gs-build-deprecated 2.2. On old Android Studio
\subsubsection android-gs-build-deprecated-scratch 2.2.1. Create an Android Project
If you're new to app development using Android Studio, we'll recommend
<a href="https://developer.android.com/training/basics/firstapp/">this tutorial</a> for getting acquainted to it.
Following <a href="https://developer.android.com/training/basics/firstapp/creating-project">this tutorial</a> create an
Android Project with an `Empty Activity`. Make sure to keep minSdkVersion >= 21, since the SDK is not compatible with
older versions. You're app's `build.gradle` file should look like:
\code
android {
    compileSdkVersion ...
    defaultConfig {
        applicationId "example.myapplication"
        minSdkVersion 21
        versionCode 1
        versionName "1.0"
	...
    }
\endcode

\subsubsection android-gs-build-deprecated-import 2.2.2. Importing ViSP SDK
In Android Studio, head to `File` -> `New` -> `Import Module` option.
\image html tut-getting-started-import-module.jpg

Head over to the directory where you've installed the SDK. Select `sdk` -> `java` folder and name the module.
\image html tut-getting-started-module-path.jpg

This only imports the Java code. You need to copy the libraries too (`.so` files in linux, `.dll` in windows and
`.dylib` in mac). Create a folder named jniLibs in `app/src/main` folder. Then depending upon your emulator/device
architecture (mostly `x86` or `x86_64`), create a folder inside `jniLibs` and copy those libraries into your project.
\image html tut-getting-started-copy-libs.jpg

\section android-gs-code 3. Code analysis

We will now study with more details the code available in the `samples/android` folder.

First, in `MainActivity.java`, we need to load ViSP libraries:
\code
public class MainActivity{

    // Used to load the 'native-lib' library on application startup.
    static {
        System.loadLibrary("visp_java3");
    }

    ...
}
\endcode

\subsection android-gs-code-camera 3.1. Begin Camera Preview
In order to be able to use the camera, we need to ask user for Camera Permissions. First, in the
`app/src/main/AndroidManifest.xml`
manifest file, we need to include:
\code
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
        package="...">

    <uses-permission android:name="android.permission.CAMERA" />

    <uses-feature
        android:name="android.hardware.camera"
        android:required="true" />

    <application ...>
        ...
    </application>
</manifest>
\endcode

Then, we need to add a <a href="https://developer.android.com/training/permissions/requesting">runtime permission</a>
for accessing camera in `MainActivity.java`. Note that detection will execute only when user allows camera access:
\code
        // Check if the Camera permission has been granted
        if (ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
            // Permission is already available, start camera preview
            Snackbar.make(mLayout,
                    "Camera permission is available. Starting preview.",
                    Snackbar.LENGTH_SHORT).show();
            startCamera();
        } else {
            // Permission is missing and must be requested.
            requestCameraPermission();
        }
\endcode

And finally we implement a request callback listener:
\code

public class MainActivity extends AppCompatActivity implements ActivityCompat.OnRequestPermissionsResultCallback {

    ...

    @Override
    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions,
                                           @NonNull int[] grantResults) {
        if (requestCode == PERMISSION_REQUEST_CAMERA) {
            // Request for camera permission.
            if (grantResults.length == 1 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                // Permission has been granted. Start camera preview Activity.
		Intent intent = new Intent(this, CameraPreviewActivity.class);
                startActivity(intent);
            } else {
                // TODO Permission request was denied. Do something
            }
        }
    }

    ...
}
\endcode

\subsection android-gs-code-start-camera 3.2. Starting Camera Preview
Now we will study the `CameraPreviewActivity.java`. This will call the camera API. The incident image is recieved as a
byte array which can be easily manipulated for our purposes. We can render the resultant image as Java Bitmap in an
`ImageView` element. In brief,
\code
public class CameraPreviewActivity extends MainActivity {

    private Camera mCamera;
    public static ImageView resultImageView;
    ...

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // Open an instance of the first camera and retrieve its info.
        mCamera = getCameraInstance(CAMERA_ID);
        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
        Camera.getCameraInfo(CAMERA_ID, cameraInfo);

        if (mCamera == null) {
            // Camera is not available, display error message
            Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
            setContentView(R.layout.camera_unavailable);
        } else {

            setContentView(R.layout.activity_camera_preview);

            resultImageView = findViewById(R.id.resultImage);

	    ...

            // Get the rotation of the screen to adjust the preview image accordingly.
            final int displayRotation = getWindowManager().getDefaultDisplay()
                    .getRotation();
	    ...
        }
    }
\endcode

Now that we get access to Camera, we need to create a Camera Preview class that'll  process the image for AprilTag
detection. This class is implemented in the `CameraPreview.java` file. In brief,
\code
public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback {

    private int w,h;
    private VpCameraParameters cameraParameters;
    private double tagSize;

    public CameraPreview(Context context, Camera camera, Camera.CameraInfo cameraInfo,
                         int displayOrientation) {
        super(context);

        // Do not initialize if no camera has been set
        if (camera == null || cameraInfo == null) {
            return;
        }

        mCamera = camera;
        mCameraInfo = cameraInfo;
        mDisplayOrientation = displayOrientation;

	...

        // init the ViSP tag detection system
        w = mCamera.getParameters().getPreviewSize().width;
        h = mCamera.getParameters().getPreviewSize().height;
        cameraParameters = new VpCameraParameters();
        cameraParameters.initPersProjWithoutDistortion(615.1674805, 615.1675415, 312.1889954, 243.4373779);
        tagSize = 0.053;
    }

    ...

    public void onPreviewFrame(byte[] data, Camera camera) {

        if (System.currentTimeMillis() > 50 + lastTime) {

            VpImageUChar imageUChar = new VpImageUChar(data,h,w,true);

            // do the image processing
            // Its working even without grey scale conversion
            VpDetectorAprilTag detectorAprilTag = new VpDetectorAprilTag();
            List<VpHomogeneousMatrix> matrices = detectorAprilTag.detect(imageUChar,tagSize,cameraParameters);
            Log.d("CameraPreview.java",matrices.size() + " tags detected");

            // Turn `data` to bitmap and display
            updateResult(data, matrices.size() + " 36h11 tags detected within " + (System.currentTimeMillis() - lastTime) +" ms");

            lastTime = System.currentTimeMillis();
        }
    }
}
\endcode

Note that the detector works on grayscale images. The camera API returns values for all pixels (R,G,B,A).
Depending on the image format rendered in Android, we can convert those color values into grayscale.
<a href="https://developer.android.com/reference/android/graphics/ImageFormat">Refer this</a> page, for a complete list
of formats.
Most commonly used format is `NV21`. In it, the first few bytes are grayscale values of the image and rest are used to
compute the color image. So AprilTag detector process only first `width*height` bytes of the image as `VpImageUChar`.

Also, we're detecting tags every 50 milli-seconds. This is simple but efficient since actual tag detection time will
vary according to the image and should be an asynchronous task.

If we wanted to display the result of the conversion, we would need to change CameraPreviewActivity accordingly,
\code
public class CameraPreviewActivity extends AppCompatActivity {

    ...
    public static ImageView resultImageView;
    ...

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        // Open an instance of the first camera and retrieve its info.
        mCamera = getCameraInstance(CAMERA_ID);
        Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
        Camera.getCameraInfo(CAMERA_ID, cameraInfo);

        if (mCamera == null) {
            // Camera is not available, display error message
            Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
            setContentView(R.layout.camera_unavailable);
        } else {
            setContentView(R.layout.activity_camera_preview);

            resultInfo = findViewById(R.id.resultTV);
            resultImageView = findViewById(R.id.resultImage);

            // init the byte array
            w = mCamera.getParameters().getPreviewSize().width;
            h = mCamera.getParameters().getPreviewSize().height;

            // Get the rotation of the screen to adjust the preview image accordingly.
            final int displayRotation = getWindowManager().getDefaultDisplay()
                    .getRotation();

            // Create the Preview view and set it as the content of this Activity.
            CameraPreview mPreview = new CameraPreview(this, mCamera, cameraInfo, displayRotation);
            FrameLayout preview = findViewById(R.id.camera_preview);
            preview.addView(mPreview);
        }
    }

    public static void updateResult(byte[] Src){
        byte [] Bits = new byte[Src.length*4]; //That's where the RGBA array goes.
        int i;
        for(i=0;i<Src.length;i++){
            Bits[i*4] = Bits[i*4+1] = Bits[i*4+2] = Src[i]; //Invert the source bits
            Bits[i*4+3] = -1;//0xff, that's the alpha.
        }

        //Now put these nice RGBA pixels into a Bitmap object
        Bitmap bm = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
        bm.copyPixelsFromBuffer(ByteBuffer.wrap(Bits));

        resultImageView.setImageBitmap(bm);
    }

    ...
}
\endcode

Note the inversion in `updateResult` method. Visp in C++ accepts image as sequence of `RGBA` values but Java Bitmap
process them as `ARGB`.

However, note that in the sample, the conversion and display of the conversion has been commented out and we instead just
write how many tags have been detected.

\subsection android-gs-further-development 3.3. Further Image manipulation
In this tutorial, we've developed a bare bones tag detection app.
We can use <a href="https://developer.android.com/guide/topics/graphics/opengl">OpenGL for Android</a> to manipulate the
image (for instance, drawing a 3D arrow on the tags) using the list of `VpHomogeneous` matrices.
You can find the complete source code of above Android App
<a href="https://github.com/lagadic/visp/tree/master/samples/android">here</a>.

\section android-gs-troubleshoutings 4. Potential troubleshoutings

\subsection android-gs-troubleshoutings-permissions 4.1. The application does not detect a camera

When you try to launch the application, you may face the following error message `"Camera unavailable"`:

\image html 01-no-permissions.jpg

It is because the permission to use the camera has not been granted.

- First, click holding your mouse on the application icon. The following menu should appear:

\image html 02-opening-menu.jpg

- Click on `"App info"`. Then, click on the `"Permissions - No permissions granted"` item in the menu that will open:

\image html 03-opening-permissions.jpg

- Then, click on the `"Camera"` item that appears in the `"Not allowed"` section:

\image html 04-selecting-camera.jpg

- Finally, click on the `"Allow only while using the app"` item of the menu that appears.

\image html 05-giving-permission.jpg

- You should now be able to use the embedded camera. Otherwise, check \ref android-gs-troubleshoutings-nothing section.

\subsection android-gs-troubleshoutings-nothing 4.2. The application displays nothing or not what is expected

When using the app, you may not see the image acquired by your camera when the application is running.
This is because you must parameterize the simulator in order to use your camera.

- First, click on the `"Device manager"` menu. It is either in the `"Tools"` section of the menu on top of Android Studio
  or on the right side of the window as follow:

\image html 10-device-manager.jpg

- Click on the three vertically-aligned dots that are beside the simulator, and then on `"Edit"`:

\image html 11-device-settings.jpg

- Click on the `"Show Advanced Settings"` that is located in the bottom of the window:

\image html 12-opening-advanced.jpg

- In the `"Camera"` settings, select the camera you want to use. In this example, the integrated camera of the
computer is used:

\image html 13-camera-settings.jpg

\subsection android-gs-troubleshoutings-ram 4.3. The application freezes

The application might lag in the simulator. It might be due to the amount of RAM that is allocated to the camera.

- To change that, open the `"Device Manager"` panel. It is either in the `"Tools"` section of the menu on top of
  Android Studio or on the right side of the window as follow:

\image html 10-device-manager.jpg

- Then, click on the three vertically-aligned dots that are beside the simulator, and then on `"Show on Disk"`.

\image html 20-locating-image.jpg

- In the files explorator, open the `"hardware-qemu.ini"` file using a text editor software:

\image html 21-opening-file.jpg

- Modify the value specified by the `hw.ramSize` item:

\image html 22-modifying-ram.jpg

\subsection android-gs-troubleshoutings-openmp 4.4. The application immediately closes

When you launch the application, it immediately closes without displaying anything. It might
be due to an OpenMP problem.

- Open the `"Logcat"` terminal, that is located on the bottom left of the screen:

\image html 30-logs.jpg

- Then, click on the icon of the application to try to launch it. If a message in red appears
  complaining about `libopenmp.so`, it is because ViSP has not been compiled with OpenMP as
  static library. Please open an issue on [github](https://github.com/lagadic/visp).

*/