1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488
|
/**
\page tutorial-android-getting-started Tutorial: Creating a simple Android App with ViSP
\tableofcontents
\section android-getting-started 1. Getting Started
In this tutorial, we'll create an Android app demonstrating AprilTag detection using the ViSP SDK. We assume that you've
already created the SDK using this tutorial: \ref tutorial-create-android-sdk.
This tutorial assumes you have the following software installed and configured:
- <a href="https://developer.android.com/studio">Android Studio</a>
- <a href="https://developer.android.com/studio/run/emulator">Android Emulator</a> - If you want to test your apps on
an emulator first
\section android-getting-started-build 2. Building the project
\subsection android-getting-started-build-recent 2.1. On recent Android Studio
This section presents how to either create from scratch or import the sample project that is located in `samples/android`
folder.
For this section, Android Studio `Ladybug Feature Drop | 2024....` has been used.
We highly recommend to follow the steps presented in \ref android-gs-build-recent-import if you want to have the example
located in `samples/android` quickly built and running.
\subsubsection android-gs-build-recent-import 2.1.1. Importing sample project
If you carefully followed \ref tutorial-create-android-sdk, when you ran `build.py` script, a folder named
`visp-android-sdk` has been created in `$VISP_WS/visp-build-android`.
In `$VISP_WS/visp-build-android/visp-android-sdk` folder, you should find as many folders as ABI targets you have
compiled ViSP for. To work with Android emulator, you will need to use either `x86_64` or `arm64-v8a` folder depending on
your arch.
Open Android Studio.
- In the welcoming window of Android Studio, enter `"File > New > Import Project..."` menu.
- Then, in the explorator that opens, navigate until you find the `"samples"` folder in the ABI folder you want
to use.
\image html 02-import.jpg Android sample project imported on a Macbook Pro M1.
The project is now ready to build and run. It will be explained in the \ref android-gs-build-recent-running section.
\subsubsection android-gs-build-recent-scratch 2.1.2. Creating a project from scratch
Open Android Studio.
- Enter `"File > New > New project"` menu, select in the `"Templates"` section `"Phone and Tablet"`.
Then, choose `"Empty Views Activity"`:
\image html 01-applichoice.jpg
- Press `Next` button. Give a name to the application. We suggest to choose `ApriltagDetection` as the Java files of
the sample application use it for the Java package naming.
Be sure to select `"Java"` as the language and `"Groovy DSL (build.gradle)"` for the `"Build configuration language"`.
Select the `"Minimum SDK"` as `"API 24"`:
\image html 02-language-choice.jpg
- The source code of this new `ApriltagDetection` project is located by default in `$HOME/AndroidStudioProjects/` folder.
- Edit the `build.gradle` file of the application in order to add `implementation project("visp")` in the
`"dependencies"` section.
\image html 03-editing-build.jpg
- Copy the `sdk` folder located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/sdk` at the root of the
project folder, at the same level than `app`.
\image html 04-copy-location.jpg
- Edit the `settings.gradle` file at the root of the project in order to add the `":visp"` project, indicating that its
sources are located in the `sdk` folder. As shown in the next image, you should add the following lines:
\code
include ':visp'
project(':visp').projectDir = new File('sdk')
\endcode
\image html 05-editing-global.jpg
- Copy the shared library `libvisp_java3.so` located in the `sdk/native/libs/${YOUR_ABI}` folder in the destination folder
`app/src/main/jniLibs/${YOUR_ABI}`. If `app/src/main/jniLibs/${YOUR_ABI}` folder doesn't exists, create it. We
recall that `${YOUR_ABI}` should be set to `arm64-v8a` or `x86_64` depending on your arch.
- Copy also all `libvisp_*.a` static libraries located in the `sdk/native/staticlibs/${YOUR_ABI}` folder in the
destination folder `app/src/main/jniLibs/${YOUR_ABI}`.
\image html 06-copying-libs.jpg Content of the app/src/main/jniLibs/${YOUR_ABI} folder after copying shared and static libraries from ViSP Android sdk.
- Finally, copy the files located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/java/com/example/apriltagdetection/`
in the `app/src/main/java/com/example/apriltagdetection/` folder of the project.
- Copy also the folders located in `$VISP_WS/visp-build-android/visp-android-sdk/${YOUR_ABI}/samples/app/src/main/res`
in the `app/src/main/res` folder.
\subsubsection android-gs-build-recent-running 2.1.3. Building and running the app in a simulator
Before running the app, we need to edit some settings to be able to emulate the camera.
- To this end, in Android Studio enter `"Run > Edit Configurations..."` menu.
- A configuration window should appear. In the `"Launch Options"` section, select `"Specified Activity"` in the
`"Launch"` section. Then, in the `"Activity"` section, click on the three dots and then choose `Camera Preview Activity`.
\image html 01-camera-preview-activity.jpg
- Finally, click on `"Apply"` and then `"Run"` buttons. If you use a simulator, a panel should appear on the right-hand side
of Android Studio and display a simulated Android device.
- Our sample app indicates in the bottom how many AprilTags are detected. As shown in the next image, when presenting
a 36h11 tag in front of the camera you should see that the tag gets detected.
\image html 02-android-app-with-tag-detected.jpg ViSP sample app emulated with the embedded camera. It indicates that one 36h11 AprilTag is detected.
\note At this point, if your app doesn't show the camera stream, follow \ref android-gs-troubleshoutings-permissions
section to modify app permissions to access the camera.
- After few seconds the app should close and you should see the home screen like in the following image.
\image html 03-app-home-screen.jpg Home screen that shows the app green icon.
\subsection android-gs-build-deprecated 2.2. On old Android Studio
\subsubsection android-gs-build-deprecated-scratch 2.2.1. Create an Android Project
If you're new to app development using Android Studio, we'll recommend
<a href="https://developer.android.com/training/basics/firstapp/">this tutorial</a> for getting acquainted to it.
Following <a href="https://developer.android.com/training/basics/firstapp/creating-project">this tutorial</a> create an
Android Project with an `Empty Activity`. Make sure to keep minSdkVersion >= 21, since the SDK is not compatible with
older versions. You're app's `build.gradle` file should look like:
\code
android {
compileSdkVersion ...
defaultConfig {
applicationId "example.myapplication"
minSdkVersion 21
versionCode 1
versionName "1.0"
...
}
\endcode
\subsubsection android-gs-build-deprecated-import 2.2.2. Importing ViSP SDK
In Android Studio, head to `File` -> `New` -> `Import Module` option.
\image html tut-getting-started-import-module.jpg
Head over to the directory where you've installed the SDK. Select `sdk` -> `java` folder and name the module.
\image html tut-getting-started-module-path.jpg
This only imports the Java code. You need to copy the libraries too (`.so` files in linux, `.dll` in windows and
`.dylib` in mac). Create a folder named jniLibs in `app/src/main` folder. Then depending upon your emulator/device
architecture (mostly `x86` or `x86_64`), create a folder inside `jniLibs` and copy those libraries into your project.
\image html tut-getting-started-copy-libs.jpg
\section android-gs-code 3. Code analysis
We will now study with more details the code available in the `samples/android` folder.
First, in `MainActivity.java`, we need to load ViSP libraries:
\code
public class MainActivity{
// Used to load the 'native-lib' library on application startup.
static {
System.loadLibrary("visp_java3");
}
...
}
\endcode
\subsection android-gs-code-camera 3.1. Begin Camera Preview
In order to be able to use the camera, we need to ask user for Camera Permissions. First, in the
`app/src/main/AndroidManifest.xml`
manifest file, we need to include:
\code
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="...">
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature
android:name="android.hardware.camera"
android:required="true" />
<application ...>
...
</application>
</manifest>
\endcode
Then, we need to add a <a href="https://developer.android.com/training/permissions/requesting">runtime permission</a>
for accessing camera in `MainActivity.java`. Note that detection will execute only when user allows camera access:
\code
// Check if the Camera permission has been granted
if (ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED) {
// Permission is already available, start camera preview
Snackbar.make(mLayout,
"Camera permission is available. Starting preview.",
Snackbar.LENGTH_SHORT).show();
startCamera();
} else {
// Permission is missing and must be requested.
requestCameraPermission();
}
\endcode
And finally we implement a request callback listener:
\code
public class MainActivity extends AppCompatActivity implements ActivityCompat.OnRequestPermissionsResultCallback {
...
@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions,
@NonNull int[] grantResults) {
if (requestCode == PERMISSION_REQUEST_CAMERA) {
// Request for camera permission.
if (grantResults.length == 1 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
// Permission has been granted. Start camera preview Activity.
Intent intent = new Intent(this, CameraPreviewActivity.class);
startActivity(intent);
} else {
// TODO Permission request was denied. Do something
}
}
}
...
}
\endcode
\subsection android-gs-code-start-camera 3.2. Starting Camera Preview
Now we will study the `CameraPreviewActivity.java`. This will call the camera API. The incident image is recieved as a
byte array which can be easily manipulated for our purposes. We can render the resultant image as Java Bitmap in an
`ImageView` element. In brief,
\code
public class CameraPreviewActivity extends MainActivity {
private Camera mCamera;
public static ImageView resultImageView;
...
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
// Open an instance of the first camera and retrieve its info.
mCamera = getCameraInstance(CAMERA_ID);
Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
Camera.getCameraInfo(CAMERA_ID, cameraInfo);
if (mCamera == null) {
// Camera is not available, display error message
Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
setContentView(R.layout.camera_unavailable);
} else {
setContentView(R.layout.activity_camera_preview);
resultImageView = findViewById(R.id.resultImage);
...
// Get the rotation of the screen to adjust the preview image accordingly.
final int displayRotation = getWindowManager().getDefaultDisplay()
.getRotation();
...
}
}
\endcode
Now that we get access to Camera, we need to create a Camera Preview class that'll process the image for AprilTag
detection. This class is implemented in the `CameraPreview.java` file. In brief,
\code
public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback {
private int w,h;
private VpCameraParameters cameraParameters;
private double tagSize;
public CameraPreview(Context context, Camera camera, Camera.CameraInfo cameraInfo,
int displayOrientation) {
super(context);
// Do not initialize if no camera has been set
if (camera == null || cameraInfo == null) {
return;
}
mCamera = camera;
mCameraInfo = cameraInfo;
mDisplayOrientation = displayOrientation;
...
// init the ViSP tag detection system
w = mCamera.getParameters().getPreviewSize().width;
h = mCamera.getParameters().getPreviewSize().height;
cameraParameters = new VpCameraParameters();
cameraParameters.initPersProjWithoutDistortion(615.1674805, 615.1675415, 312.1889954, 243.4373779);
tagSize = 0.053;
}
...
public void onPreviewFrame(byte[] data, Camera camera) {
if (System.currentTimeMillis() > 50 + lastTime) {
VpImageUChar imageUChar = new VpImageUChar(data,h,w,true);
// do the image processing
// Its working even without grey scale conversion
VpDetectorAprilTag detectorAprilTag = new VpDetectorAprilTag();
List<VpHomogeneousMatrix> matrices = detectorAprilTag.detect(imageUChar,tagSize,cameraParameters);
Log.d("CameraPreview.java",matrices.size() + " tags detected");
// Turn `data` to bitmap and display
updateResult(data, matrices.size() + " 36h11 tags detected within " + (System.currentTimeMillis() - lastTime) +" ms");
lastTime = System.currentTimeMillis();
}
}
}
\endcode
Note that the detector works on grayscale images. The camera API returns values for all pixels (R,G,B,A).
Depending on the image format rendered in Android, we can convert those color values into grayscale.
<a href="https://developer.android.com/reference/android/graphics/ImageFormat">Refer this</a> page, for a complete list
of formats.
Most commonly used format is `NV21`. In it, the first few bytes are grayscale values of the image and rest are used to
compute the color image. So AprilTag detector process only first `width*height` bytes of the image as `VpImageUChar`.
Also, we're detecting tags every 50 milli-seconds. This is simple but efficient since actual tag detection time will
vary according to the image and should be an asynchronous task.
If we wanted to display the result of the conversion, we would need to change CameraPreviewActivity accordingly,
\code
public class CameraPreviewActivity extends AppCompatActivity {
...
public static ImageView resultImageView;
...
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
// Open an instance of the first camera and retrieve its info.
mCamera = getCameraInstance(CAMERA_ID);
Camera.CameraInfo cameraInfo = new Camera.CameraInfo();
Camera.getCameraInfo(CAMERA_ID, cameraInfo);
if (mCamera == null) {
// Camera is not available, display error message
Toast.makeText(this, "Camera is not available.", Toast.LENGTH_SHORT).show();
setContentView(R.layout.camera_unavailable);
} else {
setContentView(R.layout.activity_camera_preview);
resultInfo = findViewById(R.id.resultTV);
resultImageView = findViewById(R.id.resultImage);
// init the byte array
w = mCamera.getParameters().getPreviewSize().width;
h = mCamera.getParameters().getPreviewSize().height;
// Get the rotation of the screen to adjust the preview image accordingly.
final int displayRotation = getWindowManager().getDefaultDisplay()
.getRotation();
// Create the Preview view and set it as the content of this Activity.
CameraPreview mPreview = new CameraPreview(this, mCamera, cameraInfo, displayRotation);
FrameLayout preview = findViewById(R.id.camera_preview);
preview.addView(mPreview);
}
}
public static void updateResult(byte[] Src){
byte [] Bits = new byte[Src.length*4]; //That's where the RGBA array goes.
int i;
for(i=0;i<Src.length;i++){
Bits[i*4] = Bits[i*4+1] = Bits[i*4+2] = Src[i]; //Invert the source bits
Bits[i*4+3] = -1;//0xff, that's the alpha.
}
//Now put these nice RGBA pixels into a Bitmap object
Bitmap bm = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
bm.copyPixelsFromBuffer(ByteBuffer.wrap(Bits));
resultImageView.setImageBitmap(bm);
}
...
}
\endcode
Note the inversion in `updateResult` method. Visp in C++ accepts image as sequence of `RGBA` values but Java Bitmap
process them as `ARGB`.
However, note that in the sample, the conversion and display of the conversion has been commented out and we instead just
write how many tags have been detected.
\subsection android-gs-further-development 3.3. Further Image manipulation
In this tutorial, we've developed a bare bones tag detection app.
We can use <a href="https://developer.android.com/guide/topics/graphics/opengl">OpenGL for Android</a> to manipulate the
image (for instance, drawing a 3D arrow on the tags) using the list of `VpHomogeneous` matrices.
You can find the complete source code of above Android App
<a href="https://github.com/lagadic/visp/tree/master/samples/android">here</a>.
\section android-gs-troubleshoutings 4. Potential troubleshoutings
\subsection android-gs-troubleshoutings-permissions 4.1. The application does not detect a camera
When you try to launch the application, you may face the following error message `"Camera unavailable"`:
\image html 01-no-permissions.jpg
It is because the permission to use the camera has not been granted.
- First, click holding your mouse on the application icon. The following menu should appear:
\image html 02-opening-menu.jpg
- Click on `"App info"`. Then, click on the `"Permissions - No permissions granted"` item in the menu that will open:
\image html 03-opening-permissions.jpg
- Then, click on the `"Camera"` item that appears in the `"Not allowed"` section:
\image html 04-selecting-camera.jpg
- Finally, click on the `"Allow only while using the app"` item of the menu that appears.
\image html 05-giving-permission.jpg
- You should now be able to use the embedded camera. Otherwise, check \ref android-gs-troubleshoutings-nothing section.
\subsection android-gs-troubleshoutings-nothing 4.2. The application displays nothing or not what is expected
When using the app, you may not see the image acquired by your camera when the application is running.
This is because you must parameterize the simulator in order to use your camera.
- First, click on the `"Device manager"` menu. It is either in the `"Tools"` section of the menu on top of Android Studio
or on the right side of the window as follow:
\image html 10-device-manager.jpg
- Click on the three vertically-aligned dots that are beside the simulator, and then on `"Edit"`:
\image html 11-device-settings.jpg
- Click on the `"Show Advanced Settings"` that is located in the bottom of the window:
\image html 12-opening-advanced.jpg
- In the `"Camera"` settings, select the camera you want to use. In this example, the integrated camera of the
computer is used:
\image html 13-camera-settings.jpg
\subsection android-gs-troubleshoutings-ram 4.3. The application freezes
The application might lag in the simulator. It might be due to the amount of RAM that is allocated to the camera.
- To change that, open the `"Device Manager"` panel. It is either in the `"Tools"` section of the menu on top of
Android Studio or on the right side of the window as follow:
\image html 10-device-manager.jpg
- Then, click on the three vertically-aligned dots that are beside the simulator, and then on `"Show on Disk"`.
\image html 20-locating-image.jpg
- In the files explorator, open the `"hardware-qemu.ini"` file using a text editor software:
\image html 21-opening-file.jpg
- Modify the value specified by the `hw.ramSize` item:
\image html 22-modifying-ram.jpg
\subsection android-gs-troubleshoutings-openmp 4.4. The application immediately closes
When you launch the application, it immediately closes without displaying anything. It might
be due to an OpenMP problem.
- Open the `"Logcat"` terminal, that is located on the bottom left of the screen:
\image html 30-logs.jpg
- Then, click on the icon of the application to try to launch it. If a message in red appears
complaining about `libopenmp.so`, it is because ViSP has not been compiled with OpenMP as
static library. Please open an issue on [github](https://github.com/lagadic/visp).
*/
|