Writing a Christmas themed blog post has become something of an annual tradition for Styling Android. This year I am extremely pleased to inflict Christmas Face on to the world. Christmas Face is a simple app which allows the user to magically transform themself in to either Santa Claus or an elf. Last year’s Christmas Voice was a voice changer, and this year you can change your face instead! For those who do not celebrate Christmas: Please accept my apologies for such nonsense; and for those that do celebrate Christmas: Please accept my apologies for such nonsense!
Before we continue I am obliged to point out that working with Android camera APIs is hard. Jeff Gilfelt summed it up perfectly in a tweet from 2013 and things have improved little since then. Jeff actually spoke about some of the complexities of the Camera APIs and it makes interesting reading because while APIs may have changed, the underlying issues remain. Using Camera APIs continues to be a challenge.
For this project I elected to use the Mobile Vision library which is part of Play Services. As well as offering the face detection that is the key component of the app, it also provides a wrapper around the Camera called CameraSource which I (wrongly) assumed would make the task easier. The problem isn’t CameraSource itself but more how it interacts with different hardware camera implementations. Different underlying camera drivers and hardware mean that using CameraSource in the same way on different devices will result in different behaviours. CameraSource is actually just a wrapper around the Camera 1 APIs and so suffers from many of the same inconsistencies in behaviour.
A perfect example of this is when I was working on the image capture part of things which takes a static image which can then be shared. I got things working perfectly on an Pixel XL and then tried it on a Pixel 2 XL only to find that the image was being returned in a different orientation on the Pixel 2 XL. Moreover, images being captured from the front facing camera were rotated through 90° in one direction, but from the rear facing camera they were rotated though 90° in the opposite direction. Cue a load of conditional logic to detect for and correct these behavioural differences.
Another example was the discovery that on an Asus Zenfone 2 (always an excellent device to test for oddities that manufacturers manage to build in to devices) I found that the face detection was working perfectly during the real time preview from the camera (i.e. when the user if framing the picture with a live video feed from the camera) but as soon as a still image was captured this was coming in at a ridiculously small size (320×240 pixels) and, resulting from some optimisations I was doing elsewhere, the image being used for the face detection was far too low resolution for the face detector to actually detect faces. As it stands, the overlay drawing (of Santas hat & beard, or the elf’s hat & ears) is still very low resolution on this device while it great on others.
That touches on another common issue: differences between dealing with the live video feed from the camera, and capturing a still image. The video feed is rendered using SurfaceView and so it is not possible to simply draw this to a Bitmap, instead we need to capture a Bitmap object and separately apply the face detection and rendering of the hat overlays. This sounds like a great opportunity to reuse the face detection and rendering code, but that proved more difficult in practice. The differences that I’ve already highlighted were one thing, and another was that the front facing camera is mirrored in the video view but not in the static image. The reasoning for that is sound – for a user it is more natural to view a video of themself from the front facing camera as though they were looking in a mirror. But this makes the developers life harder nonetheless.
It is worth pointing out that that the Camera 2 API is far superior to the original Camera 1 API, but it was introduced in API 21 and is only available on Marshmallow and later devices. Moreover the Camera 1 API is actually deprecated but are still used for the Mobile Vision API because it is compatible back to Android 2.3 Gingerbread. There is a version of CameraSource which has been created using the Camera2 API but I haven’t tried it so offer no guarantees of how well it works. Also it is API25 and later which somewhat limits its usefulness for now. That limitation is why I did not use it for Christmas Face.
I realise that this post is making me sound a little like Ebenezer Scrooge and that using the Camera API at Christmas is a humbug. But I refute that: The Camera API is a humbug whatever time of year you are forced to use it! However the real point of this post is to highlight some of the complexities around using the Camera API and offer the best piece of advice if you do have to use it: Test on as many different devices as you possibly can.
This becomes the perfect opportunity to offer my warmest thanks to my testers: Sebastiano Poggi, Wiebe Elsinga, Robert Sproats, Mike Imrie, Juhani Lehtimäki, Mike Wolfson, Daniele Bonaldo, Roberto Orgiu, Jorge Barroso, David González, and Sam Edwards. Thanks a lot folks, the app would have been far worse without your help. An honourable mention should also go to my long suffering wife Karen who had to put up with me keep taking pictures of her as either Santa or an elf and showed far greater patience and tolerance than I deserve!
In the concluding article in this series we’ll actually take a look at some of the code to see some of these hacks, and also gain an understanding of how to use the face detection in the Mobile Vision API.
I apologise that I do not have the code in a fit state to publish at this point as sorting out some of these behavioural issues left with too little time to get the code properly cleaned up, but it will be published along with next weeks post, I promise.
© 2017, Mark Allison. All rights reserved.
Copyright © 2017 Styling Android. All Rights Reserved.
Information about how to reuse or republish this work may be available at http://blog.stylingandroid.com/license-information.
