Writing a Christmas themed blog post has become something of an annual tradition for Styling Android. This year I am extremely pleased to inflict Christmas Face on to the world. Christmas Face is a simple app which allows the user to magically transform themself in to either Santa Claus or an elf. Last year’s Christmas Voice was a voice changer, and this year you can change your face instead! For those who do not celebrate Christmas: Please accept my apologies for such nonsense; and for those that do celebrate Christmas: Please accept my apologies for such nonsense!
I’m not going to give a detailed explanation of every aspect of Christmas Face, the source code has been published so please feel free to browse it to your heart’s content. However I will cover some of the fundamentals of how it uses the Mobile Vision API to perform face detection, and then how we overlay the Santa or elf hats and stuff.
Most of this can be seen in a single class named BitmapGenerator.kt. This is responsible for performing some optimisations to the image that we receive from the CameraSource, then performing the face detection, overlaying the relevant graphical component at the correct location, and finally saving it to a JPEG file in the file cache of the app. The convert() function does all of this
const val ASSET_NAME = "image.jpg"
class BitmapGenerator(
private val context: Context,
@DrawableRes private val drawableId: Int,
private val width: Int,
private val height: Int,
private val orientationFactor: Float = 1f) {
private val cacheFile: File by lazyFast {
File(context.filesDir, ASSET_NAME)
}
private val isPortrait: Boolean
get() = height > width
private val Bitmap.isPortrait: Boolean
get() = height > width
private val drawable: Drawable? by lazyFast {
ResourcesCompat.getDrawable(context.resources, drawableId, context.theme)
}
private var scaleFactor: Float = 1f
suspend fun convert(bytes: ByteArray): Bitmap = async(CommonPool) {
BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
.rotateIfNecessary().let { newBitmap ->
newBitmap.detectFace()?.let { face ->
newBitmap.createOverlaidBitmap(face)
} ?: newBitmap
}.also {
writeCacheFile(it)
}
}.await()
.
.
.
}
One thing worth noting is that this is all executed within an async block which is a Kotlin coroutine ensuring that we do not block the main thread, and our app remains responsive during this computationally expensive process.
Another thing worthy of mention is the orientationFactor value which is one of the constructor arguments. This is used to apply an arbitrary correction factor to the orientation of the image should it require rotation to match the orientation that we are outputting. We’ll see how this is used later, but it is used to alter the direction that we rotate the image. In the first article in this series, we covered how the image gets rotated differently depending on whether the front or rear camera is used, and this enables us to apply the appropriate correction while keeping this class completely agnostic of the differing behaviours of the front and rear cameras.
The rotation itself is performed by the following methods:
private fun Bitmap.rotateIfNecessary(): Bitmap =
if (shouldRotate(this)) {
rotate()
} else {
this
}
private fun shouldRotate(bitmap: Bitmap): Boolean =
isPortrait != bitmap.isPortrait
private fun Bitmap.rotate(): Bitmap =
Bitmap.createBitmap(height, width, config).apply {
Canvas(this).apply {
rotate(90f * orientationFactor)
matrix = Matrix().apply {
if (orientationFactor > 0f) {
postTranslate(0f, [email protected]())
} else {
postTranslate([email protected](), 0f)
}
}.also {
drawBitmap(this@rotate, it, null)
}
}
}
The orientationFactor is applied to the rotation on line 104, but then we also need to move the image inside the view port as per the conditional block in lines 106-110.
I’ll be the first to admit that this could do with a little tidying up, but time to do so escaped me, I’m afraid.
The face detection itself is actually reasonably straightforward:
private fun Bitmap.detectFace(): Face? =
createFaceDetector().run {
detect(Frame.Builder().setBitmap(scale()).build())?.first().apply {
release()
}
}
private fun createFaceDetector(): FaceDetector =
FaceDetector.Builder(context)
.setClassificationType(FaceDetector.NO_CLASSIFICATIONS)
.setLandmarkType(FaceDetector.ALL_LANDMARKS)
.build()
The detect() function of FaceDetector does all of the hard work, we provide it with a Frame instance, which we create from the Bitmap, and it returns an array of Face instances which represent all of the detected faces in the image. In this case we’re only interested in the first one.
One thing worth mentioning is that face detection is quite slow, and gets exponentially slower the larger the bitmap is. It doesn’t actually require a very high resolution image to be able to quite accurately detect faces, and that is the purpose of the scale() function – it will scale down the image in order to decrease the time needed for face detection.
The scale() function is a little more complex that it was originally because of the image size issue on an Asus Zenfone 2 that I mentioned in the previous article:
private fun Bitmap.scale(): Bitmap {
scaleFactor = Math.min(640f / Math.max(width, height).toFloat(), 1f)
return if (scaleFactor < 1f) {
Bitmap.createBitmap((width * scaleFactor).toInt(), (height * scaleFactor).toInt(), config).also {
Canvas(it).apply {
scale(scaleFactor, scaleFactor)
drawBitmap(this@scale, 0f, 0f, null)
}
}
} else {
this
}
}
The scaleFactor is the amount that we need to scale down the image, and becomes important later on when we need to overlay the Santa / elf drawables, as the Face items that are detected will contain coordinates in the coordinate space of the scaled down image, and we’ll need to map them back to the coordinate space of the original image.
The fix for the Asus and other devices which return small images is the check that the scaleFactor is less than 1. We don’t perform any scaling if it is because it means that the maximum dimension of the image from the camera is less than 640 pixels.
Next we draw the overlay:
private fun Bitmap.createOverlaidBitmap(face: Face): Bitmap =
Bitmap.createBitmap(width, height, config).apply {
Canvas(this).apply {
drawBitmap(this@createOverlaidBitmap, 0f, 0f, null)
scale(1f / scaleFactor, 1f / scaleFactor)
drawable?.draw(this, face)
}
}
private fun Drawable.draw(canvas: Canvas, face: Face) {
bounds.left = (face.position.x).toInt()
bounds.right = (face.position.x + face.width).toInt()
bounds.top = (face.position.y).toInt() - (face.height / 4).toInt()
bounds.bottom = (face.position.y + face.height).toInt() + (face.height.toInt() / 8)
canvas.rotate(-face.eulerZ, bounds.exactCenterX(), bounds.exactCenterY())
draw(canvas)
}
The scaleFactor is used on line 64 to scale the coordinates back to those of the source image.
The draw() function paints the Drawable to the Canvas. Setting the bounds of the drawable before it is drawn to the canvas position and scale it accordingly. I have added some somewhat hacky corrections to the vertical positioning and scaling to get the drawables that I have used to fit to most real heads. The actual values for these corrections were found through simple trial and error.
Finally we write the overlaid Bitmap to a JPEG file:
private fun writeCacheFile(bitmap: Bitmap): Boolean {
var outputStream: OutputStream? = null
return try {
outputStream = FileOutputStream(cacheFile)
bitmap.toJpeg().apply {
outputStream.write(this, 0, size)
}
true
} catch (e: IOException) {
Log.e(BitmapGenerator::class.java.name, "Error writing to cache file", e)
false
} finally {
outputStream?.close()
}
}
private fun Bitmap.toJpeg(): ByteArray =
ByteArrayOutputStream().run {
compress(Bitmap.CompressFormat.JPEG, 80, this)
toByteArray()
}
This is fairly standard stuff so (hopefully!) does not require any further explanation.
The only other area of the code that may be unfamiliar with some is how we then share this JPEG file with other apps. The code for this is in the onCreateOptionsMenu() function in PreviewFragment. I used a FileProvider to achieve this, and FileProvider has already been covered by a previous article, so those that are interested can learn about the technique through that article.
As I mentioned earlier, this article is just a skim through of some of the salient areas, but the full source is available here.
© 2017, Mark Allison. All rights reserved.
Copyright © 2017 Styling Android. All Rights Reserved.
Information about how to reuse or republish this work may be available at http://blog.stylingandroid.com/license-information.
