In this series we’re going to build an app to view the Mandelbrot set. While this is not necessarily something that is likely to be of direct use to the majority of app developers (except for those that have a desire to develop their own Mandelbrot set viewing apps), there are some techniques that we’ll explore along the way that most certainly will be of relevance to many.
Previously we got panning and zooming working so that we can explore the Mandelbrot set, and we now have a fully functional app. However, there is still a usability issue with it. Even though we are rendering the Mandelbrot set using Renderscript, the performance is still a little slow and when we pan and zoom, it is taking a couple of seconds in some cases to render the image. The big problem here is that it makes it quite slow to navigate when we have to wait to see what effect our panning and zooming is having, so it would be much nicer to have more immediate visual feedback. One approach that we can use is to render the image at a much lower resolution initially and then progressively re-render it at higher resolutions when the viewport becomes stable. When I was developing the app, I discussed this with Sebastiano Poggi who suggested adjusting this scaling based upon that time it took to render the previous frame. However when I started playing with it, I found that simply starting at a 16x reduction in resolution could reliably generate new frames in under 10ms (on a Pixel 2XL) so I just went with a a static value rather than following Seb’s suggestion of trying to be dynamic about it. If I were developing this commercially, then I might be inclined to follow Seb’s suggestion which is why I mentioned it here.
So the approach that we’re going to take is that we’ll render the frame at 16x reduction in resolution and, once that is complete, we’ll display the image. If we receive a touch event during rendering, then we’ll next render another 16x reduction for the updated view port following the touch event. However, if we didn’t receive a touch event (so the view port hasn’t changed), we’ll next do a render at 8x reduction. After that, we’ll render at 4x reduction, then 2x reduction, then finally at full resolution. If we receive a touch event between any of these renders, then we’ll drop back to the 16x reduction and begin again for the new view port parameters.
Essentially this will keep rendering at low resolution while touch events are being received, and the view port is changing, but once it becomes stable then we’re progressively increase the resolution.
During testing I found that often we could render at both 16x reduction and then 8x reduction between successive touch events, so we were effectively getting an 8x reduction preview image when that was happening. However, it still kept things responsive for computationally complex areas of the Mandelbrot set when we’d only manage a 16x reduction preview.
The majority of the changes are to MandelbrotRenderer.kt:
class MandelbrotRenderer(
private val imageView: ImageView,
override val coroutineContext: CoroutineContext,
private val renderScript: RenderScript = RenderScript.create(imageView.context),
private val script: ScriptC_Mandelbrot = ScriptC_Mandelbrot(renderScript)
) : CoroutineScope {
private val allocationBitmaps = SparseArray()
private var currentJob: Job? = null
private var imageRatio = imageView.width.toDouble() / imageView.height.toDouble()
fun setSize(width: Int, height: Int) {
imageRatio = width.toDouble() / height.toDouble()
var factor = LOW_FACTOR
while (factor >= 1) {
allocationBitmaps.put(
factor,
AllocationBitmap(
renderScript,
Bitmap.createBitmap(
imageView.width / factor,
imageView.height / factor,
Bitmap.Config.ARGB_8888
)
)
)
factor /= 2
}
}
private data class AllocationBitmap constructor(
val allocation: Allocation,
val bitmap: Bitmap
) {
constructor(renderScript: RenderScript, bitmap: Bitmap) : this(
Allocation.createFromBitmap(renderScript, bitmap),
bitmap
)
}
private data class RenderParameters(
val zoom: Double,
val offsetX: Double,
val offsetY: Double,
val factor: Int
)
private var queuedRender: RenderParameters? = null
fun render(zoom: Double = 1.0, offsetX: Double = 0.0, offsetY: Double = 0.0) {
if (currentJob?.isActive == true) {
queuedRender = RenderParameters(zoom, offsetX, offsetY, LOW_FACTOR)
currentJob?.cancel()
}
val renderParameters = RenderParameters(zoom, offsetX, offsetY, LOW_FACTOR)
currentJob = launch {
render(renderParameters)
}.also {
it.invokeOnCompletion { cause ->
if (cause == null) {
passCompleted(renderParameters)
}
}
}
}
private suspend fun render(renderParameters: RenderParameters) {
allocationBitmaps[renderParameters.factor].also { allocationBitmap ->
withContext(Dispatchers.Default) {
val start = System.currentTimeMillis()
script.invoke_mandelbrot(
script,
allocationBitmap.allocation,
ITERATIONS,
imageRatio,
renderParameters.zoom,
renderParameters.offsetX,
renderParameters.offsetY
)
allocationBitmap.allocation.copyTo(allocationBitmap.bitmap)
withContext(Dispatchers.Main) {
imageView.setImageBitmap(allocationBitmap.bitmap)
val elapsed = System.currentTimeMillis() - start
println("Render ${renderParameters.factor} complete in ${elapsed}ms")
}
}
}
}
private fun passCompleted(renderParameters: RenderParameters) {
val newParams = renderParameters.copy(factor = renderParameters.factor / 2)
if (renderParameters.factor > 1) {
currentJob = launch {
render(newParams)
}.also {
it.invokeOnCompletion { cause ->
if (cause == null) {
passCompleted(newParams)
}
}
}
}
}
fun destroy() {
allocationBitmaps.forEach { _, value ->
value.allocation.destroy()
value.bitmap.recycle()
}
script.destroy()
renderScript.destroy()
}
}
private const val LOW_FACTOR = 16
private const val ITERATIONS = 180
The first major change is that MandelbrotRenderer now implements a CoroutineScope as it will provide a coroutine context which allow the the render passes to be performed off of the main thread. The CouroutineContext itself is passed in as a constructor argument.
The setSize() method now creates a Bitmap and Allocation pairs for each of the resolutions: Full size, half size, quarter size, one eighth size, and one sixteenth size.
There is a RenderParameters data class which contains the view port and resolution scale. This will be used to defer and queue render updates when there is already a render pass in progress.
There are two render() methods. The first public is where an external collaborator can request a new progressive render pass with a new view port. It first checks whether there is already a render in progress by checking the status of the currentRender job. If there is one in progress we queue a render in the queuedRender variable and we send a cancel signal to the current job. This won’t actually cancel the job immediately, but it will affect the completion status of the job.
If there is already a render queued, then it will be replaced as the new viewport parameters supersede it. If there is not a render in progress then we launch a new job which calls the private render() method with a RenderParameters object constructed from the method arguments, and a resolution factor of LOW_FACTOR (which is a constant set to 16 for one sixteenth resolution. It also registers a invokeOnCompletion listener which is invoked once the job completes. If the job has been cancelled then the cause will be no-null, so we call passCompleted() (which we’ll look at shortly) if the job was not cancelled.
The private render() method is very similar to the previous public one although it takes a RenderParameters argument rather than individual arguments.
The one thing worth mentioning here is the imageRatio argument which we mentioned in the last article. The reason that this is needed is that we can get some weird rounding behaviour without it. Previously we were calculating this in the Renderscript script based on the width and height of the Allocation. However, I had a particularly annoying bug which it took me a while to track down. When rendering the lower resolution images, they were being rendered in a slightly different view port to the full resolution image despite the view port not changing. It was actually a rounding issue which became more pronounced as we zoomed in further.
Consider an image size of 100 x 90. At half resolution this is 50 x 45, and at quarter resolution it is 25 x 22 (actually it is 22.5 which gets rounded down). If we calculate a floating point ratio from these values we get 1.11111 for both full and half resolution, but 1.13637 for quarter resolution. It is this discrepancy that was causing the issue because this change in the image ratio was affecting how the Renderscript script was calculating the boundaries of the view port that would be rendered. Because of this I was seeing this weird jump of view port during the progressive rendering. The fix was to calculate this up front and pass in a consistent value to the script to ensure that we calculate a consistent view port.
The passCompleted() method will be called whenever a render pass completes without a new render being queued. It invokes another render() with the resolution doubled, and registers a invokeOnCompletion listener exactly the same as before.
Finally there is a destroy() method that will be called to clean everything up when the renderer is no longer required which releases all of the
The other changes to the code are a simplification of ViewportDelegate which is no longer a CoroutineScope, and MandelbrotLifecycleObserver which creates both ViewportDelegate and MandelbrotRenderer which both have different constructors. Both of these changes are fairly minor, so I won’t detail them here. They’re available in the sources.
If we now run this we can see that we now get pretty instant visual feedback for touch gestures, and the app feels very much more responsive as a result. Although the resolution isn’t there when we’re moving around, there is more than enough visual information to enable us to navigate effectively, but once we stop moving ,the image quality quickly improves:
That completes our exploration of the Mandelbrot set. We’ve looked at how we can perform complex computation work on the GPU using Renderscript, how to implement panning and zooming using gestures, and then how to improve responsiveness using progressive rendering.
The source code for this article is available here.
© 2019, Mark Allison. All rights reserved.
Copyright © 2019 Styling Android. All Rights Reserved.
Information about how to reuse or republish this work may be available at http://blog.stylingandroid.com/license-information.
