Dynamic Image Resolution Made Easy

Posted by Evan

When Corona SDK was first released to the public in December 2009, there was only one device screen to target: the iPhone at 320 x 480 resolution. Corona for Symbian had been set aside due to the iPhone’s overwhelming popularity, and Android wasn’t the “next big thing” yet.

In 2011, mobile developers need to consider a rapidly growing set of screen sizes, shapes, resolutions and pixel densities. Corona now addresses this in two ways.

(1) Content scaling automatically expands (or shrinks) the entire Corona stage so that your content fills different screens. This topic won’t be covered here, but you can read a full discussion of content scaling in this earlier post.

(2) In addition, you can employ dynamic image resolution, which will automatically replace your images with higher-resolution counterparts when your app runs on a larger screen. This article will provide an introduction to how this works.

What is it?

Dynamic image resolution operates in conjunction with content scaling. Using filename patterns that you specify, you can bundle larger alternate image files with your application. When Corona scales your content above certain size thresholds, the larger images will be automatically loaded in and scaled to the same screen area that the base image would have occupied, resulting in richer, more detailed graphics.

You can see a working example in the “Hardware/DynamicImageResolution” sample project on the Corona DMG (the sample can also be downloaded here).

If you run this project in the Corona Simulator, and then switch device previews between different phones and tablets, you’ll notice that alternative versions of my random vacation snapshot are automatically loaded on devices with higher-resolution screens.

In this diagram, you can see that the image always occupies the same dimensions within the (upscaled) base content area, which is marked here with a dotted white outline:

(If the above seems slightly confusing, check out my earlier post on content scaling, from which this diagram is adapted.)

In this example, the base content size has been declared as 320 x 480, so the older iPhone displays the original image (a 200 x 200 file), the iPad displays the double-sized image (a 400 x 400 file), and the Droid shows the medium-sized “1.5x” scaled image (a 300 x 300 file) — and all of this happens automatically.

Is this similar to Apple’s “@2x” solution?

With the launch of the iPhone 4, which features a double-resolution 640 x 960 “Retina Display”, Apple introduced a system for automatic high-res asset replacement. Given an image in the application bundle named (for example) foo.png, if a double-sized image named foo@2x.png is also present, the latter image will be loaded automatically whenever the former is called from Objective-C on the iPhone 4.

The important difference is that Apple’s system only handles the specific case where a screen is exactly twice the resolution of the original content size. In other words, the Apple “@2x” method is effectively a subset of Corona’s dynamic image resolution.

With Corona, you are free to adopt Apple’s naming scheme and 1x/2x scaling categories, but you can also use any other asset scaling factor (and file naming scheme) you like. This general solution allows you to also handle Android devices, the iPad — and future mobile screens we don’t know about yet.

A second advantage is that Corona does not require you to know the ultimate resolution of the target device — which may be unknown, especially for the growing range of current and future Android devices. Instead, you can define several resolution thresholds, and Corona will automatically use the best match when it runs on each device.

How does it work?

As noted above, dynamic image resolution works in conjunction with dynamic content scaling (documented in the “Configuring Projects” section of the documentation). For further reference, the dynamic image resolution docs arehere.

To use this feature, you basically need to do two things:

• Use display.newImageRect() rather than display.newImage() when loading your images
• Specify one or more scaling thresholds in your project’s config.lua file

The syntax is as follows:

display.newImageRect( [parentGroup,] filename [, baseDirectory] imageWidth, imageHeight )

• imageWidth is the base image’s width in the base dimensions of the content.
• imageHeight is the base image’s height in the base dimensions of the content.
• parentGroup and baseDirectory are optional, and can be omitted; they behave in the same way as their counterparts in display.newImage().

An actual example might look like this:

…and then your config.lua file might look like this:

Assuming your base image was 100 x 100 pixels, you would then provide the following three images:

baseImage.png (100 x 100 pixels)
baseImage-15.png (150 x 150 pixels)
baseImage-2.png (200 x 200 pixels)

Again, your naming scheme can be anything you like:

The important thing is the numerical factors listed alongside each image suffix (1.6, 2 and 3 in the above example). These values determine the scaling thresholds at which Corona will automatically load the alternative images. Of course, it’s unlikely that you’ll need to use values above 2 on today’s mobile devices, but it’s a safe bet that screens will continue to increase in resolution.

The final thing to understand is that the alternative image files are always optional. Whenever you use display.newImageRect(), Corona will automatically drop back to loading the base image if no higher-resolution alternative files are available. Specifically, it will attempt to load the best choice, then fall back to the next best (if applicable), eventually ending up at the base image if all else fails. This means that the base image file must always be present, but the other resolutions need not be — you can decide on a case by case basis whether you want to provide alternatives.

(Why does Corona make you provide the height and width manually in display.newImageRect()? Because doing it automatically would require first loading the base image to find out its size, resulting in two image-loading operations rather than one. Since image loading is a very slow pipeline on mobile devices, it’s better to avoid the overhead by having you specify the known size and avoid the extra loading.)

That’s a lot of choices — what’s the “magic recipe”?

At this point, the simplest (and most common) config.lua recipe simply replicates Apple’s Retina Display solution:

In this case, you would then provide two versions of each image:

baseImage.png
baseImage@2x.png

This will automatically swap in the double-resolution images on iPhone 4 and iPad, and use the base image on all other devices.

But wait! In the year since the iPad first appeared, there has been an explosion of 7″ Android tablets, including the Samsung Galaxy Tab, the Barnes and Noble Nook, and a flurry of newly-announced tablets at the recent Consumer Electronics Show.

Many of these devices feature a resolution of 600 x 1024. You’ll note that a width of 600 pixels is almost, but not quite, twice the classic iPhone width of 320 pixels. Therefore, if your content assumes a base size of 320 x 480 (which is still fairly common), the 2x scaling threshold won’t quite kick in on 7″ tablets, and you’ll end up displaying the lower-resolution assets. This seems suboptimal; you’ll more likely want to target high-res assets at all tablets, not just the iPad.

One solution, of course, is to implement three-tier resolution (1x, 1.5x and 2x assets) rather than merely two-tier. The “Hardware/DynamicImageResolution” sample mentioned above is an example of this technique.

But there is an easier way! Rather than creating three sets of images, you can maintain just two sets (normal and double resolution) and then force the higher-resolution images to show up on the 7″ tablets.

To do this, simply tweak the scaling threshold in the config.lua file shown above:

This is probably the closest thing right now to a “magic recipe” for dynamic image resolution. With this setup, you can create the same set of single- and double-resolution images as in the previous example, while still having the better versions loading automatically on the slightly-less-than-double-size Android tablets.

(Since the base 320 x 480 stage size given above will be upscaled by at least a factor of 1.875 on a 7″ tablet, the specified threshold of “1.8″ rather than “2″ should be triggered on all iOS and Android tablets currently available.)

The secret is that the actual image size of the file loaded with display.newImageRect() is literally irrelevant. Since you are specifying the visible target size manually, Corona can pour any arbitrary bitmap into that space, and the job of config.lua is simply to give it a guide for choosing the best image on each screen size.

Note that because Corona uses hardware scaling in OpenGL, upscaled or downscaled bitmaps will be automatically smoothed — it’s not like the old days of the web, where we all learned to avoid scaling images on the fly due to ugly pixelation in browsers.

That’s cool, but why not do this automatically for all images?

As I discussed in the Flash porting guide, Corona’s high performance is derived from hardware acceleration, which in turn relies on a texture memory buffer on the device. Although recent devices like the iPhone 4 and iPad have doubled this available memory (from around 10MB to 20MB), doubling the resolution of an image actually quadruples the memory used. Therefore, you probably shouldn’t just double all your image resolutions, since you will actually run out of memory much faster on high-res devices.

For this reason, the best practice is to be selective about your high-resolution images. For main character sprites or crisp hard-edged foreground objects (say, platform game elements), selective high resolutions may be worthwhile. On the other hand, a background texture depicting grass, dirt or fluffy clouds may not benefit much from higher dpi, and should therefore be allowed to scale up from a lower-resolution file.

What about sprite sheets?

The display.newImageRect() API is separate from sprite sheets, and does not affect them. This is because sprite sheets are a special case: they’re a method for conserving texture memory by allocating large shared images at the maximum possible size available on the hardware. For most mobile hardware, the largest image size is 1024 x 1024 (or 2048 x 2048 on the iPhone 4 and iPad). A sprite sheet cannot simply be doubled in size, and generally it can’t be increased in resolution at all, without exceeding the hardware limitations on many devices.

Also, due to the cruel mathematics noted above (2x resolution equals 4x memory used), a game that uses sprite sheets to push the graphics memory envelope on a low-end iPhone can’t be globally asset-doubled on the iPhone 4. Finally, Android devices have no reliable correlation between screen size and available texture memory, since these details are freely chosen by individual hardware manufacturers. There is no way to abstract away all of these hardware considerations without severely limiting you at the high end.

Therefore, the best practice for multi-resolution output with sprite sheets is to create alternative sets of high-resolution sprite sheets and data, which will likely mean more sheets for the higher-resolution sprites, with fewer animation frames fitting on each sheet. Then, use platform detection and scaling detection to tell when you’re running on an iPhone 4 and/or iPad, and load the alternative sprite sheets in those specific cases only.

Eventually, we’ll live in a world where graphics memory is essentially unlimited, but for now, these hardware details remain critical.