High Dynamic Range Imaging

By:

Sixue(Mira) Chen

Marissa Garcia

Shota Makino

High Dynamic Range Imaging

By:

Sixue(Mira) Chen

Marissa Garcia

Shota Makino

Online:

< http://cnx.org/content/col11471/1.1/ >

C O N N E X I O N S

Rice University, Houston, Texas

This selection and arrangement of content as a collection is copyrighted by Sixue(Mira) Chen, Marissa Garcia, Shota Makino. It is licensed under the Creative Commons Attribution 3.0 license (http://creativecommons.org/licenses/by/3.0/).

Collection structure revised: December 18, 2012

PDF generated: December 18, 2012

For copyright and attribution information for the modules contained in this collection, see p. 46.

Chapter 1

Motivation1

1.1 Motivation

When you take a picture with a modern digital camera, it can be seen that they cannot yet match the eye's ability to manage contrast. An image sensor or display medium's contrast ratio defines the distance between the darkest black and the lightest white that the device records or displays. The eye's contrast ratio of 1:100,000 is 24 times greater than the 1:4096 of a typical digital camera. As a result, even in correctly exposed photographs, shadows are often too dark and highlights too bright, creating a noticeable loss of detail.

Humans can discern very high range of brightness values. Photography is limited to a much lower dynamic range.

Figure 1.1

Figure 1.2

We can, however, compute a high dynamic range (HDR) picture from multiple exposures. Because we cannot display an HDR picture due to limitation, we will use tone mapping to compress contrast intro a displayable range.

_________

1This content is available online at <http://cnx.org/content/m45370/1.2/>.

Chapter 2

Introduction1

2.1 Introduction

High dynamic range (HDR) digital imaging systems increase visual defility by integrating contrast ratio with observer-based models of color perception across multiple exposures of a single scene. They use mathematical transformations known as tone mapping operators to display HDR images on low dynamic range (LDR) devices such as monitors.

We will uses a series of photographs of a scene to produce an image that is as close as possible to what the human eye would see. To turn a photograph into an HDR image, we first extract the radiance information from the photographs and then tone-map it into a single LDR image. Both the reconstruction and tone mapping algorithms rely on matrix operations so we will implement in MATLAB.

____________

1This content is available online at <http://cnx.org/content/m45336/1.2/>.

Chapter 3

Background1

3.1 Background

HDR increases visual fidelity by integrating contrast ratio with observer-based models of color perception. Relative values of real scene radiances are rarely captured by cameras because of nonlinearities and clipping. The response curve of a camera maps the real scene radiance to the digitized pixel values. HDR methods use response curve to find real scene radiances. Tone mapping filters such images for display on low dynamic range (LDR) displays.

Radiance Mapping

The LDR of a single digital photograph contains accurate radiance information only for correctly exposed pixels. Changing the exposure captures radiance information for different sets of pixels. To accurately analyze a sequence of photographs to extract radiance information for the entire scene, we must take into account the way the camera sensor responds to different levels of light. First, we try and get the camera response curve f, which tells us how scene radiance E is mapped to pixel brightness Z. Using the inverse of f allows us to reproduce actual scene radiance E. The curve f is different for each camera and we compute f from a series of exposures.

Figure 3.2

_________________

1This content is available online at <http://cnx.org/content/m45368/1.2/>.

Chapter 4

Implementation: HDR Method1

4.1 HDR

Figure 4.1: Image Acquisition Pipeline

From Figure 1: Image Acquisition Pipeline, we could see how the scene radiance(X) is nonlinearly mapped into its final digital values(Z). In other words, f(X)=Z, the composition of the characteristic curve of a device and all the nonlinear mapping, is a nonlinear function. Nonlinearity is mostly introduced by later processing steps such as analog to digital conversion and remapping. Therefore, our method to construct a High Dynamic Range(HDR) picture is to first recover the response curve, f(X)=Z and then by using the pixel values from the series of (LDR) pictures, we're able to get the scene radiance, which is like a reversed process. finally, our HDR picture will be constructed from these values of scene radiance.

Our general process will be:

1. We'll take pictures with different exposure times.

a) Knowing the exposure X and the exposure time delta t, we're able to recover irradiance through formula E = X/delta t.

b) Since f(X)=Z and function f could be reasonably conceived as increasing function, its inverse function is well defined. Then, we have

Figure 4.2

c) Using the formulas in a) and b), we have

Figure 4.3

where i is the index of sample (pixel) and j is the index of pictures with different exposure times. The formula also equals to

Figure 4.4

taking natural log on both sides, we will have

Figure 4.5

Making X=g(Z) the inverse function of Z=f(X), we get

Figure 4.6

d) Thus, we transform our problem into a problem minimizing least-squared error between g(Z) and ln(E) and ln(delta t), which could be summarized by the following formula:

Figure 4.7

Note that the second term here is for smoothing g(Z). And lambda is a scaling factor subject to different pictures.

3. After we get the response curve by solving the equation above, when we use the pixel values Z of those LDR pictures, we'll be able to get the values of irradiance X at each point of the scene. In other words, we're trying to figure out in different areas, which group of pixel values from different LDR pictures with different exposure times is closest to the corresponding irradiance and best to reveal details of that area. Thus, it enables us to construct HDR picture in which pixel values in different areas are extracted from the LDR picture that captures its details best.

4. We finish our task to construct a HDR picture here. However, we still have to solve another problem, which is to display a HDR picture on a LDR display. And this leads to the second part of our implementation: Tone-mapping.

_____________

1This content is available online at <http://cnx.org/content/m45500/1.1/>.

Chapter 5

Implementation: Tone Mapping Method1

5.1 Tone Mapping

Tone Mapping is a tool that allows us to map a High Dynamic Range (HDR) image for display on a Low Dynamic Range (LDR) medium, such as an LCD screen. We have implemented Reinhard's algorithms in doing this, which revolve around the idea of scaling the pixel values of the image by a factor to fit in the LDR. In Reinhard's paper, two implementations are found. Let us call these two: Initial Scaling, and Dodge-and-Burning (DAB).

For both implementations, we compress the 3 different channels of Red, Green, and Blue into luminance values by:

We denote the luminance value by Lw and Red, Green, Blue by R, G, B respectively.

As we would like to avoid singularities, we adjust all luminance values for each pixel with a small value, ε. For the Initial Scaling scheme, we first obtain the key value of the scene by computing:

where we sum over all x and y values indicating pixel location. Now, we normalize all pixel values with respect to the middle-grey of the scene. The middle-grey is the subjective middle brightness region of the scene and for images of normal brightness, this middle-grey value typically maps to 0.18, and always lies in between 0 and 1. Once we have normalized, we scale all pixel values by to get a LDR image:

In the DAB scheme, we obtain two Gaussian functions of different sizes located concentric with each other, where the ratio of variances is set to 1.6 (this value can be changed slightly to meet better quality tone mapping). We convolute the image with both Gaussian functions and calculate:

where V1 and V2 are the Gaussian functions of different sizes, φ is a parameter (=8 in our case), a is the subjective middle-grey value of our scene and s is the scaling ratio of the Gaussian functions.

In doing so, we hope to find a small outcome that is less than or equal to a tolerance level that we give (in our case 0.05). If this value meets the tolerance level, we try to enlarge the variance (and by doing so enlarge both gaussians) so that we can compute (1) over again to re-evaluate if this larger area still meets the tolerance. We keep doing this until our Gaussians cover the entire image or we meet the largest area that is less than the tolerance level. We then keep the convoluted values of the image with the smaller Gaussian function and use this for our new LDR image.

What we are essentially doing in this scheme is finding the smallest contrast difference obtainable in the largest area and keeping these values. In effect, we are smoothing the image to fit the LDR yet keeping the edges and details of the image.

In these two ways, we are able to obtain a LDR from a HDR image.

__________

1This content is available online at <http://cnx.org/content/m45499/1.1/>.

Chapter 6

Results1

6.1 Results

In our experiment, we take three groups of LDR pictures and try to construct three HDR pictures from them. For each group of these pictures, we take 6 LDR pictures with different exposure times. We'll display the response curve for each color channel: red, blue and green and our final HDR picture.

6.2 Group 1

Figure 6.1: Response Function for Channel Red

Figure 6.2: Response Function for Channel Green

Figure 6.3: Response Function for Channel Blue

Figure 6.4: LDR picture with exposure time 1/50s

Figure 6.5: LDR picture with exposure time 1/125s

Figure 6.6: LDR picture with exposure time 1/320s

Figure 6.7: LDR picture with exposure time 1/800s

Figure 6.8: LDR picture with exposure time 1/1600s

Figure 6.9: LDR picture with exposure time 1/4000s

Figure 6.10: Final HDR picture for Group 1

Comment: It's impressive to see the details both of the light and of the clothes. For these kind of pictures, it's always easy to lose the details of the light since the pixel values of the area around the light will get saturated. But, through HDR, we could keep the details of the light without losing any other details.

6.3 Group 2

Figure 6.11: Response function for Channel Blue

Figure 6.12: Response function for Channel Green

Figure 6.13: Response function for Channel Red

Figure 6.14: LDR Picture with exposure time 1/50s

Figure 6.15: LDR Picture with exposure time 1/100s

Figure 6.16: LDR Picture with exposure time 1/400s

Figure 6.17: LDR Picture with exposure time 1/1000s

Figure 6.18: LDR Picture with exposure time 1/2000s

Figure 6.19: LDR Picture with exposure time 1/4000s

Figure 6.20: Final HDR picture for Group 2

Comment: Once again, we almost see the details of the lamb. Although it's distorted a bit, we know there's something in it, instead of just completely white for the whole area. What's amazing is that we could clearly see what's going on in the background. All the details of the bags are clearly revealed.

6.4 Group 3

Figure 6.21: Response Function for Channel Blue

Figure 6.22: Response Function for Channel Green

Figure 6.23: Response Function for Channel Red

Figure 6.24: LDR Picture with exposure time 1/160s

Figure 6.25: LDR Picture with exposure time 1/320s

Figure 6.26: LDR Picture with exposure time 1/640s

Figure 6.27: LDR Picture with exposure time 1/1250s 42

Figure 6.28: Final HDR Picture for Group 3

Comment: Once again, we could see a lot of details both in over-exposed area and in under-exposed area. For example, we could see some words on the white container on the right hand side of the lamb. It's impossible for us to see those words when we take a picture with relatively large exposure time. Similarly, we could see the details in the dark area. For example, we could see that there's a purple/blue stuff lying on the black bag. However, it's likely that we'll barely see it when the exposure time of the picture is relatively small.

___________

1This content is available online at <http://cnx.org/content/m45494/1.1/>.

Chapter 7

Conclusion1

7.1 Conclusion

Through this project, we're able to use several Low Dynamic Range(LDR) pictures and their exposure times to create a High Dynamic Range(HDR) picture and display it successfully on LDR display.

7.2 Limitations and Future Work

Our project has three major limitations. First of all, since there are parameters in the HDR and Tone Mapping algorithms such as lambda, the scaling factor of the smoothness term in equation() are subjective to different pictures, so it's hard for us to figure out the parameters. Every time when we input a different series of LDR pictures, we have to readjust the parameters. Therefore, in the future, we hope to come up with an algorithm that could calculate the parameters according to our input pictures.

The second limitation happens when we take a series of pictures with different exposure times, so we have to ensure that everything in the scene has to stay in the same position. Otherwise, the HDR picture will get blurred. It gets harder when we increase the exposure time, since we're exposed more to hand shaking and the changes of environment. For this, we come up with two solutions for this limitation. On one hand, we can integrate the HDR algorithm into hardware. For example, we can build an app in Android so phones are able to take several pictures by themselves and thus our pictures are less vulnerable toward vibration. On the other hand, we can build another algorithm which could pre-process our pictures to ensure that everything stays in the same position. Additionally, for over-exposure pictures, we could increase the ISO (photosensitivity of sensors) to decrease the exposure time.

Our third limitation comes from assumption of our algorithm. When we take those LDR pictures, we assume that the irradiance of each point doesn't change. In other words, since we're doing 1-1 mapping (one exposure time to one irradiance), if we change the lighting of environment, it will mess up with the recovering of device's response curve. Therefore, we're unable to add radiance from camera ash. This will largely reduce HDR's real application. We hope to come up with a method to deal with this case.

Finally, we hope to extend our algorithm so that it will be able to manipulate other camera variables, such as aperture size and ISO as we mentioned above. These algorithms will allow us to have more degree of freedom on our control of pictures.

_____________

1This content is available online at <http://cnx.org/content/m45496/1.1/>.

Chapter 8

301 Poster1

Figure 8.1: High Dynamic Range Imaging

____________

1This content is available online at <http://cnx.org/content/m45503/1.1/>.

ATTRIBUTIONS

Collection: High Dynamic Range Imaging

Edited by: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

URL: http://cnx.org/content/col11471/1.1/

License: http://creativecommons.org/licenses/by/3.0/

Module: "Motivation"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45370/1.2/

Pages: 1-2

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "Introduction"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45336/1.2/

Page: 3

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "Background"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45368/1.2/

Pages: 5-7

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "Implementation: HDR Method"

By: Sixue(Mira) Chen, Marissa Garcia

URL: http://cnx.org/content/m45500/1.1/

Pages: 9-11

Copyright: Sixue(Mira) Chen, Marissa Garcia

License: http://creativecommons.org/licenses/by/3.0/

Module: "Implementation: Tone Mapping Method"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45499/1.1/

Pages: 13-14

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "Results"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45494/1.1/

Pages: 15-42

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "Conclusion"

By: Sixue(Mira) Chen, Marissa Garcia, Shota Makino URL: http://cnx.org/content/m45496/1.1/

Page: 43

Copyright: Sixue(Mira) Chen, Marissa Garcia, Shota Makino

License: http://creativecommons.org/licenses/by/3.0/

Module: "301 Poster"

By: Sixue(Mira) Chen, Marissa Garcia

URL: http://cnx.org/content/m45503/1.1/

Page: 45

Copyright: Sixue(Mira) Chen, Marissa Garcia

License: http://creativecommons.org/licenses/by/3.0/

About Connexions

Since 1999, Connexions has been pioneering a global system where anyone can create course materials and make them fully accessible and easily reusable free of charge. We are a Web-based authoring, teaching and learning environment open to anyone interested in education, including students, teachers, professors and lifelong learners. We connect ideas and facilitate educational communities.

Connexions's modular, interactive courses are in use worldwide by universities, community colleges, K-12 schools, distance learners, and lifelong learners. Connexions materials are in many languages, including English, Spanish, Chinese, Japanese, Italian, Vietnamese, French, Portuguese, and Thai. Connexions is part of an exciting new information distribution system that allows for Print on Demand Books. Connexions has partnered with innovative on-demand publisher QOOP to accelerate the delivery of printed course materials and textbooks into classrooms worldwide at lower prices than traditional academic publishers.