Color Calibration of Stereo Camera

 

 

Afshin Sepehri

 

Department of Electrical and Computer Engineering

University of Maryland

Email: afshin@eng.umd.edu

 

 

1. Introduction

 

This project presents a color calibration approach based on Least Squares Method. In many of the images processing and computer vision applications, we have more than one camera looking at the same scene from different view angles. One of these applications called Stereopsis [2] is using two cameras viewing a scene similar to human eyes. By using frames taken by these two cameras, we can extract depth information, which is not provided by a single camera. For example, we can extract a disparity map showing depth of different objects in the image. Sample images of a pair of cameras and their corresponding disparity map is shown in Figure 4.

 

Figure 4: Stereopsis and Disparity Map

Top: couple of images taken by two cameras

Bottom: Left: Noisy disparity may. Right: modified Disparity Map

(Adopted from [3])

 As it is shown in [4], by using color images we can gain more accurate disparity map. However since we need cross-correlation of areas in the two images to create disparity map, it is so important to have the two cameras color-calibrated. It means we need them to have same Brightness, Auto Exposure, Sharpness, White Balance, Hue and Saturation.

 

One method of making these parameters equivalent in the two cameras, is assigning same value of these parameters as the internal parameters to the two cameras. However, because of the manufacturing differences of the cameras, we cannot get really equivalent values in practice. Another method is to set internal parameters of the cameras appropriately base on some experiments. [4] To apply this method, we need to do experiences and also have access to complete library of working with internal parameters, which is not provided for me.

 

So we choose another method which is the least square method. In this method, we try to find a filter model for each camera, which change the original image colors (RGB) and make the colors of the camera images. Then, by applying the inverse filter, we can make the two images as close as possible to the original image, we call it canonical format. The steps of our experience are shown in figure 5.

Figure 5: Diagram of the calibration process

 

 

These are the required steps for this calibration process:

 

1. Using a random generator, we create a random pattern of RGB colors as shown in figure 6.

 

 

 

Figure 6: Original Pattern Image

 

2. Print the generated page using a high quality color printer. Note that printer itself can behave as a filter and cause error. However since this error is negligible in comparison to cameras’ filters, we ignore it.

 

3. We take frames of the printed page using the two cameras. We should try to make the view point of the cameras as close as possible to avoid lighting or other problems. Also, we need to find the corresponding points in the two images. This is why we choose the random pattern as figure 6. Any of the pixels in a small box is valid as a representative of the color. However to avoid noise, we apply an averaging filter on a small square around. The images taken by the cameras are in figure 7.

 

 

    

 

Figure 7: Images taken by the two cameras

 

As it can be seen, the colors in the two images are completely different. This is why we need to perform the calibration process.

 

4. We extract the colors from the original image and also the two taken frames. Now there are three matrices of color values. Each one is an N x 3 matrix. Three columns for three values R, G and B. N is the number of colors, which were randomly generated. I created 5 pages and each has 16 x 12 cells. So we have totally

 

            5 x 16 x 12 = 960

 

points. All of these points participate in minimization process.

 

5. Now it is time to consider a model for the filter, which converts the frames to the canonical image. I used two different models based on two different assumptions.

 

 

2. First Model: Independent Components

 

In this model, we assume that three components R, G, B are received through three different sensors and so each one of these components in the canonical image can be found as a function of the corresponding component in the frames. It means:

 

Rc = f(Rf)  ,  Gc = f(Gf)  , Bc = f(Bf)

 

where subscript c is for canonical or calibrated values and subscript f is used for frame values. Since our system is memory-less and each color is determined only by its corresponding value in the frame, we can model it using a polynomial of order n in the following way:

 

            f(x) = b0 + b1x + b2x2 + b3x3 + … + b4xn

 

which can be optimized using least square method. That is minimizing

 

where x2i is the color in the desired canonical image and x1i  is color in the frame. We can write the above equation in matrix format as

 

                                                   (1)

 

where columns of X1 are vectors of different powers of x1 and x2 is vector of x2i’s and b is vector of coefficients.

 

           

                                                       (2)

 

Therefore,

 

                                                                     (3)

 

                                                                (4)

 

So by solving this equation, we can find the coefficients b. However, since the matrix  may be close to singular, we need to use the QR decomposition method. [5] We solve these equations for the two cameras and three colors R, G and B all separately. And we evaluate the error found by modeling with different orders. As tested, a second order model is reasonable for our case. Result of filtering using these second order filters can be seen in figure 8. As seen, now the two images seem fairly similar. But they still differ from the original image of figure 6. So we conclude that this modeling is not appropriate and we need to use a better model as discussed in next section.

 

 

    

 

Figure 8: Images calibrated using one variable polynomial

 

 

2. Second Model: Dependent Components

 

In this model, we assume that the R, G and B colors in the camera images are function of all three components in the original image. This can be found out by looking at the values of the matrices we extract from the images. So we say

 

Rc = f 1(Rf, Gf, Bf)   ,   Gc = f2 (Rf, Gf, Bf)   ,   Bc = f3 (Rf, Gf, Bf)

 

Now we can model functions f1,  f2  and f3 as any combination of the inputs to any arbitrary order. The model I used was:

 

   

 

So we minimize squares of the errors by using equations (1) to (4). The only difference here is how we define matrix . Columns of this matrix correspond to vectors need to be multiplied to b0 to b9.

 

 

By solving equation (4) for first camera, the following b values for R, G and B are found:

 

-155.4757 -137.3428 -231.1179

    7.4198   -3.3051   -1.6277

   -3.8891   11.7290  -12.4294

   -1.9939   -4.0881   15.9975

    0.0591   -0.1489   -0.0889

    0.0363    0.0385    0.0838

   -0.2227   -0.0015   -0.0075

   -0.0509    0.0429    0.0002

    0.0923    0.0798    0.0882

    0.0855   -0.0141   -0.0724

 

and for second camera

 

-150.1773 -199.1211 -468.6306

    7.8881   -4.0535   -1.9667

   -3.1169   13.8981  -16.4781

   -3.7361   -4.0464   22.6625

    0.0816   -0.1309   -0.0775

    0.0254    0.0387    0.0890

   -0.2349    0.0589    0.0943

   -0.0542    0.0371   -0.0064

    0.0822    0.0277    0.0374

    0.1007   -0.0418   -0.1395

 

Values b4 to b9 are so small in comparison to b0 to b3 . So we can use a simpler model which is a linear combination of the values of r, g and b. By applying this model we find coefficients for camera 1 as

 

 

-109.1980  -76.2226 -145.7516

    3.8744   -2.2614   -0.4899

   -2.5049    7.4802   -6.0239

   -0.1491   -2.3869    7.8786

 

and for camera 2

 

-134.6620  -67.7648 -206.8131

    3.9442   -2.2806   -0.6596

   -2.3436    7.7031   -6.3444

   -0.3581   -2.6018    8.6325

 

By applying found b’s, we get the result as figure 9 which seems so similar to original image in figure 5.

 

 

 

    

 

Figure 9: Images calibrated using multi-variable polynomial

 

 

To see the result of our model, we apply it to images taken in figure 10 which are far from an ideal pair of images.

 

 

    

 

Figure 10: Images taken by the two cameras

 

After applying the filter, we get the result of figure 11 which seem ideal images for Stereopsis purposes.

 

 

 

   

 

Figure 11: Images Calibrated using multi-variable polynomials

 

 

 

References

 

[1] Monson H. Hayes, “Statistical Digital Signal Processing and Modeling”, Chapters 3, 4, 5, John Wiley & Sons Inc., 1996.

 

[2] http://www.du.edu/~jcalvert/optics/stereops.htm

 

[3] Fusiello A., Castellani U., Murino V., “Relaxing Symmetric Multiple Windows Stereo Using Markov Random Fields”, EMMCVPR 2001: 91-104

 

[4] Harsh Nanda, Ross Cutler, "Practical Calibrations for a real-time digital omnidirectional camera", Technical Sketches, Computer Vision and Pattern Recognition, Hawaii, US, Dec 2001

 

[5] http://ikpe1101.ikp.kfa-juelich.de/briefbook_data_analysis/node224.html