^{1}

^{1}

Light field cameras have a wide area of applications, such as digital refocusing, scene depth information extraction and 3-D image reconstruction. By recording the energy and direction information of light field, they can well solve many technical problems that cannot be done by conventional cameras. An important feature of light field cameras is that a microlens array is inserted between the sensor and main lens, through which a series of sub-aperture images of different perspectives are formed. Based on this feature and the full-focus image acquisition technique, we propose a light-field optical flow calculation algorithm, which involves both the depth estimation and the occlusion detection and guarantees the edge-preserving property. This algorithm consists of three steps: 1) Computing the dense optical flow field among a group of sub-aperture images; 2) Obtaining a robust depth-estimation by initializing the light-filed optical flow using the linear regression approach and detecting occluded areas using the consistency; 3) Computing an improved light-field depth map by using the edge-preserving algorithm to realize interpolation optimization. The reliability and high accuracy of the proposed approach is validated by experimental results.

Light Field cameras, due to their ability of collecting 4D light field information, provide many capabilities that can never be provided by conventional cameras, such as controlling of field depth, refocusing after the fact, and changing of view points [

1. The occlusion detection algorithms involved in these algorithms are sensitive to the surface texture and color of objects in some cases.

2. The accuracy of depth estimation usually depends on the distance between two focal stack images.

To overcome the aforementioned drawbacks, this work proposes an edge- preserving light-field flow estimation algorithm. Firstly, the dense optical flow field of multi-frame sub-apertures of light-field images is computed. Then, all occlusions that may occur during the computation are taken into account. Based on the geometrical features of light-field images and the correspondence among them, a robust light-field flow can be obtained, which contains the accurate depth estimation and occlusion detection. It is worth noting that linear regression is performed on multi-aperture images in our proposed algorithm. As a result, it can well handle two problems that are suffered by conventional optical flow approaches: one is the accuracy-loss during the computation of the optical flow value between two images, and the other is that the occlusion region cannot be detected correctly.

Optical flow estimation, as a hot topic, has been extensively studied [

Recently, some approaches with matching features involved are proposed. Xu et al. merged the estimated flow with matching features at each level of the coarse-to-fine scheme [

In this paper, we propose a novel optical flow calculation algorithm, where the edge preserving technique is adopted. The existing algorithms generally do not consider the edge parts of the target image, while ours does. Note that it is difficult to do the depth estimation and occlusion detection without considering the edge parts. Besides, the conventional algorithms fail to accurately estimates the model of the optical flow field when there are occluded areas during imaging. In addition, the conventional algorithms can hardly guarantee the integrity of the edge parts, which leads to the blurring of the moving boundary and thus affects the estimation of the whole depth map. For example, the results of depth estimation and occlusion detection obtained by the algorithm proposed by Wang et al. [

In this section, we propose a new optical flow detection approach, which aims to obtain the dense and accurate light field flow estimation. Firstly, we get a series of sub-aperture images from the original light-field image. Secondly, we regard the central frame as the reference frame and compute the dense optical flow between it and all the other sub-aperture images. Thirdly, a robust depth estimation is obtained by initializing the light-field flow with linear regression approach. Note that occluded areas are detected with consistency during this process. Finally, an improved light-field depth map is obtained by using the edge-preserving algorithm to realize interpolation optimization. The detailed procedure is shown in

In order to calculate the optical flow, it is necessary to compute the correspondence points between frames. Let

As shown in

array, and the central sub-aperture image is the 25^{th} one. From the sub-aperture image array, we select one row (e.g. the 29^{th} image - 35^{th} image) and one column (e.g. the 5^{th}, 12^{th}, 19^{th}, 26^{th}, 33^{th}, 40^{th}, 47^{th}) for analysis. The horizontal displacements of the pixels between the 25^{th} central sub-aperture image and the horizontal ones in the selected row are estimated and denoted by u1, u2, …, u7. And the vertical displacements of the pixels between the 25^{th} central sub-aperture image and the vertical ones in the selected column are estimated and denoted as v1, v2, …, v7. Theoretically, it should be a linear relationship between any two horizontal or any two vertical displacements when the pixel movement is stable and not occluded. Under this assumption, we try to calculate these two linear relationships. We firstly calculate the average horizontal and vertical displacements

where n is the number of sub-aperture images in this row and column. Then, we define the deviation between the measured and the calculated displacements as follows:

When

By reconstructing the optical flow estimation

, (3)

where α is a depth factor of the refocus plane, generally ranging in [0.2, 2], and and denote the displacement of the counterpart of the coordinates

During the calculation of optical flow, we can use the consistency of forward and backward flows to detect and remove the occlusions between every two sub- aperture images. Specifically, a comparison is made between the forward flow

To further improve the performance, we calculate the forward and backward flows between multiple sub-aperture images, in a way that is similar to the calculation of horizontal and vertical displacements of optical flow, and then compare their consistency. The pixels that are inconsistent are regarded as occluded and should be removed. The occlusion detection results of Wang’s and ours are shown in

In the aforementioned optical flow calculation and occlusion detection, the edge information is neglected as shown in

Therefore, we further use the Epicflow algorithm proposed by Revaud et al [

The most recently proposed Fullflow method [

In this section, a set of experiments is carried out to validate the superiority of our proposed approach over the state-of-the-art methods. Note that light field images with the resolution of 398 × 398 are chosen for test.

Firstly, totally 7 × 7 sub-aperture images are extracted from a raw light field image. Then, on the basis of conventional Lucas-Kanade algorithm, we calculate the set of corresponding points between the central sub-aperture image and all the other ones. Based on the obtained set of corresponding points, the optical flow estimation results are obtained in the horizontal direction and the vertical direction. As a result, the stable and homogeneous optical flow of the light field image can be calculated. Note that the effects of the noise, varying light intensity and occlusions are minimized during the calculation of the corresponding points of the target area. Finally, we optimize depth estimation of the light field image by the EpicFlow approach.

As shown in

the edge parts are properly processed, resulting in a better depth map estimation. Compared with some other mainstream optical flow algorithms, our approach has better performance.

Zhang, W. and Lin, L.L. (2017) Light Field Flow Estimation Based on Occlusion Detection. Journal of Computer and Communications, 5, 1-9. https://doi.org/10.4236/jcc.2017.53001