Cascaded Scene Flow Prediction using Semantic Segmentation
Zhile Ren Deqing Sun Jan Kautz Erik B. Sudderth
cs.CV







Cascaded Scene Flow Prediction using Semantic Segmentation
    Zhile Ren

Brown University

ren@cs.brown.edu
Deqing Sun

NVIDIA

deqings@nvidia.com 
Jan Kautz

NVIDIA

jkautz@nvidia.com 
Erik B. Sudderth

UC Irvine

sudderth@uci.edu

    
==============================================================================================================================================================================







Given two consecutive frames from a pair of stereo cameras, 3D scene flow methods simultaneously estimate the 3D geometry and motion of the observed scene. 
Many existing approaches use superpixels for regularization, but may predict inconsistent shapes and motions inside rigidly moving objects.
We instead assume that scenes consist of foreground objects rigidly moving in front of a static background, and use semantic cues to produce pixel-accurate scene flow estimates.
Our cascaded classification framework accurately models 3D scenes by iteratively refining semantic segmentation masks, stereo correspondences, 3D rigid motion estimates, and optical flow fields.
We evaluate our method on the challenging KITTI autonomous driving benchmark, and show that accounting for the motion of segmented vehicles leads to state-of-the-art performance.




§ INTRODUCTION

The scene flow <cit.> is the dense 3D geometry and motion of a dynamic scene.
Given images captured by calibrated cameras at two (or more) frames, a 3D motion field can be recovered by projecting 2D motion (optical flow) estimates onto a depth map inferred via binocular stereo matching.
Scene flow algorithms have many applications, ranging from driver assistance <cit.> to 3D motion capture <cit.>.

The problems of optical flow estimation <cit.> and binocular stereo reconstruction <cit.> have been widely studied in isolation. Recent scene flow methods <cit.> leverage 3D geometric cues to improve stereo and flow estimates, as evaluated on road scenes from the challenging KITTI scene flow benchmark <cit.>.
State-of-the-art scene flow algorithms <cit.> assume superpixels are approximately planar and undergo rigid 3D motion.  Conditional random fields then provide temporal and spatial regularization for 3D motion estimates.
Those methods generally perform well on background regions of the scene,
but are significantly less accurate for moving foreground objects.


Estimating the geometry of rapidly moving foreground objects is difficult, especially near motion boundaries.  Vehicles are particularly challenging because painted surfaces have little texture, windshields are transparent, and reflections violate the brightness constancy assumptions underlying stereo and flow likelihoods.
However, accurate estimation of vehicle geometry and motion is critical for autonomous driving applications.
To improve accuracy, it is natural to design models that separately model the motion of objects and background regions <cit.>.


Several recent methods for the estimation of optical flow <cit.> have used semantic cues to improve accuracy.
While motion segmentation using purely bottom-up cues is challenging,
recent advances in semantic segmentation <cit.> make it possible to accurately segment traffic scenes given a single RGB image.
Given segmented object boundaries, object-specific 3D motion models may then be used to increase the accuracy of optical flow methods.

In this paper, we use instance-level semantic segmentations <cit.> and piecewise-rigid scene flow estimates <cit.> as inputs, and integrate them via a cascade of conditional random fields (CRFs) <cit.>.
We define pixel-level CRFs relating dense segmentation masks, stereo depth maps, optical flow fields, and rigid 3D motion estimates for foreground objects.
Due to the high dimensionality of these variables, we refine them iteratively using a cascaded classification model <cit.>, where each stage of the cascade is tuned via structural SVM learning algorithms <cit.>.
We evaluate using previous scene flow annotations <cit.> of the challenging KITTI autonomous driving benchmark <cit.>, and improve on the state-of-the-art in two-frame scene flow estimation.
Our work demonstrates the importance of semantic cues in the recovery of the geometry and motion of 3D scenes.






§ RELATED METHODS FOR SCENE FLOW ESTIMATION

Vedula  <cit.> first defined the scene flow as the dense 3D motion of all points in an observed scene, and recovered voxel-based flow estimates using 2D optical flow fields from several calibrated cameras. 
Huguet and Devernay <cit.> then proposed
a variational approach and jointly solved for stereo and optical flow, while  
Wedel  <cit.> decoupled the stereo and flow problems for
efficiency. These classic algorithms only improve marginally over modern,
state-of-the-art stereo and optical flow methods.


Although scene flow algorithms require more input images than standard 
optical flow or stereo reconstruction methods, the task is still challenging due to the high dimensionality of the output disparity and motion fields.
To reduce the solution space,
Vogel  <cit.> introduced a piecewise rigid scene
flow (PRSF) model and used superpixels to constrain scene flow estimation. 
For the first time, they showed that scene flow methods could outperform stereo and optical flow methods by a large margin on the challenging KITTI dataset <cit.>. 
In follow-up work <cit.> they extended their formulation to multiple frames and improved accuracy.  However, because the PRSF model relies on bottom-up cues
for superpixel segmentation, it tends to over-segment foreground objects such as cars.
Over-segmented parts are allocated independent motion models, so global
information cannot be effectively shared.


Inspired by the success of Vogel  <cit.>,
Menze and Geiger <cit.> annotated a new KITTI dataset with dynamic foreground objects for scene flow evaluation. 
They proposed an object scene flow (OSF) algorithm that segments the scene into independently moving regions, and encourages the superpixels within each region 
to have similar 3D motion. 
Although the performance of OSF improved on baselines, the “objects” in their model are assumed to be planar and initialized via bottom-up motion estimation, so physical objects are often over-segmented.  
The inference time required for the OSF method is also significantly longer than most competing methods.


The successes of convolutional neural networks (CNNs) for high-level vision tasks has motivated CNN-based regression methods for low-level vision.
Dosovitskiy  <cit.> introduced a denoising
autoencoder network, called FlowNet, for estimating optical flow.
Mayer  <cit.> extended
the FlowNet to disparity and scene flow estimation with a large synthetic dataset.
While CNN models generate scene flow predictions rapidly, networks trained on synthetic data are not competitive with state-of-the-art methods on the real-world KITTI scene flow benchmark <cit.>.


Some related work integrates automatic motion segmentation with optical flow prediction <cit.>,
but assumes large differences between the motion of objects and cameras, and
requires multiple input frames. Exploiting the recent success of CNNs for
semantic segmentation <cit.>,
semantic cues have been shown to improve optical flow estimation <cit.>. Concurrent work <cit.> also shows that semantic cues can improve scene flow estimation.
In this paper, we propose a coherent model of semantic segmentation, scene geometry, and object motion.  We use a cascaded prediction framework <cit.> to efficiently solve this high-dimensional inference task.
We evaluate our algorithm on the challenging KITTI dataset <cit.>
and show that using semantic cues leads to state-of-the-art scene flow estimates.





§ MODELING SEMANTIC SCENE FLOW


Given two consecutive frames I, J and their corresponding stereo pairs I', J',
our goal is to estimate the segmentation mask, stereo disparity, and optical flow
for each pixel in the reference frame (Fig. <ref>).  Let
p_i = (d_i^(1), s_i^(1), m_i, f_i) denote the variables associated with pixel i in the reference frame, where 
d_i^(1)∈ℝ^+ is its disparity, 
s_i^(1)∈{0, 1, …} is a semantic label (0 is background, positive integers are foreground object instances),
m_i ∈ SE(3) is its 3D rigid motion (translation and rotation), and 
f_i=[u_i, v_i] is its optical flow.  
We denote the disparity and semantic segmentation for each pixel in the second frame by
q_i=(d_i^(2), s_i^(2)).  We only use two frames to estimate scene flow, and thus need not explicitly model motion in the second frame.

Existing scene flow algorithms make predictions at the superpixel level
without explicitly modeling the semantic content of the scene <cit.>. 
Predictions inside each semantic object may thus be noisy or inconsistent.
In this work, we assume that the scene contains foreground objects (vehicles, for our autonomous driving application) rigidly moving across a static background.
Given an accurate semantic segmentation of some foreground object,
the geometry of the pixels within that segment should be spatially and temporally consistent, and the optical flow should be consistent with the underlying 3D rigid motion.

Due to the high dimensionality of the scene flow problem, we refine our estimates using a cascade of discriminative models <cit.>, with parameters learned via a structural SVM <cit.>.  Every stage of the cascade makes a targeted improvement to one scene variable, implicitly accounting for uncertainty in the current estimates of other scene variables.
We initialize our semantic segmentation 𝒮 using an instance-level segmentation algorithm <cit.>, and our disparities 𝒟 and optical flow fields ℱ using the PRSF method <cit.>.
We discuss their cascaded refinement next.




 §.§ Refinement of Semantic Segmentation
 
The initial single-frame segmentation is unreliable in regions with shadows and reflections. Given stereo inputs, however, our depth estimates provide a strong cue to improve the segmentation.
Therefore for each segmentation instance, we define a CRF on the pixels in its enclosing bounding box B_i.
We seek to estimate the foreground segmentation s given an initial noisy segmentation
ŝ.

Our data term encourages the inferred segmentation s to be close to the initial segmentation ŝ.
The KITTI scene flow dataset <cit.> generates “ground truth” segmentations by aligning approximate CAD models, and these annotations are often inaccurate at object boundaries, thus violating that assumption that foreground and background objects typically have distinct color and geometry.  To add robustness,
we define a feature by computing the signed distance of pixel i to the 
original segmentation border and using a sigmoid function to map these distances to [0, 1], denoted by ϕ_dist(i, ŝ).
The data energy for our CRF model is then

    E_seg^data(𝒮)    =∑_i∈ B_i[λ_1 +λ_2 ϕ_dist(i, ŝ)]δ(s_i=0, ŝ_i=1)
          + [λ_3 +λ_4 ϕ_dist(i, ŝ)]δ(s_i=1,ŝ_i=0).


We demonstrate the benefits of our signed distance feature ϕ_dist(i, ŝ) in Fig. <ref>.
By allowing the CRF to reduce confidence in ŝ near boundaries, this feature allows other image-based cues to improve segmentation accuracy.




To allow spatial regularization, we add edges ℰ to our CRF connecting each pixel to its 8 spatial neighbors:

    E_seg^space(𝒮)    = ∑_(i,j) ∈ℰ[λ_5 + λ_6 ρ_img(I_i, I_j)  
       + λ_7 ρ_disp(d_i, d_j)]δ(s_i ≠ s_j).

Here, ρ_img(I_i, I_j) = exp{-||I_i - I_j||/σ_img} measures RGB color similarity,
and ρ_disp(d_i, d_j)=exp{-|d_i - d_j|/σ_disp} measures similarity of the current (approximate) disparity estimates.

To learn the parameters λ = [λ_1, …, λ_7], we use a structured SVM <cit.>
with loss equal to the average label error within bounding box B_i <cit.>.
Feature bandwidths σ_img,σ_disp are tuned using validation data.
To perform inference on E_seg^data + E_seg^space,
we use an efficient implementation of
tree-reweighted belief propagation <cit.>.
Because pixel labels are binary, inference takes less than 0.5s.
To apply our CRF model to the scene flow problem, we independently estimate the segmentation for each instance and frame.




 §.§ Estimation of Scene Geometry


Given a disparity map 𝒟 and camera calibration parameters, a 3D point cloud representation of the scene may be constructed.  Standard stereo estimation algorithms ignore semantic cues, and often perform poorly on surfaces that are shadowed, reflective, or transparent. As illustrated in Fig. <ref>, for autonomous driving applications the depth estimates for vehicle windshields are especially poor.
Because inaccurate depth estimates lead to poor motion and flow estimates, we design a model that enforces local smoothness of depths within inferred segmentation masks.

We define a CRF model of the pixels within each semantic segment previously inferred by our cascaded model.
For each pixel i in the left camera with disparity hypothesis d_i, we denote its corresponding pixel in
the right camera as P_d(i, d_i). The data term is defined to
penalize the difference in smooth census transform between pixel i and P_d(i, d_i):

    E_geom^data(𝒟) = ∑_{i|s_i=s}ρ_CSAD(I_i, I'_P_d(i, d_i)).

Here, ρ_CSAD(.,.) is the CSAD cost <cit.>
for matched pixels in different images. The CSAD difference is a
convex approximation of the census transform <cit.> that gives reliable pixel correspondences for many datasets <cit.>.

We encourage piecewise-smooth depth maps by penalizing the absolute difference of neighboring pixel depths: 

    E_geom^space(𝒟) = τ_1
    ∑_(i, j) ∈ℰ_sρ_depth(d_i, d_j).

Here ℰ_s contains neighboring pixels within segment s, ρ_depth(d_i, d_j) = |C/d_i - C/d_j|, and C is a
camera-specific constant that transforms disparity d into depth C/d. We enforce consistency of pixel depths because the scale of disparities varies widely with the distance of objects from the camera.

If naively applied to the full image, simple CRF models are often inaccurate at object boundaries <cit.>. However as illustrated in Fig. <ref>, although our stereo CRF uses standard features,  it is effective at resolving uncertainties in challenging regions of foreground objects and it is much better able to capture depth variations within a single object.
Moreover, because our pairwise distances depend only on the absolute value of depth differences, 
distance transforms <cit.> may be used for efficient inference in minimizing E_geom^data + E_geom^space.
On average, it takes less than 5s to perform inference in a 200×200 region
with 200 disparity candidates.  We refine the disparities for each frame independently.






 §.§ Estimation of 3D Motion

If the segmentation mask and disparity estimates for each object instance were perfect, we could apply 3D rigid motion to the 3D point cloud for each segment, and project back to the image plane to recover the 2D optical flow. We let (x_i,y_i) denote the motion flow constructed in this way.
Although our imperfect geometry estimates will cause the motion flow to differ from the true optical flow (u_i,v_i),
each still provides valuable cues for the estimation of the other.

For each detected segment, we let M=(R,t) denote its 3D relative motion between the first and second frames.  The motion M has 6 degress of freedom: t is a translation vector, and R = (α,β,γ) is a rotation represented by three axis-aligned rotation angles. 
We match the rigid motion M to the current flow field estimate (u,v) by minimizing the following energy function: 

    E_motion(M) = ν (ρ(α) + ρ(β) + ρ(γ))
    
        + ∑_{i|s_i=s} |x_i(M,d_i) - u_i| + |y_i(M,d_i) - v_i|.

where (x_i(M,d_i),y_i(M,d_i)) is the motion flow computed from disparity d_i, 3D motion M, and the camera calibration.
We let ρ(a)=√(a^2 + ϵ^2) be the Charbonnier penalty, a smooth function similar to the L_1 penalty that provides effective regularization for motion estimation tasks <cit.>. We regularize R to avoid unrealistically large rotation estimates.
We set the regularization constant ν using validation data, and use gradient descent to find the optimal value for M.  We visualize an example motion flow map in 
Fig. <ref>.







 §.§ Estimation of 2D Optical Flow

The estimated motion flow from the previous stage provides valuable
cues for optical flow estimation. As in the example in Fig. <ref>,
motion flow errors are primarily caused by imperfect geometries (or disparities). 
We thus seek a flow field f_i=(u_i, v_i) such that the corresponding pixel P_f(i, f_i) in the next frame matches pixel i, and f_i does not deviate too much from (x_i, y_i).
We define a CRF model of the pixels within segment s in frame 1, with likelihood

    E_flow^data(ℱ)     = ∑_{i|s_i=s}ρ_CSAD(I_i, J_P_f(i, f_i)) 
       + η_1 (|u_i - x_i| + |v_i - y_i|).

We also encourage spatially smooth flow field estimates:

    E_flow^space(ℱ)     = ∑_(i, j)∈ℰ_sη_2 (|u_i - u_j| + |v_i - v_j|).


While many optical flow methods use superpixel approximations to make inference more efficient <cit.>,
max-product belief propagation can be efficiently implemented for our pixel-level CRF using distance transforms <cit.>.
As shown in Fig. <ref>, our refined optical flow improves the initial flow by smoothly varying across the segment, while simultaneously capturing details that are missed by the motion flow.

To limit the memory consumption of our optical flow algorithm, we perform inference on a down-sampled image and then use the EpicFlow <cit.> algorithm to interpolate back to the full image resolution.  Other recent optical flow algorithms have used a similar approximation <cit.>.






  
Motion Estimation for Out-of-Frame Pixels 


We notice that the EpicFlow interpolation tends to produce significant errors for
pixels that move outside of the image border.  Outside of the camera's field of view, optical flow can only be predicted using the known 3D rigid motion, and we thus propose a flow fusion CRF <cit.> to combine the estimated optical flow and motion flow for partially occluded objects.

In particular, we use a binary CRF to determine whether the optical flow (u_i, v_i) or motion flow (x_i, y_i) provides a better estimate of the true flow (U_i,V_i) for each pixel i.
Intuitively, for within-border pixels we should use the matching cost to compare flow fields, while out-of-border pixels should be biased towards the motion flow interpolation:

    E_fuse^data(ℱ) = 
      ω_1 (|U_i - x_i| + |V_i - y_i|)δ[P_f(i, f_i)  outside]
    
      +∑_f={(u, v), (x, y)}∑_{i|s_i=s}ρ_CSAD(I_i, J_P_f(i, f_i))δ[P_f(i, f_i)  inside].


Spatial smoothness is encouraged for neighboring pixels:

    E_fuse^space(ℱ)  = ∑_(i, j)∈ℰω_2 (|U_i - U_j| + |V_i - V_j|).

We tune parameters ω_1, ω_2 using validation data, and minimize the energy using
tree-reweighted belief propagation <cit.>. We show in Fig. <ref>
that the fused flow estimate retains many details of the optical flow, while using the motion flow to better interpolate in occluded regions.
We also apply our flow fusion technique to update the noisy background flow predictions.  See Fig. <ref> for additional examples of our final optical flow estimates.









§ CASCADED SCENE FLOW PREDICTION

The CRF models defined in Sec. <ref> refine the various components of our scene model greedily, by estimating each one given the current best estimates for all others.
However, this approach does not fully utilize the temporal relationships between the segmentation and geometry at different frames.  
Also, when the initial optical flow contains major errors, our motion flow estimates will be inaccurate.
To better capture the full set of geometric and temporal relationships, we thus use multiple stages of cascaded prediction <cit.> to further refine our scene flow estimates.
The inputs and outputs for each stage of our cascade are summarized by the directed graph in Fig. <ref>.







  
Temporal Segmentation Consistency
Rather than segmenting each video frame independently, in the second stage of our cascade, we use the inferred flow field f to encourage temporal consistency.
Each pixel i in frame 1 is linked to matched pixel P_f(i, f_i) in frame 2:

    E_seg^time(𝒮) =
      λ_8 δ(s^(1)_i = 0, s^(2)_P_f(i, f_i) = 1)
    + λ_9 δ(s^(1)_i = 1, s^(2)_P_f(i, f_i) = 0)
     
      + ∑_i[λ_10 + λ_11ρ_CSAD(I_i, J_P_f(i, f_i))]δ(s^(1)_i = s^(2)_P_f(i, f_i)).

We again use S-SVM learning of CRF parameters λ on E_seg^data+E_seg^space+E_seg^time, and infer segmentations using tree-reweighted belief propagation.




  
Temporal Geometric Consistency


As in our temporal segmentation model, we also extend the stereo CRF of Sec. <ref> to encourage smooth changes for the depths of pixels linked by our optical flow estimates:

    E_geom^time(𝒟) = τ_2 ∑_{i|s_i^(1)=s}ρ_depth(d_i(m_i), d_P_f(i, f_i)).

Here, d_i(m_i) denotes the disparity value of pixel i in the second frame when rigid motion m_i is applied.  The parameters τ are learned using validation data, and efficient distance transformation <cit.> is also used to solve E_geom^data+E_geom^space+E_geom^time.
Fig. <ref> shows an example of the improved disparity and flow estimates produced across multiple stages of our cascade.





  
Recovery from a Poor Optical Flow Initialization


If the initial noisy optical flow is very inaccurate, our cascade cannot recover the correct 3D motions of objects because we assume motion flow should match optical flow. 
Since our updated semantic segmentation masks s^(1) and s^(2) are typically very accurate,
when applying rigid motion M to pixels in s^(1), 
the shape of the new segmentation mask s(M) should be similar to s^(2).
We measure this similarity via a cost defined on the second-frame bounding box B:

    1/|B|∑_i∈ Bα S(M)_i · C(S^(2)_i) + (1-α) C(S(M)_i) · S^(2)_i.

Here, C(·) is the Chamfer difference map and α=0.5. 
This cost function is widely 
used for human pose estimation <cit.>.
If this cost exceeds 0.5, we replace the first
term in Eq. (<ref>) with this silhouette cost. By optimizing this modified objective in Eq. (<ref>) using standard gradient-descent, we can recover from bad motion estimates. An illustration is in the supplementary material.



  
Second Frame Disparities 


For the KITTI scene flow dataset <cit.>, the ground truth 
disparity for the second frame is represented as per-pixel disparity changes with respect to the first frame. 
To predict this quantity for evaluation, we apply our estimated 3D rigid motion for each pixel to its estimated geometry in the first frame.  
The accuracy of these disparity estimates is thus strongly dependent on the performance of our motion estimation algorithm.




  
Global Energy Function


The global energy function implicitly minimized by our cascade of CRFs can be constructed by adding all energy terms together. Our iterative optimization of subsets of variables (as in Fig. <ref>) can be seen as block coordinate descent, where the cascaded prediction framework refines the energy function to reflect the typical accuracy of previous stages.  This cascaded framework enables efficient, adaptive discretization of a large state space for flow and disparity, and is a principled way of optimizing a limited number of inference iterations <cit.>.




§ EXPERIMENTS




 


We test our semantic scene flow algorithm (SSF) with 3 iterations of cascaded prediction
on the challenging KITTI 2015 benchmark <cit.>. We evaluate the performance of our
disparity estimates for two frames (D1, D2), flow estimates (Fl)
for the reference frame, and scene flow estimates (SF)
for foreground pixels (fg),
background pixels (bg), and all pixels (all).
See Table <ref> for experimental
results on all pixels, and Table <ref> for non-occluded pixels.
We evaluate SSF cascades learned to refine PRSF <cit.> initializations (SSF-P) and also apply the learned parameters to OSF <cit.>
initializations (SSF-O). Our cascaded approach is superior to the published two-frame scene flow algorithms
with respect to all evaluation metrics. SSF-P is about 60% more accurate than the two-frame PRSF method; SSF-P is overall 2% more accurate than OSF, while 10 times faster. At the time of submission, the only published work that performed better than our SSF approach were the multi-frame PRSM <cit.> and OSFTC <cit.> methods, which require additional data. The concurrently developed ISF method <cit.> uses external training data for instance segmentation and disparity estimation, leading to further improvements over our approach at the cost of slower speed. 

We visualize the qualitative performance of our SSF-P method on training data in Fig. <ref>. 
In Table <ref>, we evaluate the performance gain provided by each
stage of the cascade on the training set. There is an improvement at the first stage 
of the cascade when modeling segmentation and geometry independently at each frame, followed by another
improvement at the second stage when temporal consistency is introduced. At the
third stage, performance starts to saturate.

 









  
Speed
Scene flow estimation is a computationally demanding task,
and efficient algorithms <cit.>
usually sacrifice accuracy for speed.   
Although the number of variables in our scene flow representation is huge and we make 
pixel-level predictions, our cascaded algorithm with MATLAB/C++ implementation on a single-core 2.5 Ghz CPU remains efficient. The main reason is
that we disentangle the output space,
and utilize efficient message-passing algorithms <cit.> to solve each high-dimensional inference problem.
Most of the computation time is spent on feature evaluation,
and could be accelerated using parallelization.








  
Failure Cases
As shown in Fig. <ref>, in challenging cases where the semantic segmentation algorithm fails to detect vehicle boundaries, our scene flow estimates can be inaccurate.
As previously studied for semantic optical flow methods <cit.>, we conducted an experiment using ground truth segmentation masks and witnessed a significant performance gain; see Table <ref>. 
Our framework for cascaded scene flow estimation will immediately benefit from future advances in semantic instance segmentation.






§ CONCLUSION




In this paper, we utilize semantic cues to identify rigidly moving objects, and thereby produce more accurate scene flow estimates for real-world scenes.
Our cascaded prediction framework allows computationally efficient recovery of high-dimensional motion and geometry estimates, and can flexibly utilize cues from sophisticated semantic segmentation algorithms.
We improve on the state-of-the-art for the challenging KITTI scene flow benchmark <cit.>.
While our experiments have focused on using vehicle detections to improve scene flow estimates for autonomous driving, our cascaded scene flow framework is directly applicable to any category of objects with near-rigid motion.





  
Acknowledgements This research supported in part by ONR Award Number N00014-17-1-2094. Some work was completed by Zhile Ren during an internship at NVIDIA.


ieee





