\section{Introduction}

Reservoir simuations are important tools for petroleum engineering, which have been applied to model
underground flows and interations between reservoirs and wells to predict the well
performance such as oil rates, water rates and bottom hole pressure. When a reservoir model is large enough,
it may take severals days or even longer for the simulator to complete a single model. Therefore, the numerical
methods and parallel techniques are essential to accelerate the simulations.

As a type of unconventional reservoirs, the reservoir with natural fractures and hydraulic fractures which are
commonly seen in tight and shale oil and gas reservoir, the dual-porosity/dual-permeability model and the
multiple iteration contiua (MINC) model\cite{pruess,wu-pruess} homogenize the fractures and use
superpositioned cells to represent the fractures and the matrix. The multiple continuum approaches have been
successfully employed in the unconventional reservoir problems. Wu et al.\cite{wu-qin,wu-di} regarded the
fractured vuggy rocks as a triple- or multiple-continuum medium with highly permeability and well-connected
fractures, low-permeabilty rock matrix and various-sized vugs.  Wu et al. also developed a hybrid
multiple-continuum-medium modeling appoach to describe different types of fractures including hydrautic
fractures, natural fracture network and micro fractures \cite{wu-li}. Jiang and Moinfar and their
collaborators designed explicit fracture
models coupled with MINC model to simulate the unconventional reservoirs \cite{moinfar,jiang}. In reference
\cite{wu-li-ding}, the authors presented a unified
framework model to incorporate the known mechanisms and processes including gas adsorption and desorption,
geomechanics effect, Klinkenberg or gas-slippage effect and non-Darcy flow. They also used a hybrid
fracture-modeling appoach to simulate the unconventional gas reservoirs. Jiang and Younis developed two
alternative hybrid approaches to capture the effects of the multiscale fracture systems by combining the
advantages of the multi-continuum and discrete-fracture/matrix representations  \cite{jiang-younis}.

The numerical methods for reservoir simulations have been explored by both mathematicians and engineers for
decades. Killough et al. \cite{killough} studied local refinement
techniques to improve the accuracy and reduce computational cost comparing to the global grid refinement.
Those local refinement techniques are useful for the complex models such as in-situ combustion.  Dogru and his
group \cite{dogru} developed parallel simulators using structured and unstructured grids to handle faults,
pinchouts, complex wells, polymer flooding in non-fractured and fractured reservoirs. Zhang et al.  developed
a general-purpose parallel platform for large-scale scientific applications which was designed using the
adaptive element methods and the adaptive finite volumn methods \cite{zhang,zhang-cui-liu}. It has
been applied to black oil simulation using discontinuous Galerkin methods\cite{wang-zhang-chen}. Chen et al.
studied finite element emtjods and finite different methods for black oil, compositional and thermal
models \cite{chen-huan-ma}. They studied Newton methods, linear solvers and preconditioners as well.
Chen and his collaborators also developed a parallel platform to support the new-generation simulator
development, such as black oil, compositional, thermal, polymer flooding models
\cite{para-frame,WLC-JCP,full-field}.  Wheeler studied
discretization methods, linear solvers, preconditioner techniques and developed a parallel black oil
simulator \cite{wheeler}. For the linear solver and preconditioners, many preconditioning methods have been
proposed and applied to reservoir simulations such as constrained pressure residual (CPR)
methods \cite{cpr,cao}, multi-stage methods \cite{dogru-wheeler}, multiple level preconditioners \cite{turkey},
fast auxiliary space preconditioners (FASP) \cite{auxiliary} and a family of parallel CPR-like
methods\cite{cpr}.

In this paper, we present the mathematical model of the dual-porosity reservoirs.
After that, we give the numerical methods used in our simulation inlcuding the nonlinear solvers,
the preconditioner and the linear solver and the parallel technique.
Finally, we show the numerical results obtained from our simulator.

\section{Mathematical Model}

Darcy's law describes the flow of a fluid through porous media, establishing the relationship
between the volumtric flow rate and the pressure gradient:
\begin{equation}
    Q = -\frac{KA\Delta p}{\mu L}
\end{equation}
where $K$ is the permeability of a given reservoir, $A$ is the area in the flow direction,
$\Delta p$ is the pressure difference, $\mu$ is the viscosity of the fluid and $L$ is the
length of the reservoir. In three dimensional space, its differential form is
\begin{equation}
    q =\frac{Q}{A}= -\frac{K\nabla p}{\mu }
\end{equation}
Combining Darcy's law and the mass conservation law for oil and water components, the
two-phase model is as follows:

\begin{equation}
    \left\{
        \begin{array}{ll}
            \frac{\partial}{\partial t}(\phi s_o\rho_o) &= \nabla \cdot (\frac{{\bm K}K_{ro}\rho_o}{\mu_o}\nabla \Phi_o) + q_o\\
            \frac{\partial}{\partial t}(\phi s_w \rho_w ) &= \nabla \cdot (\frac{{\bm K}K_{rw}\rho_w}{\mu_w} \nabla \Phi_w) + q_w\\
        \end{array}
        \right.
\end{equation}
where $\phi$ and ${\bm K}$ are the porosity and the permeabilities (in $x$-, $y$- and $z$-directions) respectively,
$\Phi_\alpha$ is the potential, $s_\alpha$, $\mu_\alpha$, $K_{r\alpha}$ and $q_\alpha$
($\alpha = o, w$) are the saturation, viscosity, relative permeability and production/injection rate,
respectively, which satisfy the following constraints:
\begin{equation}
    \left\{
        \begin{array}{l}
            \Phi_\alpha = p_\alpha - \rho_\alpha g z,\\
            s_o+s_w = 1,\\
            p_w = p_o - p_{cow}(s_w),\\
        \end{array}
        \right.
\end{equation}
where  $p_\alpha$ and $\rho_\alpha$ are the pressure and the mass density of phase $\alpha$($\alpha=o,~w$). $p_{cow}$
is the capillary pressure between oil phase and water phase, $g$ is the gravitational constant and $z$
is the reservoir depth.

\begin{figure}[!ht]\centering
    \includegraphics[scale = 0.2]{matrix-fracture.eps}
    \caption{The flow commmunication for dual porosity model \cite{cmg}}
    \label{matrix-fracture}
\end{figure}

For the dual-porosity model, fluids can flow between matrix and fracture, as well as fracture and fracture,
but there is no flow between matrices. There flow directions are described by Fig \ref{matrix-fracture}.
The following equations are used to describe this model:
\begin{equation}\label{prob}
    \left\{
        \begin{array}{l}
            \frac{\partial}{\partial t}({\phi_f s_{o,f} \rho_{o,f}}) = \nabla \cdot (\frac{{\bm K}_fK_{ro_f}
            \rho_{o,f}}{\mu_{o,f}}\nabla \Phi_{o,f})- q_{o,mf}+ q_o,\\
            \frac{\partial}{\partial t}({\phi_m s_{o,m} \rho_{o,f}}) = q_{o,mf},\\
            \frac{\partial}{\partial t}({\phi_f s_{w,f} \rho_{w,f}}) = \nabla \cdot (\frac{{\bm K}_fK_{rw_f}
            \rho_{w,f}}{\mu_{w,f}b_{w,f}}\nabla \Phi_{w,f})-q_{w,mf}+ q_w,\\
            \frac{\partial}{\partial t}({\phi_m s_{w,m} \rho_{w,m}}) = q_{w,mf}
        \end{array}
        \right.
\end{equation}
where $(\cdot)_f$ denotes the variables for the fracture and $(\cdot)_m$ for the matrix.
Note that here $q_{\alpha,mf} = \omega (\frac{K_{r\alpha} \rho_{\alpha,t}}{\mu_\alpha})_t(p_f - p_m)$ is a source
term which represents the net addition of the fluid to the fracture from the matrix with $\omega$ the shape
factor. In our simulator, the shape factor we use is based on the work of Warren and Root\cite{warren-root}
and Gilman and Kazemi\cite{gilman-kazemi}.  Note that $(\cdot)_t$ denoted the value at $t
= m$ or $t = f$ depending on which is the upsteam. The well rate at a perforation, $i$, is modelled by the sink-source method,
\begin{equation}
    q_{\alpha,i} = W_i \frac{{\bm K}_fK_{r\alpha_f} \rho_{\alpha,f}}{\mu_{\alpha,f}}(p_b - p), \quad \alpha = o,~w
\end{equation}
where $W_i$ is the well index at the perforation, $p_b$ is the bottom-hole pressure of the well at the perforation, $p$ is the
block pressure from oil phase at the perforation.
For a producer or an injection well, its well rate is calculated as,
\begin{equation}
    q_\alpha = \sum_{i}q_{\alpha,i},  \quad \alpha = o,~w.
\end{equation}
The well index is calculated as follows:
\begin{equation}
    W_i = 2\pi* k_h*w_{frac}/(\ln(r_e/r_w)+s).
\end{equation}
Here $k_h$ is a given value and $w_{frac}$ is the facture between 0.0 and 1.0 specifying the fraction of a circle
that the well models, $r_w$ is the wellbore radius, $s$ is a real number specifying the well skin factor,
$r_e$ is the well effective radius calculated from
\begin{equation}
    r_e = w_{g}*\sqrt{A_{r}/(\pi*w_{frac})}
\end{equation}
with $w_{g}$ the geometric factor for the well element, $A_{r}$ the area perpendicular to the
wellbore (e.g., $D_x*D_y$ for a vertical well) and $w_{frac}$ as same above.
The other option for the well effective radius is the following Peaceman's form \cite{peaceman}
\begin{equation}\displaystyle
    \left\{
        \begin{array}{l}
            r_{e,x} = 0.28\frac{\displaystyle[D_x^2(\frac{k_y}{k_x})^{1/2} + D_y^2(\frac{k_x}{k_y})^{1/2}]^{\frac{1}{2}}}{\displaystyle(\frac{k_y}{k_x})^{1/4}+ (\frac{k_x}{k_y})^{1/4}},\\
            r_{e,y} = 0.28\frac{\displaystyle[D_x^2(\frac{k_z}{k_x})^{1/2} + D_z^2(\frac{k_x}{k_z})^{1/2}]^{\frac{1}{2}}}{\displaystyle(\frac{k_z}{k_x})^{1/4}+ (\frac{k_x}{k_z})^{1/4}},\\
            r_{e,z} = 0.28\frac{\displaystyle[D_y^2(\frac{k_z}{k_y})^{1/2} + D_z^2(\frac{k_y}{k_z})^{1/2}]^{\frac{1}{2}}}{(\displaystyle\frac{k_z}{k_y})^{1/4}+ (\frac{k_y}{k_z})^{1/4}},\\
        \end{array}
        \right.
\end{equation}
where $r_{e,x}$, $r_{e,y}$ and $r_{e,z}$ are the well effective radius in $x$, $y$ and $z$ direction respectively,
$D_x$, $D_y$ and $D_z$ are the block sizes in $x$, $y$ and $z$ direction respectively, $k_x$, $k_y$ and $k_z$
are the permeabilities in $x$, $y$ and $z$ direction respectively.

\section{Numerical Methods and Parallel Computing}

In this paper, the fully implicit method (FIM) is applie, and the oil phase pressure $p$, water saturation
$s_w$ and the well bottom hole pressure $p_b$ are unknowns. For the time differential and the space
differential term, we use the backward Euler scheme and cell-centered finite difference method as
discretization methods.

The system is highly nonlinear,
\begin{equation}\label{nonlinear}
    F(x) = 0
\end{equation}
where $F$ is a nonlinear map from $\mathcal {R}^N$ to $\mathcal {R}^N$ with $N =  2
\times n + \tau$. Here $n$ is the number of grid blocks and $\tau$ is the well size.
In this nonlinear equation, the properties related to the saturation are strongly nonlinear
while the properties related to the pressure are weakly nonlinear.
In our simulation, we use Newton method (or inexact Newton method) to solve it.

\subsection{Nonlinear Solver}
The inexact Newton method can be regarded as an extension of the standard Newton method. Let $A$ be the Jacobian
matrix of $F(x)$ and $y$ is the correction between the last step approximate solution and the current step one.
Its algorithm \cite{chen-gewecke} is as follows:

\begin{algorithm}[!htb]
\caption{The Inexact Newton Method}
\label{inewton-alg}
\begin{algorithmic}[1]
\STATE Given an initial guess $x^0$ and a termination tolerance $\epsilon$, let $l = 0$
 and assemble the right-hand side $b$.
\WHILE{$\left\|b \right\| \ge \epsilon$}
\STATE Compute the Jacobian matrix $A$.
\STATE Determine $\eta_l$.
\STATE Find a solution $y$ such that
    \begin{equation}
    \label{inexact-newton-alg}
    \left\| A y - b \right\| \leq \eta_l \left\| b \right\|,
    \end{equation}
\STATE    Let $l = l+1$ and $x^l = x^{l-1} + y$.
\STATE    Compute the new right-hand side $b$
\ENDWHILE
\STATE $x^* = x^l$ is the approximate solution of the nonlinear system, $F(x) = 0$.
\end{algorithmic}
\end{algorithm}

We can see that the algorithm is the same as Newton method except the choice
of the tolerance $\theta$. The standard Newton method usually applies a small constant such as $1.0e-6$, while
the {\bf Algorithm 1} uses adaptive $\theta_l$ tolerance to avoid over solution. Common methods are defined
as follows \cite{chen-gewecke}:
\begin{equation}
    \theta_l =\left\{
    \begin{array}{l}
        \displaystyle\frac{\|b(x^l) - r^{l-1}\|}{\|b(x^{l-1})\|},\\
        \displaystyle\frac{\|b(x^l)\| - \|r^{l-1}\|}{\|b(x^{l-1})\|},\\
        \displaystyle\gamma (\frac{\|b(x^l)\|}{\|b(x^{l-1})\|})^\beta, \quad \gamma\in [0,~1], \quad \beta\in (1,2]
    \end{array}
    \right.
\end{equation}
where $r^l$ is the residual of the $l$-th iteration.

\subsection{Preconditioner and Linear Solver}

For each Newton iteration, we need to solve a linear system $Ax = b$ which is very time
comsuming. To improve the efficiency, we choose a proper preconditioner, CPR-FPF method \cite{cpr}, for the
model. Each grid block has four unknowns, pressures ($p_f$ and $p_m$) and saturations ($s_{w,f}$ and
$s_{w,m}$) for matrix and fracture, and four
equations. In our linear system, each unknown and each row are arranged cell by cell, in this case, the
Jacobian matrix is a block matrix. It is also well-known that block matrix has better convergence that
point-wise matrix.

Before solving the linear system $Ax=b$, a decoupling operation $D$ is applied, and an equivalent
system is obtained:
\begin{equation}\label{decoupled-eq}
    D^{-1}Ax = D^{-1} b.
\end{equation}
A proper decoupling method can improve linear solver dramatically. Many
decoupling strategies have been proposed, such as Quasi-IMPES\cite{quasi-impes} method and ABF\cite{abf}
method. In this paper, a modified Gauss-Jordan elimination method is applied.

\subsection{Parallel Computing}

The simulator is based on our parallel platform PRSI \cite{para-frame}, which is developed using C language
and MPI (Message Passing Interface). The platform has implemented many modules, such as
grid generation, load balancing, well management, parallel input and output, distributed matrix and vector, linear solver
and preconditioner, communication management and visualization.
Based on the platform, physical modules, such as rock properties and rock-fluid properties, are implemented.
More details can be read from reference \cite{para-frame}.

\section{Numerical Results}

\begin{example}
    \label{ex1}
    The grid dimension is $10\times 10\times 1$ with sizes 102.04 ft. in $x$ and $y$ directions
    and  100.0 ft. in $z$ direction from top to bottom. The depth of the top layer center is 2000.0 ft.
    The permeabilities for the matrix in $x$, $y$ and $z$ directions are 100, 100, 10 mD respectively.
    The permeability for the fracture in $x$, $y$ and $z$ directions are 395.85 mD.
    The reference porosity for the matrix is 0.1392. The reference porosity for the fracture is 0.039585.
    The rock compressibilities for the matrix and fracture are both 3e-06 (1/psi). The reference pressure
    is 15.0 psi for both the matrix and the fracture.
    Component properties: densities of oil and water are 58.0 lbm/ft3 and 62.4 lbm/ft3 respectively.
    The reference pressure is 15 psi at which the oil formation volume factor is 1.036 RB/STB and the oil viscosity is 40.0 cp.
    The oil compressibility is const 1.313e-5 l/psi.
    The initial conditions are as follows: initial pressure for the matrix is 2000 psi, initial pressure for the fracture is 1980 psi,
    and initial water saturations are 0.08 and 0.01 in matrix and fracture respectively.
    There are one injection well and two production wells. All of them are vertical.
    Injection well has maximum water injection rate 500.0 bbl/day, maximum bottom
    hole pressure 5.0e+4 psi, well index 200.0 with perforation at cell [5 1 1].
    Both of the production well has maximum oil production rate 300.0 STB/day, minimum bottom hole pressure 15 psi
    with well radius 0.25ft. The perforation of Produer 1 is at cell [1 10 1] while the perforation of Produer 2 is at cell [10 10 1].
    The simulation period is 800 days.
\end{example}

The results of oil production rate, bottom-hole-pressure and water rate are shown in
Fig. \ref{fig-dp-oil-rate}, \ref{fig-dp-bhp}, \ref{fig-dp-water-rate}, from which we can see that the results
from our simulator and from CMG IMEX match very well. This proves our methods and implementation are
correct.

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 0.5\textwidth]{dp-oil-rate.eps}
    \caption{Example \ref{ex1}: Oil production rate (unit: STB/day).}
    \label{fig-dp-oil-rate}
\end{figure}

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 0.5\textwidth]{dp-bhp.eps}
    \caption{Example \ref{ex1}: Well bottom-hole pressure (pressure unit: psi).}
    \label{fig-dp-bhp}
\end{figure}

\begin{figure}[!ht]\centering
    \includegraphics[width = 0.5\textwidth]{dp-water-rate.eps}
    \caption{Example \ref{ex1}: Water rate (unit: STB/day).}
    \label{fig-dp-water-rate}
\end{figure}

\begin{example}
    \label{ex2}
    The grid dimension is $10\times 10 \times 3$ with mesh size 102 ft. in $x$ and $y$ directions
    and  100.0 ft. in $z$ direction from top to bottom. The depth of the top layer
    center is 2000.0 ft.
    The permeabilities for the matrix in $x$, $y$ and $z$ directions are 100, 100, 100 mD respectively.
    The permeability for the fracture in $x$, $y$ and $z$ directions are 395.85 mD.
    The reference porosity for the matrix is 0.1392. The reference porosity for the fracture is 0.039585.
    The rock compressibilities for the matrix and fracture are both 3e-06 (1/psi). The reference pressure
    is 15.0 for both the matrix and the fracture.
    Component properties: densities of oil and water are 58.0 lbm/ft3 and 62.4 lbm/ft3 respectively.
    PVT: the reference pressure is 15 psi at which the oil formation volume factor is 1.036 RB/STB and the oil viscosity is 40.0 cp.
    The oil compressibility is const 1.313e-5 l/psi.
    The initial conditions are as follows: initial pressure for the matrix is 800 psi, initial pressure for the fracture is 500 psi,
    and initial water saturations are 0.08 and 0.01 in matrix and fracture respectively.
    There are one injection well and two production wells. All of them are vertical.
    Injection well has maximum water injection rate 200.0 bbl/day, maximum bottom
    hole pressure 5.0e+4 psi, well index 200.0 with perforation
    at cell [5 5 1].
    Both of the production well has maximum oil production rate 500.0 STB/day, minimum
    bottom hole pressure 15 psi with well radius 0.25ft. The perforation of Produer 1 is at
    cell [1 1 1] while the perforation of Produer 2 is at cell [10 10 1].
    The simulation period is 1600 days.
\end{example}

The results of oil production rate, bottom-hole-pressure and water rate are shown in
Fig. \ref{fig-mxfrr3-revised-oil-rate}, \ref{fig-mxfrr3-revised-bhp}, \ref{fig-mxfrr3-revised-water-rate}.
Again, these figures show that our simulator match CMG simulator.

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 0.5\textwidth]{mxfrr003-revised-oil-rate.eps}
    \caption{Example \ref{ex2}: Oil production rate (unit: STB/day).}
    \label{fig-mxfrr3-revised-oil-rate}
\end{figure}

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 0.5\textwidth]{mxfrr003-revised-bhp.eps}
    \caption{Example \ref{ex2}: Well bottom-hole pressure (pressure unit: psi).}
    \label{fig-mxfrr3-revised-bhp}
\end{figure}

\begin{figure}[!ht]
    \centering
    \includegraphics[width = 0.5\textwidth]{mxfrr003-revised-wrate.eps}
    \caption{Example \ref{ex2}: Water rate (unit: STB/day).}
    \label{fig-mxfrr3-revised-water-rate}
\end{figure}

\begin{example}
    \label{ex3}
    This example tests the scalability of our two-phase dual porosity simulator by computing Example 1 with
    grid dimension $500\times 500\times 50$.
\end{example}

As we introducerd before, we employ the GMRES linear solver and CPR preconditioner. Table \ref{tab-scale}
shows the numerical summaries, which show that our numerical methods are stable when increasing CPU cores.
Figure \ref{fig-scale} presents the scalability of our simulator, which demonstrates that the simulator
and parallel implementation are scalable.

\begin{table}[!ht]
    \centering
    \begin{tabular}{|c|c|c|c|c|}
        \hline
        \#MPIs & 8 & 64 \\
        \#Time steps & 68 & 68  \\
        \#Newton iterations& 273 & 267 \\
        \#total linear iterations & 641& 623 \\
        \#total running time(s) &20009.71 & 2531.91 \\
        \hline
    \end{tabular}\label{tab-scale}

\end{table}
\begin{figure}[!ht]\centering
    \includegraphics[width = 0.5\textwidth]{speedup.eps}
    \caption{Example \ref{ex3}: speed-up.}
    \label{fig-scale}.
\end{figure}

\section{Conclusion}

This paper presents our work on development of two-phase oil-water simulator for natural fractured
reservoirs using dual porosity method. Effective numerical methods and parallel computing techniques
are introduced.
From the numerical experiments, we can see that our results match commercial simulator, CMG IMEX,
and the simulatro has good parallel scalability and is capable of handling large scale reservoir simulation
problems.

\section{Acknowledgements}
This work is partially supported by Department of Chemical Petroleum Engineering,
University of Calgary, NSERC, AIEES, Foundation CMG, AITF iCore, IBM Thomas J. Watson
Research Center, Frank and Sarah Meyer FCMG Collaboration Center,
WestGrid (www.westgrid.ca),
SciNet (www.scinetpc.ca) and Compute Canada Calcul Canada (www.computecanada.ca).