 on 
a varying number of training frames in the input video.
The results are shown in \cref{fig:eval_generalization}a where each dot corresponds to averaging the score of $5$ different runs (each trained on another slice of the video). As seen, our framework is consistently better than the baseline. Also, as expected, the generalization increases (lower MSE) with the size of the training set, while the naive baseline stays the same. Also see that our framework generalizes quite well to next-frame prediction with as low as $4$ frames in the training set.

The second benchmark measures how well our framework generalizes on videos with faster motions. To this end we sub-sample the original video in intervals of increasing size
, making motions in the sub-sampled video move faster.
This way we can synthesize videos (all of which have $32$ frames) with larger speeds from the same video, making the results consistent with the first benchmark. As shown in \cref{fig:eval_generalization}b, our framework is consistently better than the baseline. Also larger speeds increase the gap between our framework and the baseline, further validating our hypothesis for motion generalization.
See more details in~\cref{sec:explaining_experiments}.
\vspace{-10pt}


\begin{figure}[!htbp]
    \centering
    \begin{tabular}{c}
    \includegraphics[width=.9\columnwidth]{figs/eval_train_size.pdf} \\
    (a) \\
    \includegraphics[width=.9\columnwidth]{figs/eval_speed.pdf} \\
    (b)
    \end{tabular}
    \caption{\textbf{Evaluating Prediction to \emph{Unseen Frames}}. See text.
   
    }
    \label{fig:eval_generalization}
    \centering
\end{figure}





\paragraph{Comparison to Single-Video Generation.}


\setlength{\tabcolsep}{3pt}
\begin{table}[!ht]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{c|c|ccc}
        Dataset & Method & NNFDIV $\uparrow$ & NNFDIST $\downarrow$ & SVFID$\downarrow$ \\
        \toprule
        \multirow{3}{*}{SinGAN-GIF}  &  VGPNN\cite{haim2021diverse} & $0.20$ & $0.28$ & 0.0058\\
        &  SinGAN-GIF~\cite{arora2021singan} & $0.40$ & $1.10$ & 0.0119 \\
        &  SinFusion (Ours) & $0.30$ & $0.45$ & 0.0090  \\
        \midrule
        \multirow{3}{*}{HP-VAE-GAN}  &  VGPNN\cite{haim2021diverse} & 0.22 & 0.14  & 0.0072 \\
        &  HP-VAE-GAN~\cite{gur2020hierarchical} & 0.31 & 0.39 & 0.0081\\
        &  SinFusion (Ours) & 0.35 & 0.26 & 0.0107  \\
    \end{tabular}
    }
    \caption{\textbf{Diverse Video Generation Comparison}.
    \label{tab:comparison}
    }
\end{table}

We compare our main application of diverse video generation to existing single-video methods -- HP-VAE-GAN~\cite{gur2020hierarchical}, SinGAN-GIF~\cite{arora2021singan} and VGPNN~\cite{haim2021diverse}.
Since none of these methods is autoregressive, the evaluations above cannot be directly applied to them. However, a qualitative observation hints that the motions in most of their generated outputs are somewhat copied from the input video. Measuring motion ``novelty'' in generated videos is hard, mainly because of the inherent entanglement between motion and appearance, namely, we want to consider only novel motions with similar appearance to the original motions, and penalize novel motion with unrelated appearance (e.g., random noise is novel in the sense that it is not in the input but also has  unrelated appearance). SinGAN~\cite{shaham2019singan} proposed a diversity measure that is easy to ``fool'' (see \cref{sec:fooling_singan_diversity}).

We introduce a nearest-neighbor-field (\textbf{NNF}) based diversity measure. The NNF is computed by searching for each $(3,3,3)$ spatio-temporal patch in the generated video, its nearest neighbour patch in the real video.
Simple generated videos (e.g. a translated or shuffled version of the input), will have a rather constant NNF, while more complex generated videos will have complex NNFs.
An NNF of a single frame can be viewed as a color map by converting the spatio-temporal direction to the nearest neighbour using a color wheel~\cite{baker2011database}.
In \cref{fig:nnf_plot} we show how such an NNF color map looks in a frame generated by VGPNN~\cite{haim2021diverse}, compared to a frame generated by our framework. The former is very simple, and corresponds to copying large chunks from a single input frame, whereas ours is more complex, which corresponds to a diverse generation from the input video (\emph{see full videos in the project page}).

Inspired by \emph{Kolmogorov complexity}~\cite{kolmogorov1963tables} (simpler objects have simpler ``description''), which can be easily bounded by any compression algorithm, we use ZLIB~~\cite{gailly2004zlib} to compress the NNF, and record the compression ratio. This gives a diversity measure (in $[0,1]$) that we term \emph{NNFDIV}. We can measure the RGB-similarity (\emph{NNFDIST}) by recording the MSE of each generated patch to its nearest-neighbor. In \cref{tab:comparison} we report the results of these metrics on two diverse video generation datasets (\cref{sec:video_dataset_comparison_appendix}). As seen in the table, in both cases SinFusion has the best trade-off in terms of diversity and similarity. We also report SVFID~\cite{gur2020hierarchical} that measures similarity. while VGPNN is expected to have the best score (since it is copying chunks of frames), our score is between SinganGIF and HP-VAE-GAN. We attribute this to patch outliers in the generations (as apparent from the results), however similarity is not our main goal but the generation of novel motions.




\paragraph{Further Experiments and Ablations} can be found in \cref{sec:comparison_appendix} and \cref{sec:ablations_appendix} (e.g., comparison to VDM~\cite{ho2022video}).





\begin{figure}
    \centering
    \includegraphics[width=.9\columnwidth]{figs/nnf_plot.png}
    \caption{\textbf{Nearest-Neighbour Field (NNF) Color Map.}
    By computing the patch-NNF between generated video and the input video we see that VGPNN~\cite{haim2021diverse} outputs are actually copies of chunks, while ours generate novel patches.
    }
    \label{fig:nnf_plot}
    \centering
\end{figure}





\section{Limitations}
\label{sec:limitations}

As all methods for generation from a single video, our method is limited to videos with little to no camera motion. Also, when objects have relatively complicated motion (e.g., with many moving parts) our framework can have difficulties learning and discerning between different parts of the complex movements.
Some of the above may be mitigated by including suitable priors.

\section{Conclusions}
We propose SinFusion, a DDPM framework trained on a single video or image. Our unified framework can be applied for a variety of tasks. Our main application, generation from a single video, exhibits generalization capabilities that were not shown neither by previous single-video methods, nor by DDPMs for video generation.


\section*{Acknowledgements}
We thank Assaf Shocher and Barack Zackay for useful discussions. This project received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 788535) and from the D. Dan and Betty Kahn Foundation.



{\small
\bibliographystyle{plainnat}
I love the filters and the editing system for photos. The app is somewhat hard to navigate and I wish that the options that come to Apple would come to Android devices much sooner.
I subscribed to vscox and I'm still being asked to purchase new presets. Please fix this.
I can't find HB filters. Please bring them back to me. It is free filters but why I can't find them on VSCO.
The app UI have some