abstract: We describe SLING, a framework for parsing natural language into semantic frames. SLING supports general transition-based, neural-network parsing with bidirectional LSTM input encoding and a Transition Based Recurrent Unit (TBRU) for output decoding. The parsing model is trained end-to-end using only the text tokens as input. The transition system has been designed to output frame graphs directly without any intervening symbolic representation. The SLING framework includes an efficient and scalable frame store implementation as well as a neural network JIT compiler for fast inference during parsing. SLING is implemented in C++ and it is available for download on GitHub.
author: Michael Ringgaard
Google Inc.
`email@example.com`
Rahul Gupta
Google Inc.
`firstname.lastname@example.com`
Fernando C. N. Pereira
Google Inc.
`email@example.com`
bibliography: sling.bib
title: SLING: A framework for frame semantic parsing

# Introduction

Recent advances in machine learning make it practical to train recurrent multi-level neural network classifiers, allowing us to rethink the design and implementation of natural language understanding (NLU) systems.

Earlier machine-learned NLU systems were commonly organized as pipelines of separately trained stages for syntactic and semantic annotation of text. A typical pipeline would start with part-of-speech (POS) tagging, followed by constituency or dependency parsing for syntactic analysis. Using the POS tags and parse trees as feature inputs, later stages in the pipeline could then derive semantically relevant annotations such as entity and concept mentions, entity types, coreference relationships, and semantic roles (SRL).

For simplicity and efficiency, each stage in a practical NLU pipeline would just output its best hypothesis and pass it on to the next stage . Obviously, errors could then accumulate throughout the pipeline making it much harder for the system to perform accurately. For instance, F1 on SRL drops by more than 10% when going from gold to system parse trees .

However, applications may not need the intermediate annotations produced by the earlier stages of a NLU pipeline, so it would be preferable if all stages could be trained together to optimize an objective based on the output annotations needed for a particular application.

Earlier NLU pipelines often used linear classifiers for each stage. Linear classifiers achieve simplicity and training efficiency at the expense of feature complexity, requiring elaborate feature extraction, many different feature types, and feature combinations to achieve reasonable accuracy. With deep learning, we can use embeddings, multiple layers, and recurrent network connections to reduce the need for complex feature design. The internal learned representations in model hidden layers replace the hand-crafted feature combinations and intermediate representations in pipelined systems.

The SLING parser exploits deep learning to bypass those limitations of classic pipelined systems. It is a transition-based parser that outputs frame graphs directly without any intervening symbolic representation (see Section ). Transition-based parsing is often associated with dependency parsing, but we have designed a specialized transition system that outputs frame graphs instead of dependency trees.

We use a recurrent feed-forward unit for predicting the actions in the transition sequence, where the hidden activations from predicting each transition step are fed back into subsequent steps. A bidirectional LSTM (biLSTM) encodes the input into a sequence of vectors(Figure ). This neural network architecture has been implemented using DRAGNN  and TensorFlow .

The SLING framework and a semantic parser built in it are now available as open-source code on GitHub.[^1]

In Section  we introduce *frame semantics*, the linguistic theory that inspired SLING, as well as the SLING frame store, a C++ framework for representing and storing semantic frames compactly and efficiently. Section  introduces the parser's frame-semantics-oriented attention mechanism, and Section  describes the transition system used for producing frame graphs. Section  describes the features used by the parser. In sections  and  we describe our experiments on OntoNotes, and section  describes the fast parser runtime.

# Frame semantics

While frames in SLING are not tied to any particular linguistic theory or knowledge ontology, they are inspired by *frame semantics*, the theory of linguistic meaning originally developed by Charles Fillmore . Frame semantics connects linguistic semantics to encyclopedic knowledge, with the central idea that understanding the meaning of a word requires access to all the essential knowledge that relates to that word. A word *evokes* a frame representing the specific concept it refers to.

A semantic frame is a set of statements that give "characteristic features, attributes, and functions of a denotatum, and its characteristic interactions with things necessarily or typically associated with it." . A semantic frame can also be viewed as a coherent group of concepts such that complete knowledge of one of them requires knowledge of all of them.

Frame semantics is not just for individual concepts, but can be generalized to phrases, entities, constructions, and other larger and more complex linguistic and ontological units. Semantic frames can also model world knowledge and inferential relationships in common sense, metaphor , metonymy, action , and perspective .

# Frames in SLING

SLING represents frames with data structures consisting of a list of slots, where each slot has a name (role) and a value. The slot values can be literals like numbers and strings, or links to other frames. A collection of interlinked frames can thus be seen as a directed graph where the frames are the (typed) nodes and the slots are the (labeled) edges. A frame graph can also be viewed as a feature structure  and unification can be used for induction of new frames from existing frames. Frames can also be used to represent more basic data structures such as a C struct with fields, a JSON object, or a record in a database.

SLING frames live inside a *frame store*. A store is a container that tracks all the frames that have been allocated in the store, and serves as a memory allocation arena for them. When making a new frame, one specifies the store where the frame should be allocated. The frame will live in this store until the store is deleted or the frame is garbage collected because there no remaining live references to it.[^2]

SLING frames are externally represented in a superset of JSON that allows references between frames (JSON objects) with the `#n` syntax. Frames can be assigned identifiers (*ids*) using the `=#n` syntax. SLING frames can have both numeric and named ids and both slot names and values can be frame references. Where JSON objects can only represent trees, SLING frames can be used for representing arbitrary graphs. SLING has special syntax for built-in slot names:

| Syntax | Symbol   | RDF             |
|:-------|:---------|:----------------|
| =name  | id:name  | rdf:ID          |
| :name  | isa:name | rdf:InstanceOf  |
| +name  | is:name  | rdfs:subClassOf |

Documents are also represented using frames, where the document frame has slots for the document text, the tokens, and the mention phrases and the frames they evoke. See Figure  for an example.

# Attention

The SLING parser is a kind of sequence-to-sequence model that first encodes the input text token sequence with a bidirectional LSTM encoder and then runs the transition system on that encoding to produce a sequence of transitions, where each transition updates the system state that combined with the input encoding form the input for the transition feed-forward cell that predicts the next transition (Figure ).

Sequence-to-sequence models often rely on an "attention" mechanism to focus the decoder on the parts of the input most relevant for producing the next output symbol. In this work, however, we use a somewhat difference attention mechanism, loosely inspired on neuroscience models of attention and awareness  . In our model, attention focuses on parts of the frame representation that the parser has created so far, rather than focusing on (encodings of) input tokens as is common for other sequence-to-sequence attention mechanisms.

We maintain an *attention buffer* as part of the transition system state. This an ordered list of frames, where the order represents closeness to the center of attention. Transition system actions maintain the attention buffer, bringing a frame to the front when the frame is evoked or re-evoked by the input text. When a new frame is evoked, it will merge the concept and its roles into a new coherent chunk of meaning, which is represented by the new frame and its relations to other frames, and this will become the new center of attention. Our hypothesis is that by maintaining this attention mechanism, we only need to look at a few recent frames brought into attention to build the desired frame graph.

# Transition system

*Transition systems* are widely used in parsing to build dependency parse trees as a side effect of performing a sequence *state transitions* $(s_i,a_i)$ where $s_i$ is a *state* and $a_i$ is an *action*. Action $a_i$ computes the new state $s_{i+1}$ from state $s_i$. For example, the *arc-standard* transition system  uses a sequence of **SHIFT**, **LEFT-ARC(label)**, and **RIGHT-ARC(label)** actions, operating on a state whose main component is a stack, to build a dependency parse tree.

We use the same idea to construct a frame graph where frames can be evoked by phrases in the input. But instead of using a stack in the state, we use the attention buffer introduced in the previous section that keeps track of the most salient frames in the discourse.

The attention buffer is a priority list of all the frames evoked so far. The front of the buffer serves as the working memory for the parser. Actions operate on the front of the buffer and in some cases other frames in the buffer. The transition system simultaneously builds the frame graph and maintains the attention buffer by moving the frame involved involved in an action to the front of the attention buffer. At any time, each evoked frame has a unique position in the attention buffer.

The transition system consists of the following actions:

-   **SHIFT** – Moves to next input token. Only valid when not at the end of the input buffer.

-   **STOP** – Signals that we have reach the end of the parse. This is only valid when at the end of the input buffer. Multiple STOP actions can be added to the transition sequence, e.g. to make all sequences in a beam have the same length. After a STOP is issued, no other actions are permitted except more STOP actions.

-   **EVOKE(type, n)** – Evokes a frame of type **type** from the next **n** tokens in the input. The evoked frame is inserted at the front of the attention buffer, becoming the new center of attention.

-   **REFER(frame, n)** – Makes a new mention from the next **n** tokens in the input evoking an existing frame in the attention buffer. This frame is moved to the front of the attention buffer and will become the new center of attention.

-   **CONNECT(source, role, target)** – Adds slot to **source** frame in the attention buffer with name **role** and value **target** where **target** is an existing frame in the attention buffer. The **source** frame become the new center of attention.

-   **ASSIGN(source, role, value)** – Adds slot to **source** frame in the attention buffer with name **role** and constant value **value** and moves the frame to the front of the buffer. This action is only used for assigning a constant value to a slot, in contrast to **CONNECT** where the value is another frame in the attention buffer.

-   **EMBED(target, role, type)** – Creates a new frame with type **type** and adds a slot to it with name **role** and value **target** where **target** is an existing frame in the attention buffer. The new frame becomes the center of attention.

-   **ELABORATE(source, role, type)** – Creates a new frame with type **type** and adds a slot to an existing frame **source** in the attention buffer with **role** set to the new frame. The new frame becomes the center of attention.

In summary, **EVOKE** and **REFER** are used to evoke frames from text mentions, while **ELABORATE** and **EMBED** are used to create frames not directly evoked by text.

This transition system can generate any connected frame graph where the frames are either directly on indirectly evoked by phrases in the text. A frame can be evoked by multiple mentions and the graph can have cycles.

The transition system can potentially have an unbounded number of actions since it is parameterized by phrase length and attention buffer indices which can be arbitrarily large. In the current implementation, we only consider the top $k$ frames in the attention buffer ($k=5$) and we do not consider any phrases longer than those in the training corpus.

Multiple transition sequences can generate the same frame annotations, but we have implemented an oracle sequence generator that takes a document and converts it to a canonical transition sequence in a way similar to how this is done for transition-based dependency parsing . For example, the sentence "John hit the ball" generates the following transition sequence:

      EVOKE(/saft/person, 1)
      SHIFT
      EVOKE(/pb/hit-01, 1)
      CONNECT(0, /pb/arg0, 1)
      SHIFT
      SHIFT
      EVOKE(/saft/consumer_good, 1)
      CONNECT(1, /pb/arg1, 0)
      SHIFT
      STOP

# Features

The biLSTM uses only lexical features based on the current input word:

-   The current word itself. During training we initialize the embedding for this feature from pre-trained word embeddings  for all the words in the the training data.

-   The prefixes and suffixes of the current input word. We use only prefixes up to three characters in our experiments.

-   Word shape features based on the characters in the current input word: hyphenation, capitalization, punctuation, quotes, and digits. Each of these features has its own embedding matrix.

The TBRU is a simple feed-forward unit with a single hidden layer. It takes the hidden activations from the biLSTM as well as the activations from the hidden layer from the previous steps as raw input features, and maps them through embedding matrices to get the input vector for the hidden layer. More specifically, the inputs to the TBRU are as follows:

-   The left-to-right and right-to-left LSTMs supply their activations for the current token in the parser state.

-   The attention feature looks at the top-$k$ frames in the attention buffer and finds the phrases in the text (if any) that evoked them. The activations from the left-to-right and right-to-left LSTMs for the last token of each of those phrases are are included as TBRU inputs, serving as continuous lexical representations of the top-$k$ frames in the attention buffer.

-   The hidden layer activations of the transition steps which evoked or brought into focus the top-$k$ frames in the attention buffer are also inputs to the TBRU, providing a continuous representation for the semantic frame contexts that evoked those frames most recently.

-   The history feature uses the hidden activations in the feed-forward unit from the previous $k$ steps as feature inputs to the current step.

-   Embeddings of triples of the form $(s_i, r_i, t_i)$, $0<s_i,t_i\le k$ encode the fact that the frame at position $s_i$ in the attention buffer has a role $r_i$ with the frame at position $t_i$ in the attention buffer as its value. Back-off features are added for the source roles $(s_i,r_i)$, target role $(r_i, t_i)$, and unlabeled roles $(s_i,t_i)$.

# Experiments

We derived a corpus annotated with semantic frames from the OntoNotes corpus . We took the PropBank SRL layer  and converted the predicate-argument structures into frame annotations. We also annotated the corpus with entity frames based on entity types from a state-of-the-art entity tagger. We determined the head token of each argument span and if this coincided with the span of an existing frame, then we used it as the evoking span for the argument frame, otherwise we just used the head token as the evoking span of the argument frame.

The various frame types mentioned above are listed in Table . They include 7 conventional entity types, 6 top-level non-entity types (e.g. date), 13 measurement types, and more than 5400 PropBank frame types. All the frame roles are collapsed onto /pb/arg0, /pb/arg1, and so on. Our training corpus size was $111,006$ sentences, $2,206,274$ tokens.

```latex
\begin{table*}[t]\begin{tabular}{|l|p{11cm}|}
\hline
{\bf Type set} & {\bf Details} \\
\hline
Entity types & /saft/\{person, location, organization, art, consumer\_good, event, other\} \\
\hline
Top-level non-entity types & /s/\{thing, date, price, measure, time, number\} \\
\hline
Fine-grained measure types & /s/measure/\{area, data, duration, energy, frequency, fuel, length, mass, power, speed, temperate, voltage, volume\} \\
\hline
PropBank SRL types & 5426 types, e.g. /pb/write-01, /pb/tune-02  \\
\hline
\end{tabular}
\caption{Frame types used in the experiments.}
\label{tab:types}
\end{table*}
```

| **Action Type** | **\# Unique Args** | **Raw Count** |
|:----------------|-------------------:|--------------:|
| SHIFT           |                  1 |     2,206,274 |
| STOP            |                  1 |       111,006 |
| EVOKE           |              5,532 |     1,080,365 |
| CONNECT         |              1,421 |       635,734 |
| ASSIGN          |                 13 |         5,430 |
| **Total**       |              6,968 |     4,038,809 |

Action statistics for the transitions that generated the gold frames for the OntoNotes training corpus.

Table  shows action statistics for the transition sequences that generate the gold frames in the training corpus. As expected, there is one SHIFT action per training token, and one STOP action per training sentence. The EVOKE action occurred with $5,532$ unique (length, type) arguments in the corpus, for a raw count of roughly $1.08$ million action tokens. Overall our action space had $6968$ action types, which is also the size of the softmax layer of our TBRU decoder.

Our final set of hyperparameters after grid search with a dev corpus was: $\mbox{learning\_rate} = 0.0005$, $\mbox{optimizer} = \mbox{Adam}$  with $\beta_1 = 0.01$, $\beta_2 = 0.999$, $\epsilon = 1e-5$, no dropout, gradient clipping at $1.0$, exponential moving average, no layer normalization, and a training batch size of $8$. We use $32$ dimensional word embeddings, single layer LSTMs with $256$ dimensions, and a $128$ dimensional hidden layer in the feed-forward unit.

We stopped training after $120,000$ steps, where each step corresponds to processing one training batch, and evaluated on the dev corpus ($15,084$ sentences) after every checkpoint (= $2,000$ steps). Figure  shows the how the various evaluation metrics evolve as training progresses. Section  contains the details of these metrics are evaluated. We picked the checkpoint with the best 'Slot F1' score.

# Evaluation

An annotated document consists of a number of connected frames as well as phrases (token spans) that evoked these frames. We evaluated annotation quality by comparing the generated frames with the gold standard frame annotations from the evaluation corpus.

Two documents are matched by constructing a virtual graph where the document is the start node. The document node is then connected to the spans and the spans are connected to the frames that the spans evoke. This graph is then extended by following the frame-to-frame links via the roles. Quality is computed by aligning the golden and predicted graphs and computing precision, recall, and F1. Those scores are separately computed for spans, frames, frame types, roles that link to other frames (referred to as 'roles'), and roles that link to global constants (referred to as 'labels').

We also report two aggregate quality scores: (a) *Slot*, which is an aggregate of *Type*, *Role*, and *Label*, and (b) *Combined*, which is an aggregate of *Span*, *Frame*, *Type*, *Role*, and *Label*.

We rated the checkpoints using the Slot-F1 metric and selected the checkpoint with the best Slot-F1. Intuitively, a high *Slot* score reflects that the right type of frames are being evoked, along with the right set of slots and links to other frames.

Figure  shows that as training progresses, the model learns to output the spans and frames evoked from those spans with fairly good quality (SPAN F1 $\approx$ FRAME F1 $\approx$ $93.81\%$). It also gets the type of those frames right with a TYPE F1 of $= 85.88\%$. ROLE F1 though is lower at just $69.65\%$. ROLE F1 measures the accuracy of correctly getting the frame-frame link, including the label of the link. Further error analysis will be required to understand how frame-frame links are missed by the model. Also note that currently the *roles* feature is the only one that captures inter-frame link information. Augmenting this with more features should help improve ROLE quality, as we will investigate in future work.

Finally, we took the best checkpoint, with SLOT F1 $= 79.95\%$ at $118,000$ steps, and evaluated it on the test corpus. Table  lists the quality of this model on the test and dev corpora. With the exception of LABEL accuracies, all the other metrics exhibit less than half a percent difference between the test and dev corpora. This illustrates that despite the lack of dropout, the model generalizes well to unseen text. As for the disparity on LABEL F1 ($95.73$ on dev against $92.81$ on test), we observe from Figure  that the LABEL accuracies follow a different improvement pattern during training. On the dev set, LABEL F1 peaked at $96.18$ at $100,000$ steps, and started degrading slightly from there on to $95.73$ at $118,000$ steps, possibly showing signs of overfitting which are absent in the other metrics.

| **Metric** |           | **Dev** | **Test** |
|:-----------|:----------|--------:|---------:|
| Tokens     |           | 291,746 |  216,473 |
| Sentences  |           |  15,084 |   11,623 |
| Span       | Precision |   93.42 |    93.04 |
|            | Recall    |   94.21 |    94.34 |
|            | F1        |   93.81 |    93.69 |
| Frame      | Precision |   93.47 |    93.20 |
|            | Recall    |   94.16 |    94.08 |
|            | F1        |   93.81 |    93.64 |
| Type       | Precision |   85.56 |    85.67 |
|            | Recall    |   86.20 |    86.49 |
|            | F1        |   85.88 |    86.08 |
| Role       | Precision |   70.21 |    69.59 |
|            | Recall    |   69.11 |    69.20 |
|            | F1        |   69.65 |    69.39 |
| Label      | Precision |   96.51 |    95.02 |
|            | Recall    |   94.97 |    90.70 |
|            | F1        |   95.73 |    92.81 |
| Slot       | Precision |   80.00 |    79.81 |
|            | Recall    |   79.90 |    80.10 |
|            | F1        |   79.95 |    79.96 |
| Combined   | Precision |   87.46 |    87.20 |
|            | Recall    |   87.79 |    87.91 |
|            | F1        |   87.63 |    87.55 |

Evaluation on dev and test corpora, model chosen on the Slot-F1 metric on dev corpus.

We have tried increasing the sizes of the LSTM dimensions, hidden layers, and embeddings, but this did not improve the results significantly.

# Parser runtime

The SLING parser uses TensorFlow  for training but it also supports annotating text with frame annotations at runtime. It can take advantage of batching and multi-threading to speed up parsing. However, in practical applications of the parser, it may not be convenient to batch documents for processing, so to have a realistic benchmark, we set the batch size to one at runtime. In this configuration, the TensorFlow-based SLING parser runs at 200 tokens per CPU second.

To speed up parsing, we have created *Myelin*, a just-in-time compiler for neural networks that compiles network cells into x64 machine code at runtime. The generated code exploits such specialized CPU features as SSE, AVX, and FMA3, if available. Tensor shapes and model parameters are fixed at runtime. This allows us to optimize the network by folding constants, unrolling loops, and pre-computing embeddings, among other transformations. The JIT compiler can also fix the data instance layout at compile-time to speed up runtime data access.

The Myelin-based SLING parser runs at 2500 tokens per CPU second, more than ten times faster than the TensorFlow-based version (Table ).

|         |          |           |          |
|:--------|---------:|----------:|---------:|
| Runtime |    Speed |   Runtime |     Load |
|         |          |      size |     time |
| TF      |  200 TPS | 37.000 KB |  10 secs |
| Myelin  | 2500 TPS |    500 KB | 0.5 secs |

Comparison between TensorFlow-based SLING parser runtime and Myelin-based parser runtime using JIT compilation. Speed is measured as tokens parsed per CPU second, i.e. user+sys in time(1).

The Myelin-based SLING parser is independent of TensorFlow so it only needs to link with the Myelin runtime (less than 500 KB) instead of the TensorFlow runtime library (37 MB), and it is also much faster to initialize (0.5 seconds including compilation time) than the TensorFlow-based parser (10 seconds). Figure  shows a breakdown of the CPU time for the Myelin-based parser runtime.

Half the time is spent computing the logits for the output actions. This is expensive because the OntoNotes-based corpus has 6968 actions, where the vast majority of the actions are of a form like **EVOKE(/pb/hit-01, 1)**, one for each PropBank roleset predicate in the training data. Table  shows that only about 26% of all the actions are EVOKE actions. The output layer of the FF unit could be turned into a cascaded classifier, where if the first classifier predicts a generic **EVOKE(/pb/predicate, 1)** action, it would use a secondary classifier to predict the predicate type. This could almost double the speed of the parser.

# Conclusion

We have described SLING, a framework for parsing natural language into semantic frames. Our experiments show that it is feasible to build a semantic parser that outputs frame graphs directly without any intervening symbolic representation, only using the tokens as inputs. We illustrated this on the joint task of predicting entity mentions, entity types, measures, and semantic role labeling. While the LSTMs and TBRUs are expensive to compute, we can achieve acceptable parsing speed using the Myelin JIT compiler. We hope to make use of SLING in the future for further exploration into semantic parsing.

# Acknowledgements

We would like to thank Google for supporting us in this project and allowing us to make SLING available to the public community. We would also like to thank the Tensorflow and DRAGNN teams for making their systems publicly available. Without it, we could not have made SLING open source.

[^1]: <https://github.com/google/sling>

[^2]: See the [SLING Guide](https://github.com/google/sling/blob/master/frame/README.md) for a detailed description of the SLING frame store implementation.


abstract: It has long been speculated that supernova remnants represent a major source of cosmic rays in the Galaxy. Observations over the past decade have ceremoniously unveiled direct evidence of particle acceleration in SNRs to energies approaching the knee of the cosmic ray spectrum. Nonthermal X-ray emission from shell-type SNRs reveals multi-TeV electrons, and the dynamical properties of several SNRs point to efficient acceleration of ions. Observations of TeV gamma-ray emission have confirmed the presence of energetic particles in several remnants as well, but there remains considerable debate as to whether this emission originates with high energy electrons or ions. Equally uncertain are the exact conditions that lead to efficient particle acceleration.
  .
  Based on the catalog of EGRET sources, we know that there is a large population of Galactic gamma-ray sources whose distribution is similar to that of SNRs. With the increased resolution and sensitivity of GLAST, the gamma-ray SNRs from this population will be identified. Their detailed emission structure, along with their spectra, will provide the link between their environments and their spectra in other wavebands to constrain emission models and to potentially identify direct evidence of ion acceleration in SNRs. Here I summarize recent observational and theoretical work in the area of cosmic ray acceleration by SNRs, and discuss the contributions GLAST will bring to our understanding of this problem.
author: Patrick Slane
title: Supernova Remnants and GLAST

address=Harvard-Smithsonian Center for Astrophysics

# Introduction

Supernova remnants represent an energetic class of objects that is intimately connected to sources of $\gamma$-ray emission in our Galaxy. Shock acceleration by the SNR blast wave provides ample energy for the production of multi-TeV particles, and the presence of nearby material in dense clouds from which the remnant progenitors collapsed forms a natural target for pion production with subsequent $\pi^0 \rightarrow \gamma \gamma$ decay. Nonthermal bremsstrahlung of electrons off ambient material, as well as inverse Compton (IC) scattering of electrons off ambient photons, can also lead to $\gamma$-ray production – and in many cases are the dominant mechanisms. SNRs are thus strong candidates for the emission of $\gamma$-rays. X-ray observations have revealed numerous remnants for which nonthermal emission from their shells provides direct evidence of TeV electrons. More recently, TeV $\gamma$-ray emission has been detected from a number of SNRs, possibly providing evidence for acceleration of ions as well.

The EGRET source catalog contains a large number of unidentified sources in the Galactic plane, some of which have been tentatively associated with SNRs, and such associations are plausible – at least in principle; particularly for remnants that are expanding into dense environments, we expect emission that extends well into the EGRET band. However, the large positional uncertainties for these sources, along with the large sizes and high number density of SNRS, yield a relatively high probability for false identifications of EGRET sources with SNRs. Yet the emission in this band holds particular importance in helping to understand the emission at higher energies, and in addressing the overall picture of cosmic-ray acceleration in SNRs. Below I discuss the production of $\gamma$-ray emission from SNRs and describe recent results on evidence for cosmic-ray acceleration in SNRs, with particular emphasis on contributions that are expected from GLAST observations.

# Particle Acceleration by SNR Shocks

As the blast wave from a supernova explosion expands, the surrounding circumstellar material (CSM) and interstellar medium (ISM) is swept up into a shell of hot gas. The shock jump conditions yield an increase in density by a factor of $n/n_0 = (\gamma +1)/(\gamma -1)$, and the associated postshock temperature is given by $$T = \frac{2 (\gamma -1)}{(\gamma + 1)^2} \frac{\mu}{k} m_p V_s^2$$ where $\gamma$ is the adiabatic index for the gas ($\gamma = 5/3$ for an ideal gas), $V_s$ is the shock speed, $m_p$ is the proton mass, and $\mu$ is the mean molecular weight ($\mu \approx
0.6$ for solar abundances). This shock-heated gas produces X-ray emission characterized by a thermal bremsstrahlung spectrum accompanied by emission lines. As the blast wave decelerates upon sweeping up increasing amounts of material, a reverse shock propagates back into the ejecta. At early times, the X-ray spectrum is dominated by emission from the reverse-shocked ejecta; as the amount of swept-up material increases, the spectrum becomes dominated by emission from material with ISM abundances.

In addition to thermal heating of the swept-up gas, some fraction of the shock energy density goes into production of relativistic particles through diffusive shock acceleration wherein energetic particles streaming away from the shock form turbulent waves which act to scatter other particles back toward the shock. Subsequent reacceleration builds up a nonthermal population of high energy particles, with the maximum energy being limited by radiative losses, the age of the SNR, or particle escape. Such particle acceleration by SNR shocks has long been suggested as a process by which cosmic rays are produced, and radio observations provide direct evidence of nonthermal electrons with energies of several GeV.

If the relativistic particle component of the energy density becomes comparable to that of the thermal component, the shock acceleration process can become highly nonlinear. The gas becomes more compressible (with $\gamma$ becoming much larger than 5/3), which results in a higher density and enhanced acceleration. The resulting efficient production of nonthermal particles has a significant impact on the dynamical evolution of the shock. Figure 1 (left, from ) shows the SNR temperature as a function of radius at an age of 500 yr, assuming expansion into a uniform density $n_0 =
0.1 {\rm\ cm}^{-3}$ with an ambient magnetic field strength $B_0 = 15 \mu$G. The short-dashed curve (labeled TP) corresponds to the test-particle case in which no diffusive shock acceleration operates. The regions corresponding to the forward shock (FS), the reverse shock (RS) and the contact discontinuity (CD) that separates the swept-up and ejecta components are indicated. Also shown are temperature plots for cases of moderate ($\epsilon
= 36\%$) and efficient ($\epsilon = 63\%$) particle acceleration. (Here $\epsilon$ refers to the fraction of the energy flux crossing the shock that ends up in relativistic particles.) Two distinct observable effects are immediately evident: at a given age, the separation between the FS and the RS (or CD) decreases considerably with increased particle acceleration; and the temperature at the forward shock is reduced for cases of high acceleration efficiency.

# Nonthermal X-ray Emission from SNRs

While the shock-heated ejecta and CSM/ISM components of shell-type SNRs produce line-dominated X-ray spectra, as described above, a growing number of remnants also reveal evidence of hard, nonthermal X-ray emission components, apparently associated with high energy electrons accelerated by the SNR shock. In several cases – e.g., SN 1006 , G347.3$-$`<!-- -->`{=html}0.5 , and G266.2$-$`<!-- -->`{=html}1.2 – the nonthermal emission components completely dominate the thermal components, and the X-ray spectra from the shells are featureless. For others, including most of the known young SNRs – e.g., Cas A , Tycho , and Kepler – thin rims of nonthermal emission surround the remnants directly along their forward shocks. These observations make it clear that SNRs are capable of accelerating electrons to very high energies. It is assumed that ions are accelerated by the same process although, as we describe below, the evidence for this is less direct. Since ions dominate the cosmic-ray energy density, however, it is exactly this evidence that is of particular importance for our understanding of cosmic ray production.

The *Chandra* image of Tycho's SNR is shown in Figure 1 (right). Spectral studies show that the thin outer rim is dominated by featureless emission consistent with synchrotron radiation from energetic electrons, while the brighter emission seen in a broad shell behind the rim is from shock-heated ejecta . The position of the reverse shock is along the inner regions of this ejecta component while the contact discontinuity separating the ejecta and forward shock is located at the inner edge of the faint region separating the ejecta and the forward shock.

As discussed above, the separation between the forward shock and the contact discontinuity is strongly modified by the efficient acceleration of cosmic ray particles (Figure 1, left). In Tycho's SNR, this separation is considerably smaller than expected at the known age of the remnant unless a significant fraction of the explosion energy has gone into the acceleration of cosmic ray ions . These measurements, along with evidence that the forward shock temperature in the young remnant 1E 0102.2-7219 is considerably lower than that expected from the measured expansion velocity (see equation 1 above, which requires $\gamma
\gg 5/3$ for the observed velocity to yield the temperature in this remnant), provide *dynamical* evidence for the acceleration of cosmic rays ions (which dominate the dynamics because of their large mass) by SNR shocks; despite the lack of a direct spectral signature (see below), these observations show that at least some SNRs accelerate ions, as well as electrons, to multi-TeV energies.

# Gamma-Ray Emission Mechanisms

At higher particle energies, $\gamma$-ray production may result from nonthermal bremsstrahlung of electrons colliding with ambient gas, from IC scattering of ambient photons, and from the decay of neutral pions created by the collision of energetic protons. Without additional constraints, it can be difficult to differentiate between these mechanisms based on $\gamma$-ray spectra alone. For example, although pion-induced emission leads to a broad spectrum that, in general, can be distinguished from the more sharply-peak IC spectrum from a cut-off electron population (see Figure 2 and the discussion below), high magnetic fields can produce a steepening of the electron spectrum over a wide energy range, which can broaden the IC spectrum and make it difficult to differentiate from a pion-decay spectrum . The relative contributions from the different $\gamma$-ray production mechanisms depend highly on ambient conditions, however, and these conditions also have significant effects on the emission in other spectral bands. For a strong pion-decay component, for example, a reasonably high ambient density is required to provide ample target material with which energetic protons from the SNR shock can collide. Such dense material should also lead to significant thermal X-ray emission. A strong IC component, on the other hand, should be accompanied by synchrotron emission in the radio and X-ray bands whose luminosity is consistent with expected values for the postshock magnetic field. In this sense, then, $\gamma$-ray measurements accompanied by spectra from other bands hold the best promise for interpreting the high energy emission – and, in particular, for providing spectral evidence of high energy ions.

ASCA observations of G347.3$-$`<!-- -->`{=html}0.5 (RX J1713.7$-$`<!-- -->`{=html}3946) revealed that the X-ray spectrum of this large-diameter SNR (Figure 2, left) is dominated by nonthermal emission , indicating the presence of electrons with energies in excess of $\sim 10$ TeV. Subsequent detection with HESS confirmed the presence of energetic particles, but the nature of the $\gamma$-ray emission remains unclear. The broadband spectrum of the remnant is shown in Figure 3 (right) along with models for the emission. The radio and X-ray emission is well-described by synchrotron radiation from a power-law distribution of particles accompanied by a high energy cutoff. The high energy spectrum can be described by a model of emission induced by both protons (through pion decay) and electrons (through IC scattering), shown as dashed curves in Figure 2. Here the pion component dominates , although the assumed ambient density ($n_0 = 1 {\rm\ cm}^{-2}$) is hard to reconcile with upper limits derived from the lack of observed thermal X-ray emission from the remnant .

A larger overall normalization for the electron spectrum is able to adequately reproduce the TeV emission from G347.5$-$`<!-- -->`{=html}0.5 without any pion contribution (dotted curve in Figure 3), but this overpredicts the synchrotron emission unless a small filling factor is assumed for the magnetic field . The GLAST sensitivity expected from all-sky survey mode exposures after 1 month and 1 year are indicated in Figure 2, and show that the pion component will either be detected or clearly ruled out within the first year of observations.

# GLAST and SNRs

The distribution of EGRET sources in the Galactic plane has considerable overlap with that of SNRs (Figure 3, left). Given the discussion above, this would seem reasonable; we *expect* to detect $\gamma$-rays from SNRs. However, the low angular resolution provided by EGRET results in large uncertainties for the source positions. The chance probability that some of these large error circles would overlap with the position of an SNR is relatively high, especially since the remnants themselves are extended. Moreover, the SNR population is closely linked to the distribution of pulsars and star-forming regions, both classes of which are potential $\gamma$-ray emitters.

The expected pion-decay flux from a SNR is approximately $$F(>100 {\rm MeV}) \approx 4.4  \times 10^7 \theta E_{51} d_{\rm kpc}^{-2}
n_0 {\rm\ phot\ cm}^{-2} {\rm\ s}^{-1}$$ where $\theta$ is a slow function of the SNR age, $E_{51}$ is the explosion energy in units of $10^{51}$ ergs, $d_{\rm kpc}$ is the distance in kpc, and $n_0$ is the preshock density . For reasonable values of these parameters, the predicted fluxes are very near the sensitivity limit of EGRET. While the above estimates are conservative in that any contributions from IC scattering and bremsstrahlung have been ignored, it remains clear that only SNRs in particularly dense regions would be expected to produce detectable emission. Torres et al. have investigated candidate associations of EGRET sources with SNRs for which there is some evidence for dense surrounding molecular material, and some of these associations are indeed plausible. But it is fair to say that there are no unambiguous detections of SNRs by EGRET.

This somewhat disconcerting view will receive a considerable boost from GLAST. In Figure 3 (right) we plot the estimated flux in the GLAST band as a function of SNR age, for a range of ambient densities. The solid horizontal line represents the LAT sensitivity for detection of a point source in one year of survey data. The sensitivity is a strong function of source extent; the hatched region shows a nominal factor of ten range in sensitivity to account for different SNR radii. While remnants in regions with densities typical of the ISM ($n_0 \sim 0.3 {\rm\ cm}^{-3}$), appear unlikely to be detectable, there is a wide range of densities typical of those found in regions of massive star formation that should yield detectable SNR emission. For these sources, the prognosis is quite good: GLAST observations will provide spectra that probe the important region of the $\gamma$-ray spectrum than can differentiate between pion-decay and IC mechanisms, thus making a significant contribution to our understanding of particle acceleration in SNRs.

# Summary

Several mechanisms exist by which SNRs may be associated with the emission of energetic $\gamma$-rays. Recent X-ray and TeV $\gamma$-ray observations have provided powerful evidence for the presence of high energy particles, and GLAST observations promise contributions in the crucial spectral range where pion-induced emission can potentially be discriminated from IC emission. The significant increase in angular resolution over that provided by EGRET will provide more confident associations between GLAST sources and SNRs, and the large collecting area will result in sufficient sensitivity to accurately determine spectra for comparison with broadband models. In addition, a significant increase in the discovery space of $\gamma$-ray emission from SNRs can be expected, allowing us to probe new associations and potentially resulting in GLAST detections that point the way to discoveries in other bands.


abstract: Recently the topic of *how to effectively offload cellular traffic onto device-to-device (D2D) sharing among users in proximity* has been gaining more and more attention of global researchers and engineers. Users utilize wireless short-range D2D communications for sharing contents locally, due to not only the rapid sharing experience and free cost, but also high accuracy on deliveries of interesting and popular contents, as well as strong social impacts among friends. Nevertheless, the existing related studies are mostly confined to small-scale datasets, limited dimensions of user features, or unrealistic assumptions and hypotheses on user behaviors. In this article, driven by emerging Big Data techniques, we propose to design a big data platform, named **D2D Big Data**, in order to encourage the wireless D2D communications among users effectively, to promote contents for providers accurately, and to carry out offloading intelligence for operators efficiently. We deploy a big data platform and further utilize a large-scale dataset (3.56 TBytes) from a popular D2D sharing application (APP), which contains 866 million D2D sharing activities on 4.5 million files disseminated via nearly 850 million users in 13 weeks. By abstracting and analyzing multi-dimensional features, including online behaviors, content properties, location relations, structural characteristics, meeting dynamics, social arborescence, privacy preservation policies and so on, we verify and evaluate the D2D Big Data platform regarding predictive content propagating coverage. Finally, we discuss challenges and opportunities regarding D2D Big Data and propose to unveil a promising upcoming future of wireless D2D communications.
author: Xiaofei Wang$^{1*}$, Yuhua Zhang$^{1}$, Victor C. M. Leung$^{2}$, Nadra Guizani$^{3}$, Tianpeng Jiang$^{4}$
${^1}$Tianjin Key Laboratory of Advanced Networking (TANK), School of Computer Science and Technology, Tianjin University, Tianjin, China.
${^2}$Department of Computer Engineering, University of British Columbia, Vancouver, Canada.
${^3}$Department of Electrical and Computer Engineering, Purdue University, USA.
${^4}$Beijing Anqi Zhilian Technology Co. Ltd., Beijing, China.
\*Prof. Xiaofei Wang is the corresponding author.
title: D2D Big Data: Content Deliveries over
  Wireless Device-to-Device Sharing in
  Large Scale Mobile Networks

# Introduction

With the increasing quantity and quality of multimedia services over mobile networks, there has been a huge increase in traffic over recent years . This has caused major hardships in effectively handling such traffic. This is mainly due to the lack of scalable mobile network operators (MNOs)' infrastructure and the congested wireless network.

A recent new trend is to encourage users to obtain interesting content files from the devices of other users around them. This can be carried out by caching and sharing files with the help of advanced device-to-device (D2D) communications, which is an effective method to largely offload the duplicate cellular downloads . D2D can thus significantly break the "bottle-neck" of the MNOs' infrastructured networks. Note that in this article, we consider various types of wireless D2D communication techniques between devices directly, even including the 3GPP D2D communication underlaying LTE networks.

Many studies, like , have shown that by mining the social and mobile behaviors of users, they are inclined to share popular content files via frequently offline encounters over D2D communications . However, all of the previous studies in the literature have focused only on one particular group of users (in the scale of hundreds or thousands), based on small-scale data analysis and algorithms design, which cannot exploit D2D for main-stream high-quality services rather than just a tool. The promising D2D framework for now should be significantly enhanced to be able to elaborate the huge potential of D2D deliveries over a large number of mobile users (in the scale of millions), in a wide area (cities or even countries), which motivates us to integrate state-of-the-art big data techniques to encourage smart D2D deliveries among users effectively, to promote contents for providers accurately, and to carry out offloading intelligence for operators efficiently. This is because there has been increasingly a vast amount of different types of data (e.g., activity logs, traffic logs, profiles of users and contents, etc.) not yet discovered and fully utilized.

The emerging big data techniques pose many new unprecedented opportunities and challenges to conventional data analytics over groups of mobile users for exploiting the capacity of D2D sharings because of the significant dimensionality, heterogeneity, and complex features therein, e.g., that is, *Volume, Variety, Velocity, Value and Veracity* . Hence it is crucial to shift our attention from single group-based research to a big data-based framework for D2D, as illustrated in Fig. . First, the D2D platform can easily support to adopt general statistical measurement algorithms on the large scale user base (e.g., millions of people) in order to grasp a common understanding over the analysis results. In fact, the utilization of **D2D Big Data** highly depends on the exploration of multidimensional features extracted from the complex mobile user behaviors, e.g., geographical homophily of clustered users, social relationships, interest similarity and mobility models, and the complicated content properties, e.g., content size, categories, and time-varying popularity. Particularly, the sharing activities among mobile users can formulate a connected graph so that D2D Big Data framework can apply advanced theories and algorithms based on complex network theory. However, the integration of D2D and big data has not been discussed and analyzed towards a significant volume of real data collected from practical D2D sharing applications in mobile networks. Although the online virtual communities of social network services, in which people with a shared interest, such as a specific hobby or activity, can interact and socialize with each other, have been researched thoroughly, the offline realistic people-to-people (i.e., D2D) content-based interactions can be further socialized and opportunistically connected toward a promising future with the assistance of big data techniques.

Therefore, the integration of wireless D2D communications with big data techniques can extend the limited applications in practice, leading to a new paradigm of offline user-to-user sharing. And thus, it is necessary to deploy a platform based on big data techniques (e.g., Hadoop, Spark, MLlib etc.) for carrying out comprehensive measurement and analytics based on *large-scale* real-world D2D sharing activities, in order to enhance the functionality and capability of D2D Big Data and to improve the quality of D2D services finally.

To the best of our knowledge, we are the first group to study big data-based large-scale offline D2D content dissemination service in mobile networks. Our contribution can be summarized as follows: 1) we propose an advanced framework of D2D Big Data platform over cloud-based systems; 2) we discuss potential system and modules for D2D Big Data framework to strongly encourage D2D sharing; 3) we provide valuable measurement, analytics, learning, prediction and recommendation results; and 4) we discuss useful implications and guidelines for exploiting D2D Big Data for enriching offline services. In addition, we also discuss opportunities and challenges to hopefully unveil the upcoming future of wireless D2D communications. g coverage percentage.

# D2D Big Data Framework

The original objective of utilizing D2D has been just to query neighboring peers for obtaining desired content files, and also to broadcast interesting or urgent information among mobile users. Extending the scope of existing D2D studies in Delay-tolerant Networks (DTNs), Opportunistic Networks (ONs) and Mobile Social Networks (MSNs), which are mostly designed for offloading and sharing based on small number of users, we propose to design a D2D Big Data framework for large-scale user base. It is useful in facilitating the functionality of D2D sharing tools to level up towards an enhanced offline content dissemination service and a mobile interest exploitation platform, which can serve a significant amount of mobile users over a large area (even nation).

From the perspectives of mobile users, operators and content providers, as shown in Fig. , D2D Big Data consists of 3 parts:

-   **Content Promotion Big Data (*Cloud of Promoting*)**: It is a platform open to customers who want to promote content effectively and rapidly. The main functions are content uploading, budgeting, promotion supervising and strategy visualizing. The target is to help the content provider make better promoting strategies and popularize content by D2D deliveries in a short time. Additionally, multidimensional features involving users' interests, relationship, user impacts, social groups will be analyzed to generate a proper strategy for content providers to make their promotion decisions.

-   **Interest Exploitation Big Data (*Cloud of Interests*)**: There is an interest mining and analyzing platform that focuses on mining users' preferences to realize the functions of interest recommendation, discovery the interest of nearby users, establishing groups and obtaining interesting contents. The interest exploitation functions are realized by extracting features of users' interactive traces, their preferences, the sharing habits and content popularity to perform customized and accurate recommendation.

-   **D2D Analysis Big Data (*Cloud of Analysis*)**: As the core of the scheme, the D2D Big Data analysis platform is built to process and analyze any data sets and to gain meaningful results that the content providers need. The main part is a series of algorithms and frameworks of measuring and extracting multi-features from the traces, including time-varying online behaviors, location relations and similarity, content properties and users' preference entropy, characteristics of social groups, meeting dynamics, friendship tree analysis etc. In addition, they are implemented by the Spark-based big data framework to ensure the high performance and accurate prediction.

The industrial vision of D2D Big Data is to fix the problems of overloaded online Internet content, which have been severely confusing users' demands and wasting large amounts of marketing costs with low efficiency and precision instead. Hence to offload valuable content into offline "face-to-face" (i.e., D2D) deliveries due to the truth of word of mouth effect , are much more accurate for satisfying mobile users. And from the academic perspective on D2D Big Data, the objective is firstly to effectively process large scale data set generated by millions of users; secondly, it is not only to efficiently harmonize modules of machine learning, graph computing, complex network theory etc., but also to further invent novel theories and techniques. Therefore, it is desired to deploy the D2D Big Data framework carefully for satisfying the needs, and we will elaborate details in following sections.

# Measurement and Analytics over Realistic Data and System

The main architecture of D2D Big Data framework is illustrated in Fig. . There are two essential conceptual components, the input of large scale **data set** including user profiles, content statistics, activity logs, traffic records etc. While users move and share content files, the **system** carries out streaming-based and batch-based processing. Prediction and recommendation are made by learning-based tools involving machine learning, unsupervised learning and collaborative group recommendation algorithms to realize content promotion and friendship recommendation. Meanwhile, analytics and findings can further contribute to the mobile APP marketing and developing strategies.

A robust data processing platform based on our server clusters and Spark system is built. To solve the storage problem of large data sets and processed results, Hadoop Distributed File System (HDFS) is implemented to ensure the scalability and robustness of the system. Furthermore, the in-memory computing platform of Spark is deployed to improve the big data processing ability of the framework. We used Python scripts along with Anaconda scientific package and graph-tool library (v2.18), Hadoop, Spark, MLlib, GraphX, Spark Streaming and batch-based data processing tools like Kafka, to support the analytics and computing of the large data. All tasks are supported by a cluster of 24 servers, each of which has two 2620v4 CPUs, 64G or 96G RAM and 6T SAS hard disk.

However, we do not deploy big data system on the servers directly, but firstly implement OpenStack over the servers, and then D2D Big Data processing tools and software packages can be set up on the virtual machines. We use OpenStack for a cloud computing infrastructure, because it is an open source cloud computing software with functions of building up the cloud operating system that cloud control large pools of compute, storage, and networking resources throughout a datacenter in the IaaS layer. The virtual machines with the widely used big data processing softwares including Spark, Hadoop and Hive, thus can be elastically created and used through OpenStack. This way, the platform can enjoy the elasticity and the optimality of the consolidated cloud resource all the time.

For testing and evaluating D2D Big Data framework, we capture and import a large scale data set from *Xender*, which is a content delivery mobile application, serving millions of users all over the world to share content files via smart phones with ease. The service has been increasingly supported more than 100 million monthly active users and approximately 110 million content delivery activities per day. We measured the data sets from 01/08/2016 to 29/10/2016, 13 weeks in total, and a data set of approximate total size of 3.56 TBytes. We claim that the statistical information of the service and trace in this article can be only used for research purposes as it does not exactly reflect a realistic business scale, because while capturing the trace, we already filtered out a large number of incomplete or invalid records. More details of the data set and system are shown in Table . Note that the scale of the data set and system in our experimental lab are actually not so large compared with industrial big data systems run by big companies. But we believe they are enough to ensure high computing performance, memory capacity and robust performance for digging potential capacity of D2D.

| **Dataset Details**         |                         |
|:----------------------------|:------------------------|
| Data Periods                | 2016.08.01 - 2016.10.29 |
| Raw Dataset Size            | Around 3.56 TBytes      |
| Cleaned Dataset Size        | Around 96 GBytes        |
| Total Num. of Users         | 854 millions            |
| Num. of Involved Users      | 24 millions             |
| Total Num. of Activities    | 8.24 billions           |
| Num. of Involved Activities | 886 millions            |
| Total Num. of Files         | 6.22 billions           |
| Num. of Involved Files      | 4.45 millions           |
| Num. of GPS Records         | 4.19 millions           |
| **Platform Details**        |                         |
| Server Nodes                | 24                      |
| CPU model                   | E2620v4                 |
| Core Frequency              | 2.1 Ghz                 |
| Memory per Node             | 96 G                    |
| Storage per Node            | 6 TB                    |

Details of the Social D2D Big Data Trace and Platform

Multi-dimensional features, including online behaviors of users, location relations, meeting dynamics, content properties, social characteristics, popularity trends and so on, are extracted and mined by the *Cloud of Analysis* to exploit the properties of social graphs, user behaviors and relationships. User preferences and content properties are then analyzed by the *Cloud of Interests* and *Cloud of Promoting*, respectively. We can step to advanced utilization and exploitation over the potential of offline D2D deliveries. For example, incentive-based promotion, push over most influential users, interest exploration and so on are mostly yet not discovered well.

# Results of Measurement, Analysis, and Prediction by D2D Big Data Platform

Over the proposed D2D Big Data platform, there have been running out more and more results regarding general statistical measurement, analytics on users and contents. We explore the social graph properties by complex network theory, temporal and spatial user relationships and so on . Due to limited space in this article, we cannot show all of our recent results but several interesting research items regarding predictive content propagation based on D2D Big Data.

From Fig. , we can see that a conventional cellular-based sharing activity can be changed by wireless D2D communication where the BS is not necessarily needed. When many mobile users share files via D2D communications, as time goes on, the wireless links among users formulate "virtual" connectivities that transform the sharing records into an offline social network, which user behavior pattern and social properties of offline MSNs are quite similar to online social networks. Due to the temporal and spatial constraints of D2D deliveries, which are much difficult to carry out compared to online sharing activities, the graph in the D2D social network is not strongly connected but a bit sparse, and our strategy is to split just by the physical encounters, i.e., if two users have ever shared at least one content file via D2D, they are in the same group. And thus we can approximately split the large user base into multiple groups, in which members are tightly connected. As shown in Fig. , the sizes of the groups follow the power law exactly, which means there are few groups with large sizes, but most of the groups have small sizes. This provides us with the space of emphasizing on some system optimization as well as group recommendation policies. Also we are further applying in-depth social network analysis based on complex network theory, and try to find out the properties of user groups, for example, global cluster and local cluster, average path length and diameters, which are illustrated inside Fig. as well. For instance, the diameters of those social groups in D2D sharing are around 6 to 10, which are quite larger than online social network services (SNSs), indicating that the sharing by D2D happens not as easy as that in online SNSs. Instead, D2D activities take place with strict conditions on space and time, as well as very strong demands among users.

The key function driven by D2D Big Data must be the accurate prediction of user activities. This way, platform resource can be allocated in prior to the practically happening, and also we can show proper recommendation system to improve the quality of service. We consider several important features, including time-varying online behaviors and customs, content properties and user preference entropy, meeting dynamics and mobility models, location relations and trajectory similarity, characteristics of social groups by complex networks, social arborescence and friendship tree analysis as well as privacy preserving and sharing security as shown in Fig. . The privacy preserving and sharing security here can be realized by setting different permission levels to users according to their relationships. Only users with the same permission levels like family or friends can access to others' content files and hence the privacy is ensured. We are trying to carry out nowcasting experiments on D2D sharing activities, based on SVM and MLlib, by given several happened factors . As shown in Fig. , multiple features are extracted from the empirical data (e.g. 6 weeks' data) to train the learning tools of SVM or Spark Mlib. And we nowcast the potential D2D sharing activities among users with test data (e.g. 7 weeks' data). For example, when a user who likes playing free mobile games is taking a rest at a cafeteria, we can analyze and predict a potential D2D sharing of the user with nearby users with common interests, and thus utilize push service for encouraging an D2D sharing activity with highest probability. Results in Fig. show that with different combinations of the features, the prediction accuracy varies. From the perspective of efficiency, when we desire higher prediction rate of forecasting by multiple features, we can selectively analyze 2 or 3 features mentioned above, not all of them. There is a significant tradeoff between features put into the model.

We can easily obtain a number of statistical measurement results time by time. One interesting finding is that, as shown in Fig. , there is huge amount reduplicate transmissions among millions of users. Particularly APPs and videos have very high percentage of reduplicate traffic, which indicates the skewed content popularity of the two types, It may motivate us to apply related prioritization algorithms for related resource optimization, like CDN deployment and content recommendation strategies.

Another key function of D2D Big Data is to find the most influential users within a large number of mobile users, considering various properties of the users, content files, groups, and so on. We have designed algorithms to discover the seeding users by finding the main roots of the tree structures in the social groups of the large user base from the first half of the trace. Fig. is the results of the evaluation on the efficiency of propagating contents in 100 randomly selected groups as representatives, and note that we only find the most influencing seed for each group to spread the content to others, by assuming that two users may share the content once they practically encounter in the second half of the trace. It is clearly shown that in the groups, nearly a half of the mobile users can get the content via offline D2D sharings, and it appears similar patterns of spreading popular contents rapidly boosted by seeding users' recommendations in online SNSs .

# Opportunities and Challenges

We hereby discuss a few essential promising research directions of D2D Big Data, and highlight several pending problems from the perspective of elaborating and widening the usage of D2D techniques.

## Privacy, Security and Energy Issues for D2D Big Data

Despite the tremendous benefits of D2D deliveries and the attached values of Big Data-driven intelligence, D2D still has many security and privacy challenges that are not yet well solved. There have been many studies on privacy and security issues over DTNs/ONs/MSNs, like and . Nonetheless, the problems may be enlarged seriously when the platform grows to a huge one for nation-wide or even world-wide MNOs, big content provider companies, and millions of mobile users. Offloading content traffic onto D2D with big data techniques should provide a fair mechanism for appropriately negotiating the cost, profit, and quality of service among all the involved entities. Particularly, the effective balance between the privacy protection and freedom of sharing in mobile devices should be dug harder, and also the tradeoff between passive transparent sharing or pull-based active sharing via D2D should be discussed with more experiments and user-centric trials. The achievement of a "triple-win" for the safe and secure D2D industry benefiting content providers, users and the platform developers will be complicated, and the success highly relies on innovative privacy schemes, copyright mechanism, and security protections for D2D Big Data for a large-scale user base, which are still undiscovered. One more critical issue of D2D sharing is the precious energy of mobile devices , and hence how to utilize the emerging energy harvesting techniques for D2D communications becomes interesting but challenging.

## Big Data-assisted Base Station Caching Integrated with D2D

There have been a number of studies towards the Mobile Edge Computing, Content-Centric Networks and so on, which mostly highly rely on the caching and offloading capability on the base stations (BSs) . BSs' edge cache can formulate a collaborative "buffer" to significantly optimize the mobile applications by the last hop. There have been a few proposed techniques and systems for integrating edge caching functionality into MNO's network to analyze and facilitate the mobile services; however, it is still not discussed or even tested to combining BS caching with D2D communications together for a more pervasive and more universal caching space around mobile users. We are faced with interesting but challenging research issues, like how to optimally allocate the networking and caching resource among BSs and smart devices, and how to even deploy big data techniques to analyze and disseminate contents to a nation-wide coverage of mobile users with the best performance.

## Business Models for D2D Big Data

There has been an explosive growth of the online peer-to-peer (P2P) techniques since the beginning of 21th centaury, along with the growth of Internet topology and the number of Internet hosts, which has enabled people to collaborate for sharing contents online massively in a distributed manner. For now, can we expect a second round of P2P, the thriving offline D2D deliveries via face-to-face encounters among people? One motivation for mobile users to share content with each other is the common-interest, but beyond that D2D Big Data should carry out fine-grained incentive-based forwarding mechanism to push the content promotion activities naturally. For example, while there are so many mobile application (APP) markets online, but can we distribute most of the free APP files (i.e., APK files for Android) via user-to-user sharing . Also can we deliver music and video files to proper audiences, while each user is both a viewer and a marketing participant? More business models over D2D Big Data should be explored in the near future.

# Conclusion

In this article, we discussed the potential of utilizing wireless Device-to-device (D2D) communications to offload mobile traffic and to disseminate contents among mobile users assisted by the integration of big data techniques. We discussed the D2D Big Data framework, which can encourage the D2D deliveries among users effectively, promote contents for providers accurately, and carry out offloading intelligence for operators efficiently. We implemented a realistic experimental D2D Big Data platform, and exploited a significant volume of real data set. Based on the measurement, analysis and evaluation, we exploited and discussed important implications, under-going techniques, opportunities and challenges to improve quality of D2D sharing services, and to enhance the functionality and capability of D2D Big Data. For our future work, we concentrate on not only more satisfactory group recommendation techniques and more accurate prediction of content popularity and user behaviors, but also the consolidated multi-disciplinary collaboration of D2D Big Data with MNOs' wireless systems, industrial vendors, content service providers, and governments' copyright policies.

# Acknowledgement

This work was supported by National Natural Science Foundation of China (No. 61702364)