\section{Introduction and overview}

\IEEEPARstart{S}{hannon's} information theory \cite{shannon1948mathematical} answers two fundamental questions in digital communication theory \cite{cover1991elements}: the ultimate data compression rate (answer: the entropy $H$) and the ultimate reliable transmission rate of communication (answer: the channel capacity $C$). It addresses technical problems in communication systems, enabling it to receive the same symbols as the sender. However, with the ever-increasing demand for intelligent wireless communications, the communication architecture is evolving from only focusing on technical level to intelligent interconnection of everything	\cite{shi2021semantic}. Weaver \cite{weaver1953recent} categorized communications into three levels about 70 years ago:

\textbf{Level A.} How accurately can the symbols of communication be transmitted? (The technical problem.)

\textbf{Level B.} How precisely do the transmitted symbols convey the desired meaning? (The semantic problem.)

\textbf{Level C.} How effectively does the received meaning affect conduct in the desired way? (The effectiveness problem.)

Recently, semantic information theory including semantic communications has attracted much attention. One reasonable and feasible measure of semantic messages is still an open problem, which may be the greatest challenge for the new developments of semantic communication (SC) systems. On the other hand, currently in 6G networks, intelligent interconnections of everything will bring the conventional paradigm to the communication mode with higher requirements of semantic interaction. In this regard, finding a way to quickly reflect the semantic processing of messages from the sender and the receiver will become one more promising pathway to the 6G intelligent networking systems.

This paper proposes a new communication system based on information framework expansion and knowledge collision, by extending Shannon's theory of communication (Level A) to a theory of semantic communication (Level B). Our work is initially influenced by Jinho and Jihong \cite{choi2022semantic}, with new contributions in the following:
\begin{itemize}
	\item We generalize the work of Jinho and Jihong from single fixed semantic type to semantic dynamically scaling up mode, referred to as expansion, and get the idea of knowledge collision, which can reflect the asynchronous knowledge update process of the sender and the receiver in some degree. In addition, we also take into account the effect of channel noise on the model.
	\item We present a new measure related to the semantic communication system based on semantic expansion and knowledge collision, called Measure of Comprehension and Interpretation, which can measure the semantic entropy of discrete sources.
	\item We discuss the additional gains from expansion and demonstrate that knowledge matching of its asynchronous scaling up plays a key role in semantic communications.
\end{itemize}

As a primary work, the system proposed in this paper only focuses on the discrete cases and makes some drastic simplifications. Moreover, we are not concerned with the effective problem (Lever C), which is beyond the scope of this paper. However, we believe that these simplifications are necessary for us to focus on the "core" issues, and that even this model may form a foundation for a general semantic information theory. To best of our knowledge, this is the first study to explore the semantic expansion and asynchronous scaling up of semantic knowledge. The insights gained from this work may be of assistance to the future intelligent semantic communication system and 6G.

The rest of this paper is organized as follows. Section \ref{sec:related work} introduces the related work of semantic communications. It summarizes key concepts and conclusions of Shannon's information theory and semantic information theory. In Section \ref{sec:models}, we present the generalized model of semantic communication from two aspects, one is trying to setup a close relationship with Shannon communication model, and the other is trying to find the feasible modification of the model so that it can exactly reflect the quickly surging of semantic communications from the user requirements. In Section \ref{sec:experiment}, we lay out the implementation details of the simulation and discuss the experimental results. Finally, we conclude the paper in Section \ref{sec:conclusion}.

\section{Related Work}
\label{sec:related work}

In this section, we introduce the framework of semantic information theory, especially the key concept, semantic entropy. It will then go on to the work that we are concerned with, semantic communication as a signaling game with correlated knowledge bases.

\IEEEpubidadjcol

\subsection{Preliminaries}

Although Shannon's information did not address the semantic problem of communications, it provided important insights on the message processing techniques associated with the focus of both the sender and the receiver's attention. Thus, in this subsection, we briefly introduce the main concepts and theorems in Shannon's information theory.

{\bf{Entropy.}} It is a measure of the uncertainty of a random variable \cite{cover1991elements}. The \emph{entropy} $H(X)$ of a discrete random variable $X$ with probability distribution $p(x)$, is defined as
\begin{equation}
	H(X) = -\sum_{x\in \mathcal{X}} p(x) \log p(x).
\end{equation}

{\bf{Mutual information.}} It is a measure of the amount of information that one random variable contains about another. Mutual information can be seen as the reduction in the uncertainty of one random variable due to the knowledge of the other. Consider two random variables $X$ and $Y$ with a joint probability mass function $p(x,y)$ and marginal probability mass functions $p(x)$ and $p(y)$, the mutual information between $X$ and $Y$ is defined as
\begin{equation}
	I(X;Y) = \sum_{x\in \mathcal{X}} \sum_{y \in \mathcal{Y}}  p(x,y) {p(x,y) \over p(x)p(y)}.
\end{equation}

{\bf{Channel capacity.}} The channel capacity is the maximum of mutual information with given the conditional transit probability from $X$ to $Y$, $p(y|x)$. It is defined by
\begin{equation}
	C = \max_{p(x)} I(X;Y),
\end{equation}
where $X$ and $Y$ are the input and output of the channel, respectively.

{\bf{Source coding theorem.}} As $N\to \infty$, $N$ i.i.d. (independent identically distribution) random variables with entropy $H(X)$ can be compressed into more than $NH(X)$ bits to represent them completely, that is, the information loss can be negligible in the viewpoint of Shannon's information theory. Conversely, if they are compressed into fewer than $NH(X)$ bits to represent them, there will be a loss of information certainly.

{\bf{Channel coding theorem.}} For a discrete memoryless channel, all rates below capacity $C$ are achievable. Conversely, any sequence of codes with negligible error must have obey the rule, its transmission rate is not greater than the channel capacity, $R \leq C$.

\subsection{Semantic Information Theory}

Carnap and Bar-Hillel \cite{carnap1952outline} were the first to propose the concept of semantic information theory, using logical probability rather than statistical probability to measure the semantic entropy of a sentence. It's necessary here to clarify exactly that we use $p$ and $m$ to denote statistical probability and logical probability in this paper, respectively. The logical probability of a sentence is measured by the likelihood that the sentence is true in all possible situations \cite{bao2011towards}. Then, the semantic information of the message $e$ is defined as
 \begin{equation}
	H_S(e) = -\log_2(m(e)),
\end{equation}
where $m(e)$ is the logical probability of $e$. However, this metric led to a paradox that any fact has an infinite amount of information when it contradicts itself, i.e., $H_S(e\wedge \neg e ) = \infty$. Floridi \cite{floridi2004outline} solved the paradox in Carnap and Bar-Hillel's proposal \cite{carnap1952outline}, adopting the relative distance of semantics to measure the amount of information.

Bao \emph{et al.} \cite{bao2011towards} defined semantic entropy of a message $x$ as
\begin{equation}
	\label{Eq:sc entropy -1}
	H_S(x) = -\log_2(m(x)),
\end{equation}
where the logical probability of $x$ is given by
\begin{equation}
	m(x) = {\mu(W_x) \over \mu(W)}= {\sum\limits_{w \in W,w \models x} \mu (w) \over \sum\limits_{w \in W}\mu(w)}
\end{equation}
Here, $W$ is the symbol set of a source, $\models$ is the proposition satisfaction relation, and $W_x$ is the set of models for $x$. In addition, $\mu$ is a probability measure, $\sum\limits_{w\in W}\mu(w)=1$.

Besides logical probability, there are some definitions of semantic entropy based on different backgrounds \cite{qin2021semantic}\cite{shi2021new}. D'Alfonso \cite{d2011quantifying} utilized the notion of truthlikeness to quantify semantic information. Kolchinsky and Wolpert \cite{kolchinsky2018semantic} defined semantic entropy as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. Kountouris and Pappas \cite{kountouris2021semantics} advocated for assessing and extracting the semantic value of data at three different granularity levels:  microscopic scale, mesoscopic scale and macroscopic scale.

Analogous to Shannon's information theory, some related theories at the semantic level, such as semantic channel capacity, semantic rate distortion and information bottleneck \cite{tucker2022towards}, are also being explored. Based on (\ref{Eq:sc entropy -1}), Bao \emph{et al.} further proposed \emph{semantic channel coding theorem} \cite{bao2011towards}. For every discrete memoryless channel, the channel capacity
\begin{equation}
	C_S = \sup_{P(X|W)}\{ I(X;Y)-H_{K_S,I_S} (W|X)+\overline{H_{S;K_r,I_r}(Y)} \},
\end{equation}
has the following property: For any $\epsilon>0$ and $R \leq C_S$, there is a block coding strategy such that the maximal probability of semantic error is $\leq \epsilon$. $I(X;Y)$ is the mutual information between the input $X$ of the channel and the output $Y$ of the channel, $H_{K_S,I_S} (W|X)$ is the equivocation of the semantic encoder, given the sender's local knowledge $K_S$ and inference procedure $I_S$. $\overline{H_{S;K_r,I_r}(Y)}=\sum_y p(y)H_s(y)$ is the average information of received messages, given the receiver's local knowledge $K_S$ and inference procedure $I_S$.

Moreover, H. Vincent Poor \cite{liu2021rate} formulated the rate-distortion in semantic communication as
\begin{equation}
	R(D_s,D_w) = \min I(W;\hat{X},\hat{W})
\end{equation}
where $D_s$ is the semantic distortion between source, $X$, and recovered information, $\hat{X}$, at the receiver. $D_w$ is the distortion between semantic representation, $W$, and received semantic representation, $\hat{W}$.

In recent years, the fusion of semantic communication algorithms and learning theory \cite{weng2021semantic,huang2021deep,zhou2021semantic,zhang2022deep} is also driving the continuous development of the communication architecture. Although these explorations did not fully open the door to semantic communication, they provided us important insights to move forward theoretically.

\subsection{Semantic Communication as a Lewis Signaling Game with Knowledge Bases}
\label{sec:sc}

Jinho and Jihong \cite{choi2022semantic} proposed a SC model based on the Lewis signaling game, which derived some interesting results. This work really gives us some motivations to conduct this work in some degree. Further, we make a generalization of this model with adding the asynchronous knowledge scaling up updates of the sender and the receiver, which may close to the real semantic communications in the era of intelligent communication. Thus, we briefly re-describe it.

Suppose there is a semantic communication system where Alice is the sender and Bob is the receiver. Let $T \in \mathcal {T}$ denote the semantic type that includes semantic information or messages. Alice wishes to transmit $T$ to Bob by sending a signal $S\in \mathcal{S}$, and Bob chooses its response $R \in \mathcal{R}$. Both Alice and Bob utilize their local knowledge bases $\mathcal{K}_A$ and $\mathcal{K}_B$, respectively. The semantic architecture can be described as

\begin{align*}
	&K_A~~~~~~~~~~~~K_B \\
	&\downarrow ~~~~~~~~~~~~~~\downarrow \\
	&T~\rightarrow~S~\rightarrow~R~(=\hat{T}) .
\end{align*}
If $R=T$, one communication process can be successful. Moreover, the reward for one communication is defined as:
\begin{equation}
	u=\left\{
	\begin{aligned}
		&1, \quad \text{if }R=T\text{;}\\
		&0, \quad \text{otherwise.}\\
	\end{aligned}
	\right
	.
	\label{Eq:payoff}
\end{equation}
Our object is to maximize the average reward. We use \emph{the success rate of semantic agreement (SRSA)} to represent it.

The knowledge base can be regarded as the side information. Alice has her knowledge base $\mathcal{K}_A$. The instance of Alice's knowledge base at each time is $K_A \in \mathcal{K}_A$, which affects the generation of semantic types. On the other hand, Bob has his knowledge base $\mathcal{K}_B$ and its instance is $K_B \in \mathcal{K}_B$. $K_B$ is the "core", used to infer the intended message together with the received signal.

Based on these foundations, Jinho and Jihong \cite{choi2022semantic} derived some interesting results. On the one hand, the similarity of the two knowledge instances plays a crucial role in SC. With a limited bandwidth, the communication quality is regulated by $I(K_A;K_B|S)$. On the other hand, the SRSA is dependent on the similarity of the two communication parties' knowledge. However, it did not clearly setup the close relationship with Shannon's communication model. In this work, we will make a generalization of the model by adding the asynchronous knowledge scaling up updates of the sender and the receiver, which may closely reflect the quickly surging of semantic communications in many scenarios of applications.

\section{Semantic Communication Models}
\label{sec:models}
In this section, we will present the generalized model of semantic communication from two different aspects, one is trying to setup a close relationship with Shannon communication model, and the other is trying to find the feasible modification of the model so that it can exactly reflect the quickly surging of semantic communications from the user requirements. In this regard, we first setup a basic model, and then to extend it by adding the knowledge base update functional mode with expansion and collision.

\subsection{Motivation}
Shannon's communication system is concerned with the accurate transmission of symbols over channels. However, it does not take into account the differences in the knowledge backgrounds of two parties, which affect whether the communication can convey the desired meaning. In other words, the matching of knowledge is also another special channel, although it does not actually exist. We want to fill this gap and propose an intelligent communication system with coexistence of the real channel and virtual channel, making information theory more complete. Thus, we consider not only the physical channel, but also the transfer of knowledge between two parties. Moreover, the knowledge of both parties is also evolving, so we take into account the asynchronous knowledge scaling up updates and semantic expansions.

\subsection{Basic Model}
\label{sec:model1}

Let us first observe the \emph{channel}, which is the physical medium of  message exchange, and play the key role in the model setup of  a complete semantic communication system. A \emph{communication channel} is a system in which the output depends on probabilistically on its input \cite{cover1991elements}. For simplification, we still use Alice and Bob as the two parties	participating in the communication. Suppose Alice and Bob are at opposite ends of a memoryless channel, which can transmit physical signals. The channel is said to be memoryless if the probability distribution of the output depends only on the input at that time and is conditionally independent of previous inputs or outputs. That is, the output of the channel is only related to the input at the current moment and no any feedback processing is considered here..

We use $S$ and $\hat{S}$ to represent the input and output of the channel. Under the interference of the noise, the signal $S$ becomes $\hat{S}$ through channel transmission. Specifically, Alice observes the input $S$ of the channel, and Bob observes the output $\hat{S}$. Let $K_A$ and $K_B$ represent the knowledge instance possessed by Alice and Bob, respectively. On the one hand, Alice encodes $S$ as $T$ with $K_A$. This means that Alice uses the knowledge instance $K_A$ to process the signal $S$, resulting in the semantic type $T$, i.e., $(S,K_A) \to T$. On the other hand, Bob decodes $\hat{S}$ into the response $R$ with $K_B$, which can be expressed as $(\hat{S},K_B \to R)$. The architecture of this process can be expressed as

\begin{equation}\label{Eq:structure}
	\begin{split}
		T~\leftarrow~ &~S~~~\mathop{\longrightarrow}\limits^{(a)}~~~\hat{S}~\rightarrow ~R~(=\hat{T}) \\
		&~\uparrow~~~~~~~~~~\uparrow \\
		&~K_A~~\mathop{\dashrightarrow}\limits^{(b)}~~K_B
	\end{split}
\end{equation}

We denote the process $(S,K_A) \to T$ as semantic encoding and $(\hat{S},K_B)\to \hat{T}$ as semantic decoding. They indicate the understanding of meaning at the semantic level by both parties.

The definitions of symbols are summarized as:
\begin{itemize}
	\item \emph{Semantic Types}: $T=t_k,k=1,\ldots,|\mathcal{T}|$, is a random variable that is generated by Alice.
	\item \emph{Signals}: $S=s_l, l=1,\ldots,|\mathcal{S}|$, is a signal that Alice sends to Bob.
	\item \emph{Responses}: $R=r_n,n=1,\ldots,|\mathcal{R}|$, is a response that Bob chooses.
	\item \emph{Knowledge instances}: $K_A \in \mathcal{K}_A$ and $K_B \in \mathcal{K}_B$ represent the knowledge instance used by Alice and Bob when semantic encoding and decoding, respectively.
\end{itemize}

\begin{lemma}
	\label{lemma:1}
	Under the knowledge instance $K_A$ and $K_B$, the mutual information between $T$ and $\hat{T}$ is given by
	\begin{equation}
		\label{Eq:Lemma 1}
		I(T;\hat{T}) = I(S;\hat{S}) + I(K_A;K_B|S, \hat{S}) + I(S;K_B|\hat{S})+ I(\hat{S};K_A|S)
	\end{equation}
where $I(K_A;K_B|S,\hat{S})$, $I(S;K_B|\hat{S})$ and $I(\hat{S};K_A|S)$ represent the
conditional mutual information.
\end{lemma}
\begin{proof}
	Since $(S,K_A)\to T$ and $(\hat{S},K_B) \to \hat{T}$, we have $H(T)=H(K_A,S)$ and $H(\hat{T})=H(K_B,\hat{S})$. By the characteristics of entropy and mutual information, it follows that
\begin{align}
	I(T;\hat{T}) & = H(T)-H(T|\hat{T}) \\
	             &= H(K_A,S)- H(K_A,S|K_B,\hat{S}) \\
	             &= H(S) + H(K_A|S)-H(S|K_B,\hat{S})-H(K_A|K_B,S,\hat{S}) \\
	             &= H(S)-H(S|K_B,\hat{S})+H(K_A|S)-H(K_A|K_B,S,\hat{S}) \\
	             &= I(S;K_B,\hat{S}) + I(K_A;K_B,\hat{S}|S) \\
	             &= I(S;\hat{S}) + I(K_A;K_B|S, \hat{S}) + I(S;K_B|\hat{S})+ I(\hat{S};K_A|S),
\end{align}
which completes the proof.
\end{proof}

The mutual information between $T$ and $\hat{T}$ reflects the effectiveness of communication. It indicates the highest rate in bits per channel use at which information can be sent with arbitrarily low probability of error. We note that $I(T;\hat{T})$ consists of three terms:

\begin{enumerate}
	\item $I(S;\hat{S})$, the mutual information between the input and output of the channel. It corresponds to the \emph{channel capacity} in Shannon's information theory.
	\item $I(K_A;K_B|S,\hat{S})$, the mutual information between $K_A$ and $K_B$ given $S$ and $\hat{S}$. It indicates the amount of information that two knowledge instances contain about each other when signals are known.
	\item $I(S;K_B|\hat{S})+I(\hat{S};K_A|S)$, the conditional mutual information between $S$ and $K_B$, $\hat{S}$ and $K_A$.
\end{enumerate}

From Eq. (\ref{Eq:Lemma 1}), we know that if Bob wants to fully understand what Alice means, the communication process needs to meet two conditions, one is the accuracy of the symbols during channel transmission, and the other is the matching degree of the knowledge bases of both parties. In other words, i) The transmission $S\to\hat{S}$  should be reliable. It indicates the effect of channel noise on the signal. Moreover, the carrier of this process actually exists, we call it an \emph{explicit channel}. ii) The two knowledge instances should be similar. Although there is no actual transmission between $K_A$ and $K_B$, we assume that there is a virtual channel that reflects the probabilistic relationship between knowledge instances, which we call an \emph{implicit channel}. The \emph{explicit channel} and \emph{implicit channel} together form the transmission medium of a communication system. Specifically, the characteristics of the explicit channel determine the value of the first term in Eq. (\ref{Eq:Lemma 1}), and the implicit channel determines the second term. In addition, the last term is affected by both explicit and implicit channels.

If the communication system has only the explicit channel, it degenerates to the Shannon case, and $I(T;\hat{T})=I(S;\hat{S})$. Furthermore, We discuss three special cases.

\begin{enumerate}[I.]
	\item{\textbf{The explicit channel is noiseless, i.e., \bm{$S=\hat{S}$}.}
	In this setting, the formula (\ref{Eq:Lemma 1}) can be simplified to
	\begin{equation}
		\label{Eq:speical case 1}
		I(T;\hat{T}) = H(S) + I(K_A;K_B|S),
	\end{equation}
	which is consistent with the conclusion in \cite{choi2022semantic}. The mutual information between $T$ and $\hat{T}$ is subjected to the entropy of the signal $S$ and the conditional mutual information between $K_A$ and $K_B$ given $S$.

    }
	\item{\textbf{The implicit channel is noiseless, i.e., \bm{$K_A=K_B$}.} This means that Alice and Bob have the same knowledge instance, so the communication performance will not be affected by the difference in the background of both parties. $I(T;\hat{T})$ can be expressed as:
	\begin{subequations}
		\begin{align}
			&I(T;\hat{T}) = I(S;\hat{S}) + H(K_A) - I(K_A;S;\hat{S}) \label{Eq:speical case 2a}\\
			&I(T;\hat{T}) = I(S;\hat{S})+H(K_B)-I(K_B;S;\hat{S}) \label{Eq:speical case 2b},
		\end{align}
	\end{subequations}
	where $I(K_A;S;\hat{S})=I(K_A;S)-I(K_A;S|\hat{S})$ is the mutual information between $K_A$, $S$ and $\hat{S}$.
	\begin{corollary}
		If the implicit channel is noiseless, the mutual information between $T$ and $\hat{T}$ satisfies
		\begin{equation}
			I(S;\hat{S}) \mathop{\leq}\limits^{(a)} I(T;\hat{T}) \mathop{\leq}\limits^{(b)} I(S;\hat{S}) + H(K_A).
		\end{equation}
	Moreover, if $K_A$ is a function of $S$ and $\hat{S}$, the left equation $(a)$ holds; if $K_A$ is independent of the signal $S$ or $\hat{S}$, the right equation $(b)$ holds.
	\end{corollary}
	\begin{proof}
	Since the nonnegativity of mutual information, $I(K_A;S;\hat{S})\geq 0$. Thus, $I(T;\hat{T})$ can be upper bounded by $I(S;\hat{S})+H(K_A)$. By the fact that the mutual information is lower than the entropy, we have $H(K_A)-I(K_A;S;\hat{S})\geq 0$. Combining these results, we obtain
	\begin{equation}
		I(S;\hat{S}) \leq I(T;\hat{T}) \leq I(S;\hat{S}) + H(K_A),
	\end{equation}
	which completes the proof of the bound of $I(T;\hat{T})$.
	\end{proof}

    }
	\item{\textbf{Both explicit and implicit channels are noiseless, i.e., \bm{$S=\hat{S},K_A=K_B$}.} In this setting, the mutual information between $T$ and $\hat{T}$ equals the entropy of $T$ or $\hat{T}$,
	\begin{equation}
		\label{Eq:speical case 3}
		I(T;\hat{T}) = H(T) = H(\hat{T}).
	\end{equation}
	Eq. (\ref{Eq:speical case 3}) indicates that Bob can perfectly understand the meaning of Alice when channels are noiseless. That is, no information is lost.
    }
\end{enumerate}

The following result shows the bounds of SRSA, constrained by the characteristics of the explicit and implicit channels.
\begin{lemma}
	The SRSA satisfies
\begin{equation}
	\text{SRSA}=\Pr\{T = \hat{T}\} \leq 1 - { H(K_A,S|K_B,\hat{S})-1   \over  \log|\mathcal{T}|}
	\label{Eq:SRSA}
\end{equation}
\end{lemma}
\begin{proof}
	Since $(K_B,\hat{S})$ can be seen as an estimator for $(K_A,S)$. Let $P_e =$ \text{Pr}$\{ T\neq \hat{T}\}$, then by Fano's Inequality, we obtain
	\begin{equation}
		P_e \geq { H(K_A,S|K_B,\hat{S})-1   \over  \log|\mathcal{T}|}
	\end{equation}
    Since SRSA$=1-P_e$, we can obtain Eq. (\ref{Eq:SRSA}), which completes the proof.
\end{proof}

\subsection{EXP-SC Model}

Let $S_1$ and $S_2$ denote the input signals of the explicit channel, and $\hat{S}_1$ and $\hat{S}_2$ are their corresponding output signals. Similarly, $K_A^1$ and $K_A^2$ represent Alice's knowledge instances. Alice encodes $S_1$ as $T_1$ with $K_A^1$, and $S_2$ as $T_2$ with $K_A^2$. On the other hand, Bob uses $K_B^1$ and $K_B^2$, decoding $\hat{S}_1$ and $\hat{S}_2$ into $\hat{T}_1$ and $\hat{T}_2$, respectively. In the basic model, the encoding and decoding processes of the two transmissions are
\begin{subequations}
\begin{align}
	S_1,K_A^1 & \to T_1~~~~\hat{S}_1,K_B^1\to \hat{T}_1 \label{Za}\\
	S_2,K_A^2 & \to T_2~~~~\hat{S}_2,K_B^2\to \hat{T}_2 \label{Zb}
\end{align}
\end{subequations}

Expansions combine simple things into complex systems and even generate intelligence, which is consistent with the evolution of the human language system. We extend the basic model proposed in Section \ref{sec:sc}, based on expansion. In this context, Alice wishes to send an expansion of multiple signals to Bob. We first consider the case of two signals, which can be generalized to more. Now, what Alice sends is expanded from $S = S_1$ to $S = S_1 \oplus T_2$.  We use $\oplus$ to denote semantic expansion. Specifically, the expansion of signals only represents their combination. For instance, '\emph{Carol published a paper}' expands to '\emph{Carol published a paper in \textbf{IEEE Communications Letters}}'.

Expansion often implies collision and fusion. Similarly, the expansion of signals corresponds to the collision of knowledge bases. For Alice, we use $K_A=K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2$ to represent the collision process, where $\odot$ denotes the collision and $\alpha$ is the \emph{collision factor}. Specifically, $\alpha$ is between $0$ and $1$, determined by the task. The collision factor reflects the role of $K_A^2$ compared to $K_A^1$ when the collision occurs. It can also be understood as the relative proportion of the contribution to the newly generated knowledge instance. The process of collision represents the asynchronous knowledge scaling up updates. Similarly, we use $K_B= K_B^1 \mathop{\odot}\limits^{\beta} K_B^2$ to represent the knowledge collision of Bob, where $\beta$ is a collision factor. The expansion and collision process can proceed continuously as follows:
\begin{subequations}
	\begin{align}
	&S = S_1 \oplus S_2 \oplus S_3  \cdots \oplus S_{n}  \label{Za1} \\
	&K_A = K_A^1 \mathop{\odot}\limits^{\alpha_1} K_A^2 \mathop{\odot}\limits^{\alpha_2} K_A^3  \cdots \mathop{\odot}\limits^{\alpha_{n-1}} K_A^{n}  \label{Za1} \\
	&K_B= K_B^1 \mathop{\odot}\limits^{\beta} K_B^2 \mathop{\odot}\limits^{\beta_2} K_B^3 \cdots
	\mathop{\odot}\limits^{\beta_{n-1}} K_B^{n}. \label{Zb1}
\end{align}
\end{subequations}

The expansion architecture of semantic communications is described as

\begin{equation}\label{Eq:structure}
	\begin{split}
		&S~~~~~~~\mathop{\longrightarrow}\limits^{(a)}~~~~~~~\hat{S} \\
		&\Uparrow~~~~~~~~~~~~~~~~~~\Downarrow \\
		T~\leftarrow~ ~S_1& \oplus S_2~~~~~~~~~~~~\hat{S_1}\oplus \hat{S_2}~\rightarrow ~R~(=\hat{T}) \\
		&\uparrow \begin{footnotesize} \text{(c)} \end{footnotesize}~~~~~~~~~~~~~~~~\uparrow \begin{footnotesize} \text{(d)} \end{footnotesize}\\
		K_A^1 &\mathop{\odot}\limits^{\alpha} K_A^2  ~~~\mathop{\dashrightarrow}\limits^{(b)}~~K_B^1 \mathop{\odot}\limits^{\beta} K_B^2
	\end{split}
\end{equation}

We named it EXP-SC. Moreover, $H(S_1\oplus S_2, K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2 )$ is called the Measure of Comprehension and Interpretation (MCI), which reflects the generation and evolution of semantics. It should be noted that $T$ is not a simple logic combination of $T_1$ and $T_2$, i.e., $T \neq T_1 \oplus T_2$. For example, $T_1$ is \emph{'Apple Inc.'}, it is a company and $S_1$ can be \emph{'Apple'}. $T_2$ is \emph{'the thirteenth generation'}, it is a number and $S_2$ can be \emph{'thirteen'}. But their collision may give rise to a new word called \emph{'iphone'}, which is a mobile communication product. In particular, $T$ reflects the result of $S$ under the influence of knowledge collision.

Based on these definitions above, we get some new results.
\begin{lemma} The mutual information between $T$ and $\hat{T}$ is given by
	\begin{equation}
		I(T;\hat{T}) = I(S;\hat{S}) + I(K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2;K_B^1 \mathop{\odot}\limits^{\beta} K_B^2|S, \hat{S}) + I(S;K_B^1 \mathop{\odot}\limits^{\beta} K_B^2|\hat{S})+ I(\hat{S};K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2|S)
		\label{Eq:mutual information}
	\end{equation}
\end{lemma}

\begin{proof}
	It is similar to the proof of Lemma \ref{lemma:1}, we omit the proof.
\end{proof}

Eq. (\ref{Eq:mutual information}) indicates that besides the characteristics of explicit and implicit channels, the relationship between $\alpha$ and $\beta$ also affects the performance of communication.

\begin{lemma}
	\label{lemma:gain}
	When the explicit channel is noiseless, the gain brought by semantic expansion is given by

	\begin{equation}
		\begin{split}
		{{I(T;\hat{T}) - I(T_1;\hat{T}_1)} } = (1+\gamma) ({H(S)} -H(S_1)), \\
		\end{split}
	\end{equation}
	where

    \begin{equation}
	\gamma =  { {I(K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2;K_B^1 \mathop{\odot}\limits^{\beta} K_B^2 \vert S_1 \oplus S_2) - I(K_A^1;K_B^1|S_1)} \over {H(S_1 \oplus S_2)} -H(S_1)}.
	\end{equation}
\end{lemma}
\begin{proof} We note that
	\begin{equation}
		I(T_1;\hat{T}_1) =  H(S_1)+ I(K_A^1;K_B^1|S_1) .
	\end{equation}
Then,
\begin{align}
	&{I(T;\hat{T}) - I(T_1;\hat{T}_1)} \over {H(S)} -H(S_1)  \\
	& = {{H(S) + I(K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2;K_B^1 \mathop{\odot}\limits^{\beta} K_B^2 \vert S) - H(S_1) - I(K_A^1;K_B^1|S_1)} \over {H(S_1 \oplus S_2)} -H(S_1)} \\
	&= 1 + { {I(K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2;K_B^1 \mathop{\odot}\limits^{\beta} K_B^2 \vert S) - I(K_A^1;K_B^1|S_1)} \over {H(S_1 \oplus S_2)} -H(S_1)},
\end{align}
which completes the proof.
\end{proof}

Lemma \ref{lemma:gain} shows that the asynchronous knowledge scaling up updates determine the effect of semantic expansion.

\begin{lemma}
	When the explicit channel is noiseless, the mutual information between $T$ and $\hat{T}$ is bounded by
	\begin{equation}
		{1 \over 2}{({H(S_1) + H(S_2)})} \leq I(T;\hat{T}) \leq H(K_A^1 \mathop{\odot}\limits^{\alpha} K_A^2) + H(S)
		\label{Eq:bound}
	\end{equation}
\end{lemma}
\begin{proof}
	Since the fact that expansion would create more possibilities, leading to the increase of uncertainty, we can get
		\begin{equation}
			 H(S_1 \oplus S_2) \geq H(S_1)
			~~ H(S_1 \oplus S_2) \geq H(S_2)
		\end{equation}
Through the properties of \emph{non-negativity of entropy} and  \emph{conditioning reduces entropy}, Eq. (\ref{Eq:bound}) can be derived directly. We omit the proof.
\end{proof}

\section{Experiment and Numerical Results}
\label{sec:experiment}
In this section, we use SRSA to measure the performance of SC, especially the impact that asynchronous knowledge scaling up has on the system.

\textbf{Basic Model.} We use Q-learning in \cite{catteeuw2014limits} to complete semantic encoding and decoding. Let $\vert \mathcal{S} \vert = M = 2$, $\vert \mathcal{K}_A \vert =\vert \mathcal{K}_B \vert =L=2$, $\vert \mathcal{T} \vert = L^2M^2$. In addition, $S$, $K_A$ and $K_B$ are uniformly distributed. We would like to discuss the impact of the characteristics of the explicit channel and implicit channel on SC. We use the binary symmetric channel (BSC) to denote the explicit channel. The BSC is shown in Fig. \ref{Fig:bsc}, which shows the probabilistic relationship between $S$ and $\hat{S}$, with the error probability $\epsilon_1$.

\begin{figure}[!t]
	\centering
	\includegraphics[width=2in]{bsc.pdf}
	\caption{Binary symmetric channel.}
	\label{Fig:bsc}
\end{figure}
For the implicit channel, we assume the correlation between $K_A$ and $K_B$ satisfies that
\begin{equation*}
	{K_B} = \begin{cases}
		K_A,&{\text{with probability}}\ {1-\epsilon_2 } \\
		U,  &{\text{with probability }} \epsilon_2 ,
	\end{cases}
\end{equation*}
where $U \sim \text{Unif}\{ 1,L\}$ is an independent random variable. In Fig. \ref{Fig:error2}, we show the SRSA with $\epsilon_1 \in $ $[0,0.5]$ and $\epsilon_2 \in $ $[0,0.5]$. When $\epsilon_1=\epsilon_2=0$, SRSA reaches the maximum 1. As $\epsilon_1$ and $\epsilon_2$ increase, SASA will decrease in both directions. This indicates that two channels jointly determine the quality of communication. For high quality SC, It is necessary to meet the condition that $\hat{S}=S$ and $K_B=K_A$ with a high probability.
\begin{figure}[!t]
	\centering
	\includegraphics[width=4.5in]{error2.pdf}
	\caption{The simulation results of SRSA with the explicit channel error probability $\epsilon_1$ and the implicit channel error probability $\epsilon_2$.}
	\label{Fig:error2}
\end{figure}

\textbf{EXP-SC Model.} We want to explore how communication quality varies in the context of different relationships between the receiver and the sender's knowledge instance. For simplicity, we assume that the explicit channel is noiseless, so we can focus on knowledge updates. That is, SRSA varies with the relationship of $K_B^1,K_B^2,\beta$ and $K_A^1, K_A^2,\alpha$. We categorize it into four cases.
\begin{itemize}
	\item Case \uppercase\expandafter{\romannumeral1}: $K_B^1=K_A^1$, $K_B^2=K_A^2$ and $\beta=\alpha$. The implicit channel is noiseless. Bob has the same asynchronous knowledge scaling up updates mode as Alice. That is, the receiver has all the knowledge of the sender.
	\item Case \uppercase\expandafter{\romannumeral2}: $K_B^1\neq K_A^1$ (with error probability 0.5), $K_B^2=K_A^2$ and $\beta=\alpha$. The receiver has partial knowledge of the sender.
	\item Case \uppercase\expandafter{\romannumeral3}: $K_B^1=K_A^1$, $K_B^2 \neq K_A^2$ (with error probability 0.5) and $\beta=\alpha$. The receiver has partial knowledge of the sender.
	\item Case \uppercase\expandafter{\romannumeral4}: $K_B^1=K_A^1$, $K_B^2=K_A^2$ and $\beta={1 \over 2} \alpha$. The collision factors do not equal.
\end{itemize}

Fig. \ref{fig_sim} illustrates the simulation results of SRSA with the number of runs, where Fig. \ref{fig_sim} (a) is the mean and (b) is the variance. In all four cases, the mean of SRSA gradually converges to a stable region. Case \uppercase\expandafter{\romannumeral1} is close to 1, which implies that when the receiver has all the knowledge of the sender, it can express exactly the same semantics as the sender. In other words, Bob has the same asynchronous knowledge scaling up updates mode as Alice, which facilitates the success of SC. The curve of case \uppercase\expandafter{\romannumeral2} and \uppercase\expandafter{\romannumeral3} are almost the same, but the value of case \uppercase\expandafter{\romannumeral2} is always higher than that of case \uppercase\expandafter{\romannumeral3}. This is as expected because $K_A^1$ and $K_B^1$ play a leading role in the collision compared to $K_A^2$ and $K_B^2$.
Case \uppercase\expandafter{\romannumeral4} reflects the learning of SRSA when Bob and Alice have different collision factors. The value of case \uppercase\expandafter{\romannumeral4} is higher than that of \uppercase\expandafter{\romannumeral2} and \uppercase\expandafter{\romannumeral3}, which indicates that the collision factor plays a smaller role than the knowledge itself in SC. These results also show that learning can improve SC quality with only partial background knowledge, but there is a limit.
On the other hand, the variance of case \uppercase\expandafter{\romannumeral1} keeps decreasing, and the other three cases also stabilize after peaking quickly. This further suggest that learning can evolve continuously with full knowledge, but there is an upper limit to learning with partial or no knowledge.

\begin{figure*}[!t]
	\centering
	\subfloat[]{\includegraphics[width=3.25in]{mean.pdf}%
		\label{fig:I}}
	\hfil
	\subfloat[]{\includegraphics[width=3.25in]{var.pdf}%
		\label{fig:II}}
	\caption{The simulation results of SRSA with the number of runs. We divide it into four cases. Case \uppercase\expandafter{\romannumeral1}: $K_B^1=K_A^1$, $K_B^2=K_A^2$ and $\beta=\alpha$; \uppercase\expandafter{\romannumeral2}: $K_B^1\neq K_A^1$ (with error probability 0.5), $K_B^2=K_A^2$ and $\beta=\alpha$; \uppercase\expandafter{\romannumeral3}: $K_B^1=K_A^1$, $K_B^2 \neq K_A^2$ (with error probability 0.5) and $\beta=\alpha$; \uppercase\expandafter{\romannumeral4}: $K_B^1=K_A^1$, $K_B^2=K_A^2$ and $\beta={1 \over 2} \alpha$. (a) the mean of SRSA. (b) the variance of SRSA.}
	\label{fig_sim}
\end{figure*}

\section{Conclusion}
\label{sec:conclusion}
In this paper, we presented the concept of semantic expansion and knowledge collision in SC. It represents the combination and superposition of information by the sender. Based on semantic expansion, we further proposed a semantic communication system called EXP-SC. Moreover, the semantic expansion corresponds to knowledge collision, which provides possibility for the evolution and upgrading of communication systems. On the other hand, we reached some conclusions for semantic information theory in the context of asynchronous knowledge scaling up updates, getting some bounds for SC. Specifically, the receiver's understanding of the knowledge collision and updates determines the effect of communication.

Semantic communication is evolving towards intelligence. The insights gained from this work may be of assistance to the future semantic communication system and 6G. It is also expected to pave the way for the design of next-generation real-time data networking and will provide the foundational technology for a plethora of socially useful services, including autonomous transportation, consumer robotics, VR/AR and metaverse.

\section*{Acknowledgments}
We thank Prof. Jinho for the help and source code in \cite{choi2022semantic}.

{\appendices
\section{Proof of the mutual information for three special  channel cases}
\begin{enumerate}[I.]
	\item{\textbf{The explicit channel is noiseless.} Since $S=\hat{S}$, we have $I(S;\hat{S})=H(S)$ and $I(K_A;K_B|S,\hat{S})=I(K_A;K_B|S)$. Moreover, $S$ is a function of $\hat{S}$, it follow that $I(S;K_B|\hat{S})=0$. Similarly, $\hat{S}$ is also a function of $S$, we obtain $I(\hat{S};K_A|S)=0$.
		Thus, the formula (\ref{Eq:Lemma 1}) can be simplified to
		\begin{equation}
			I(T;\hat{T}) = H(S) + I(K_A;K_B|S),
		\end{equation}
	}
	\item{\textbf{The implicit channel is noiseless}. Since $K_A=K_B$, the mutual information between $T$ and $\hat{T}$ is written
		\begin{small}
		\begin{align}
			I(T;\hat{T}) &= I(S;\hat{S}) + I(K_A;K_B|S, \hat{S}) + I(S;K_B|\hat{S})+ I(\hat{S};K_A|S) \\
			&= I(S;\hat{S}) + H(K_A|S,\hat{S})+I(S;K_A|\hat{S})+I(\hat{S};K_A|S) \\
			&= I(S;\hat{S}) + H(K_A)-I(K_A;S,\hat{S}) +I(S;K_A|\hat{S})+I(\hat{S};K_A|S) \\
			&= I(S;\hat{S}) + H(K_A) - I(K_A;S)-I(K_A;\hat{S}|S) +I(S;K_A|\hat{S})+I(\hat{S};K_A|S) \\
			&= I(S;\hat{S}) + H(K_A) - I(K_A;S)+I(K_A;S|\hat{S}) \\
			&= I(S;\hat{S}) + H(K_A) - I(K_A;S;\hat{S}).
		\end{align}
		\end{small}
	 It can also be expressed as
	 \begin{equation}
	 	I(T;\hat{T}) = I(S;\hat{S}) + H(K_B) - I(K_B;S;\hat{S}).
	 \end{equation}

	}
	\item{\textbf{Both explicit and implicit channels are noiseless, i.e.,} \bm{$S=\hat{S},K_A=K_B$}. In this setting, we have $I(K_A;K_B|S)=H(K_A|S)$. Thus, Eq. (\ref{Eq:speical case 1}) is written
		\begin{align}
			I(T;\hat{T}) &= H(S) + H(K_A|S)
			= H(K_A,S)
			= H(T).
		\end{align}
		Similarly, we also obtain $I(T;\hat{T})=H(\hat{T})$.
	}
\end{enumerate}

\bibliographystyle{IEEEtran}


\section{Introduction}\label{sec:intro}

When applying statistical testing of hypotheses to data it is often recommended not only to report the corresponding p-value, but in addition to provide a measure for the effect associated with a possible rejection of the null hypothesis, see e.g. \citet{wilkinson1999statistical}. Such a measure may be useful when sample sizes are to be fixed during the planning phase of a study, or when it is desired to assess the  relevance of an actual rejection when given sample sizes are large. Effect size measures are strongly related to power analysis as carried out in the seminal book by \citet{cohen1988statistical}.

A widely used measure is the so-called Cohen's $d$, see also \citet{hedges1981distribution, kraemer1983theory}, which is an effect size measure for the two-sample $t$ test with equal variances. Consider independent samples of sizes $n_{1}$ and $n_{2}$ of a statistical variable $y$ in two groups such that
$y$ follows a normal distribution with expectation $\mu_{1}$ and variance $\sigma^2$ in group 1 and expectation $\mu_{2}$ and the same variance $\sigma^2$ in group 2. Let $t$ denote the usual two-sample test statistic for the null hypotheses
$H_{0}: \mu_{1} = \mu_{2}$ versus the alternative $H_{1}: \mu_{1} \not = \mu_{2}$. As a measure for the size
of an effect, \citet[p. 66ff]{cohen1988statistical} considers the absolute value of
\begin{equation}\label{E1}
d = \frac{\overline{y}_{1}-\overline{y}_{2}}{\sqrt{\frac{s_{1}^2 + s_{2}^2}{n_{1} + n_{2} -2}}}\; ,
\end{equation}
where $\overline{y}_{j}$ is the sample mean in group $j$, $j=1,2$, and $s_{j}^2 =\sum_{i} (y_{i} - \overline{y}_{j})^2$, where summation is carried out with respect to all observation from group $j$. The effect size $d$ is related to the test statistic $t$ by the formula
\begin{equation}\label{E2}
d = t \sqrt{\frac{n_{1}+ n_{2}}{n_{1} n_{2}}}\; ,
\end{equation}
see (2.5.3) in \citet{cohen1988statistical}.
According to \citeauthor{cohen1988statistical}, values $|d|=0.2$, $|d|=0.5$ and $|d|=0.8$ indicate a small, medium and large effect, respectively.

It may also be of interest to have a corresponding measure when the variable $y$ depends on further independent variables. In his Chapter 9, \citet{cohen1988statistical} deals with such a multiple regression situation and discusses the effect size measure $f^2$ at length, as will further be explicated in our Section \ref{C1}.

However, an analogous measure to $d$ is rare to find, see
\citet[Sect. 3.14]{wilson2016formulas}, \citet{lipsey2001practical} for such a proposal. Nonetheless, it may be of particular interest to have comparable measures of an effect size for the very same  grouped variable $y$  but additionally depending on different sets of independent variables. This is exemplarily carried out in our Section \ref{C2}.
In the following we introduce such a measure as a generalization to $d$ by considering a linear regression model
\begin{equation}\label{E3}
y = \beta_{0}+ \beta_{1} z + \beta_{2} x_{1} + \cdots + \beta_{w+1} x_{w} +\varepsilon\; ,
\end{equation}
where $z$ takes the value $z_{i} = 0$ if the corresponding observation $y_{i}$ of the dependent variable $y$ belongs to group 1 and $z_{i} = 1$ if $y_{i}$ belongs to group 2, $i= 1,\ldots, n_{1} + n_{2}$. It is assumed that there are $w$ independent variables $x_{1}, \ldots, x_{w}$.
The error variable $\varepsilon$ is assumed to follow a normal distribution with expectation $0$ and variance $\sigma^2$.

As will be shown in the following Sections \ref{sec:part}, \ref{C3}, and \ref{C1}, a natural generalization of Cohen's $d$ is given by
\begin{equation}\label{E9}
d_{\ast} = \frac{\overline{y_{\ast}}_{1}-\overline{y_{\ast}}_{2}}{\sqrt{\frac{s_{\ast 1}^2 + s_{\ast 2}^2}{n_{1} + n_{2} - 2 - w}}}, \quad  s_{\ast j}^2 = \sum_{i} (y_{\ast i} - \overline{y_{\ast}}_{j})^2, \quad j=1,2\; ,
\end{equation}
where $y_{\ast} = y - \widehat{\beta}_{2} x_{1} - \cdots - \widehat{\beta}_{w+1} x_{w}$ is the dependent variable adjusted for the independent variables. The $\widehat{\beta}_{k}$ are the ordinary least squares estimates  of the regression coefficients $\beta_{k}$, $k=2,\ldots, w+1$ in model (\ref{E3}). In case $w=0$, the adjusted $y_{\ast}$ coincides with the original $y$, so that (\ref{E9}) reduces to (\ref{E1}) and therefore can be seen as a natural generalization of Cohen's $d$.

\section{Partitioned Linear Regression}\label{sec:part}

Let $n= n_{1} + n_{2}$ be the total sample size.  The above model (\ref{E3}) may also be written in vector-matrix notation as
\begin{equation}\label{E4}
y = X_{1} \delta_{1} + X_{2} \delta_{2} + \varepsilon, \quad
\end{equation}
where now $y$ represents the $n \times 1$ vector of observations of the dependent variable. Without loss
of generality it is assumed that the first $n_{1}$ observations belong to group 1, while the last $n_{2}$ observations belong to group 2. By introducing the notation $1_{m}$ for an $m\times 1$ vectors of ones, the $n\times 2$ matrix $X_{1}$ and the corresponding  $2\times 1$ parameter vector $\delta_{1}$ may be written as
\begin{equation}\label{E5}
X_{1} = \begin{pmatrix}
1_{n_{1}} & 0\\
1_{n_{2}} & 1_{n_{2}}
\end{pmatrix} \quad \text{and}\quad
\delta_{1} = \begin{pmatrix}
\beta_{0}\\
\beta_{1}
\end{pmatrix}\; .
\end{equation}
The $n\times w$ matrix $X_{2}$ contains the observations of the independent variables with corresponding regression coefficients $\delta_{2}^{T} = (\beta_{2}, \ldots, \beta_{w+1})$, where the $T$ superscript denotes transposition. The $n\times 1$ random vector $\varepsilon$ is assumed to follow a  multivariate normal distribution with expectation vector $0$ and variance-covariance matrix $\sigma^2 I_{n}$, where $I_{n}$ stands for
the $n\times n$ identity matrix. It is assumed that the $n\times (2+w)$ model matrix $(X_{1}, X_{2})$ has full column rank
$2 +w$. Equation (\ref{E4}) represents a partitioned linear regression model as considered e.g. in
\citet{fiebig1996frisch}. Generalizations and further properties are investigated  by
\citet{puntanen1996some, gross2000estimation, gross2005extensions, ding2021frisch}, among others.

Under model (\ref{E4}) the ordinary least squares estimator for the parameter vector $(\delta_{1}^{T}, \delta_{2}^{T})$ is given by
\begin{equation}\label{E6}
\begin{pmatrix}
\widehat{\delta}_{1}\\
\widehat{\delta}_{2}
\end{pmatrix}  = (X^{T} X)^{-1} X^{T} y,  \quad X=(X_{1},  X_{2})\; .
\end{equation}
The Frisch-Waugh-Lovell theorem, see \citet{fiebig1996frisch,lovell1963seasonal,frisch1933partial}, states that
\begin{equation}\label{E7}
\widehat{\delta}_{2} = (X_{2}^{T} M_{1}  X_{2})^{-1} X_{2}^{T} M_{1} y,\quad M_{1} = I_{n}- X_{1} (X_{1}^{T} X_{1})^{-1} X_{1}^{T}\; .
\end{equation}
For the specific choice (\ref{E5}), the matrix $M_{1}$ becomes
\begin{equation}\label{E8}
M_{1} = \begin{pmatrix}
C_{1} & 0\\
0 & C_{2}
\end{pmatrix},\quad C_{j} = I_{n_{j}} - n_{j}^{-1} 1_{n_{j}} 1_{n_{j}}^{T},\quad j=1,2\; .
\end{equation}

The following result is not  restricted to the case (\ref{E5}) but remains valid
in situations where  the matrix $X_{1}$ corresponds to an arbitrary set of $v$ independent variables such that the assumptions of (\ref{E4}) are satisfied.

\begin{thm}\label{T1} Under the partitioned linear regression model (\ref{E4}),
\begin{equation}
\widehat{\delta}_{1} = (X_{1}^{T}X_{1})^{-1} X_{1}^{T} (y - X_{2} \widehat{\delta}_{2})
\end{equation}
is the ordinary least squares estimator of $\delta_{1}$.
\end{thm}

A proof is given in the appendix. Theorem \ref{T1} means that if $\widehat{\delta}_{2}$ is known (e.g. computed by
(\ref{E7})), then the remaining parameters $\delta_{1}$ can  be estimated by regressing the adjusted \begin{equation}
y_{\ast} = y - X_{2} \widehat{\delta}_{2}\end{equation}  on the remaining  $X_{1}$ and this procedure just yields the identical estimate of $\delta_{1}$ from (\ref{E6}).

\begin{thm}\label{T2} Under the partitioned linear regression model (\ref{E4}) and (\ref{E5}),
\begin{eqnarray}
\widehat{\sigma}^2 & = & (y - X_{2} \widehat{\delta}_{2})^{T} M_{1} (y - X_{2} \widehat{\delta})/(n-2-w)\label{T2E1}\\
& = & (s_{\ast1}^{2} + s_{\ast 2}^{2})/(n-2-w)\label{T2E2}
\end{eqnarray}
is an unbiased estimator for $\sigma^2$.
\end{thm}

As a matter of fact, $\widehat{\sigma}^2$ coincides with the usual estimator for $\sigma^2$ in model (\ref{E4}). Identity (\ref{T2E2}) follows immediately from (\ref{E8}) with the above definition of $y_{\ast}$.

\section{Testing for a group effect}\label{C3}

From Theorem \ref{T1} with  $X_{1}$ from (\ref{E5}) it follows that
\begin{equation}\label{E18}
\widehat{\delta}_{1} =  \begin{pmatrix}
\widehat{\beta}_{0}\\
\widehat{\beta}_{1}
\end{pmatrix}= \begin{pmatrix}
n_{1}^{-1} 1_{n_{1}} & 0\\
- n_{1}^{-1} 1_{n_{1}} & n_{2}^{-1} 1_{n_{2}}
\end{pmatrix} y_{\ast}
= \begin{pmatrix}
\overline{y_{\ast}}_{1}\\
\overline{y_{\ast}}_{2} - \overline{y_{\ast}}_{1}
\end{pmatrix}\; .
\end{equation}
Hence, it is seen that $|d_{\ast}|$ from (\ref{E9}) is identical to
\begin{equation}\label{E17}
|d_{\ast}| = \frac{|\widehat{\beta}_{1}|}{\widehat{\sigma}}
\end{equation}
with $\widehat{\sigma}$ being the square root of $\widehat{\sigma}^2$ from Theorem \ref{T2}. The
statistic $d_{\ast}$ is closely related to the test statistic $t_{\ast}$ for the null hypothesis
$H_{0}: \beta_{1} = 0$ in model (\ref{E4}).

\begin{thm}\label{T3} Under the partitioned linear regression model (\ref{E4}) and (\ref{E5}) let
$M_{2} = I_{n} - P_{2}$, $P_{2} = X_{2} (X_{2}^{T}  X_{2})^{-1} X_{2}^{T}$, and let
$\gamma$ be the lower-right element of the $2\times 2$ matrix $(X_{1}^{T} M_{2} X_{1})^{-1}$. Then the  statistic
\begin{equation}\label{E10}
t_{\ast} = d_{\ast} /\sqrt{\gamma}
\end{equation}
follows a central $t$ distribution with $n - 2-w$ degrees of freedom, provided $\beta_{1} =0$.
\end{thm}

In the above theorem, $\gamma$ is the scaled variance of $\widehat{\beta}_{1}$, i.e. $\text{Var}(\widehat{\beta}_{1}) = \sigma^2 \gamma$, see the proof of Theorem \ref{T3} in the appendix. The standard error of
$\widehat{\beta}_{1}$ is thus $\text{se}(\widehat{\beta}_{1}) = \widehat{\sigma} \sqrt{\gamma}$ with $\widehat{\sigma}$ being the square root of $\widehat{\sigma}^2$ from Theorem \ref{T2}.

Note that in case $w = 0$ by setting $M_{2}=I_{n}$ one gets
\begin{equation}
(X_{1}^{T} X_{1})^{-1} = \begin{pmatrix}
n_{1}^{-1} & - n_{1}^{-1}\\
- n_{1}^{-1} & \gamma
\end{pmatrix}, \quad \gamma
= \frac{n_{1} + n_{2}}{n_{1} n_{2}}\; ,
\end{equation}
and hence
\begin{equation}
d_{\ast} = t_{\ast} \sqrt{\frac{n_{1} + n_{2}}{n_{1} n_{2}}}\; ,
\end{equation}
which is just a reformulation of (\ref{E2}). These considerations show that $d_{\ast}$ is a natural extension
of Cohen's $d$ in the context of additional independent variables.

\section{Effect Size in Multiple Regression}\label{C1}

In his Chapter 9, \citet{cohen1988statistical} discusses the effect size measure $f^{2}$ based on the $F$ test of a linear hypothesis. It may be applied  when $X_{1}$ does not only comprise intercept and one dummy as under model (\ref{E4}), but a total of $u$ independent variables. Then it might be of interest to measure the effect size of the set of variables in $X_{1}$ given the set in $X_{2}$, which is Cohen's case 1. \citeauthor{cohen1988statistical} suggests values $f^2=0.02$, $f^2=0.15$ and $f^2=0.35$ for a small, medium and large effect, respectively.
Since the measure $d_{\ast}$ refers to one dummy ($u=1$), one might expect a relationship between $d_{\ast}$ and the corresponding $f^2$.  Actually, as noted in our Remark below, such a relationship can be specified.

The measure $f^2$ for Cohen's case 1 is  given by
\begin{equation}\label{E11}
f^2  = F \frac{u}{v},\quad  v = n-u-w- 1\; ,
\end{equation}
where under model (\ref{E4}) $F$ is the $F$ statistic for testing the null hypothesis $H_{0}: \beta_{1}=0$. From (9.2.3) in \citet{cohen1988statistical},
\begin{equation}\label{E15}
f^2 = \frac{R^2 - R_{0}^2}{1 -R^2}\; ,
\end{equation}
where $R^2$ is the coefficient of determination from model (\ref{E4}) and $R_{0}^2$ is the coefficient of determination in the reduced model with $\beta_{1}=0$, admitting model matrix $X_{3}=(1_{n}, X_{2})$. If $P$ denotes
the orthogonal projector onto the column space of the model matrix of a regression model with intercept, the coefficient of determination is given by
\begin{equation}
R^2 = 1 - \frac{y^{T} (I_{n} - P) y}{y^{T} C y
\end{equation}
with $C= I_{n} - n^{-1} 1_{n} 1_{n}^{T}$ being the so-called centering matrix, e.g. see \citet[Sect. 6.2]{gross2003linear}. From this, (\ref{E15}) becomes
\begin{equation}
f^2 = \frac{y^{T}(P - P_{3}) y}{y^{T} (I_{n} - P) y}
\end{equation}
with $P=X(X^{T} X)^{-1} X^{T}$, $X=(X_{1}, X_{2})$, and $P_{3} = X_{3} (X_{3}^{T} X_{3})^{-1} X_{3}^{T}$.
In view of $\text{rank}(P-P_{3}) =1$ and $\text{rank}(I_{n} - P) = n - (2 + w)$, the corresponding $F$ statistic reads
\begin{equation}
F = \frac{y^{T}(P - P_{3}) y/\text{rank}(P-P_{3})}{y^{T} (I_{n} - P) y/\text{rank}(I_{n} - P)}\; .
\end{equation}
Then, from Theorem 3.2.1 (ii) in \citet{christensen2020plane}, $F$ follows a central $F$ distribution with  $1$ and $n-2-w$ degrees of freedom, provided $\beta_{1}=0$.

Now, it is well known and readily verified that the squared $t$ statistic for the null hypothesis $H_{0}: \beta_{1}=0$ is identical to the test statistic of the $F$ test for the very same hypothesis. Thus, by combining
(\ref{E10}) and (\ref{E11}) the following is true.

\begin{rmk} The identity
\begin{equation}\label{E14}
f^2 = \frac{d_{\ast}^2 /\gamma}{n-2-w}
\end{equation}
specifies the exact relationship between the effect size measures $f^2$ and $d_{\ast}$ from above.
\end{rmk}

Note that in case $w=0$ the above identity reads
\begin{equation}\label{E12}
f^2 = d^{2} \frac{n_{1} n_{2}}{n (n-2)}\; ,
\end{equation}
which slightly differs from formula (9.3.5) in \citet{cohen1988statistical} and lacks some of its beauty. Since (\ref{E12})
is expected to coincide with (9.3.5) this reveals a fallacy in the latter formula. Formula (\ref{E12}) may also be verified independently by directly assuming model (\ref{E4}) without any additional independent variables $X_{2}$. In most cases actual computations of the two formulas in question only differ in a later digit after the dot (say the fifth or sixth), so the difference usually has no practical meaning. The correctness of (\ref{E14}) and (\ref{E12}) is additionally confirmed by  applications to real data.

\section{Data Example}\label{C2}

To give a possible outline for applications and an illustration of the previous formulas we employ a data set available from the UCI machine learning repository, see \citet{Dua:2019}. It contains
student achievement in secondary education of two Portuguese schools, see \citet{cortez2008using}. In the following, computations are carried out with the  statistical software {\sf R} \citep{Rsoftware}.

\begin{figure}[hbt]
\centering
\includegraphics[width=19pc,angle=0]{G3FM.pdf}\\
  \caption{Frequency of final results (variable G3) in Portuguese language of $n=649$ students}\label{F1}
\end{figure}

As the dependent variable $y$ we consider the final grade with integer values ranging between 0 and 20 in Portuguese language (variable G3) of $n_{1} = 383$ female and $n_{2} = 266$ male students. The dummy variable $z$ takes values $0$ for female and $1$ for male. As also indicated by Figure \ref{F1}
female students perform better with an average of $\overline{y}_{1} = 12.25326$ compared to
$\overline{y}_{2} = 11.40602$ for male students. The corresponding equal variances two-sample test statistic admits $t=3.310938$ with p-value
$0.0009815287$. Although this implies strong significance the corresponding effect size from
(\ref{E2}) reads $d = 0.264261$, thereby indicating only a slightly more than low effect. This value may also
be obtained by function {\tt cohens\_d} from the {\sf R} package {\tt effectsize}, see \citet{R:effectsize}.

As additional independent variables we consider the education of the father (Fedu) $x_{1}$ and the travel time from home to school (traveltime) $x_{2}$. Both variables are measured on an ordinal scale with integer values ranging from 0 to 4 and 1 to 4, respectively, and are included as quantitative variables in our regression approach, implying $w=2$.

\begin{table}[hbt]
\centering
\begin{alltt}
                    Estimate Std. Error t value Pr(>|t|)
        (Intercept)  11.4138     0.4306  26.507  < 2e-16 ***
        ZM           -0.9406     0.2503  -3.759 0.000186 ***
        X1            0.6096     0.1144   5.329 1.37e-07 ***
        X2           -0.3369     0.1676  -2.010 0.044826 *
\end{alltt}
\caption{Extract from {\sf R} output by fitting the complete model (\ref{E4}) with function {\tt lm}, where variable sex is included as a factor, implying that {\tt ZM} represents the dummy $z$}\label{Tab1}
\end{table}
Least squares estimates of the coefficients with corresponding $t$ statistic values are given in Table \ref{Tab1}. The intercept estimate $\widehat{\beta}_{0}=11.41385$ is the average of the adjusted final grade $y_{\ast}$ of females, while the dummy variable estimate $\widehat{\beta}_{1}=-0.9406209$ is the difference between the average of $y_{\ast}$ in the male group minus the average in the female group as given in \ref{E18}. As it is also seen, better father's eduction comes along with better grades (positive $\widehat{\beta}_{2}$) while longer travel times to school come along with lower grades (negative $\widehat{\beta}_{3}$) when the other variables are held constant, respectively.  The coefficient of determination from this model reads $R^2 = 0.07238847$, while for the reduced model (omitting sex) it is $R_{0}^2 = 0.05207054$. Then $f^2$ from (\ref{E15}) is $f^2 = 0.0219035$, implying a slightly more than small effect concerning the difference between female and male final grades, ceteris paribus.

The relationship (\ref{E14}) may also be used to infer Cohen's $d_{\ast}$ from $f^2$. For this
\begin{equation}
(X_{1}^{T} M_{2} X_{1})^{-1} = \begin{pmatrix}
0.019062813 & -0.001591796\\
-0.001591796 & 0.006438624
\end{pmatrix}\; ,
\end{equation}
the lower-right element being the scaled variance $\gamma$ of $\widehat{\beta}_{1}$. With $\widehat{\sigma} = 3.118756$ it follows $d_{\ast}= 0.3016013$.
This indicates a slightly stronger effect when variables $x_{1}$ and $x_{2}$ are held constant than seen before from  $d = 0.264261$ not considering any additional independent variables at all. Alternatively, $d_{\ast}$ may be computed by either of the formulas (\ref{E17}) or (\ref{E9}) yielding the very same absolute value.