{'fraction_non_alphanumeric': 0.03554880364603114, 'fraction_numerical': 0.01769844284086593, 'mean_word_length': 4.828021248339973, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Curriculum Temperature for Knowledge Distillation

Most existing distillation methods ignore the flexible role of the temperature in the loss function and fix it as a hyper-parameter that can be decided by an inefficient grid search. In general, the temperature controls the discrepancy between two distributions and can faithfully determine the difficulty level of the distillation task. Keeping a constant temperature, i.e., a fixed level of task difficulty, is usually sub-optimal for a growing student during its progressive learning stages. In this paper, we propose a simple curriculum-based technique, termed Curriculum Temperature for Knowledge Distillation (CTKD), which controls the task difficulty level during the student's learning career through a dynamic and learnable temperature. Specifically, following an easy-to-hard curriculum, we gradually increase the distillation loss w.r.t. the temperature, leading to increased distillation difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing knowledge distillation frameworks and brings general improvements at a negligible additional computation cost. Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the effectiveness of our method. Our code is available at https://github.com/zhengli97/CTKD.


Introduction
Knowledge distillation (Hinton, Vinyals, and Dean 2015) (KD) has received increasing attention from both academic and industrial researchers in recent years. It aims at learning a comparable and lightweight student by transferring the knowledge from a pretrained heavy teacher. The traditional process is implemented by minimizing the KL-divergence loss between two predictions obtained from the teacher/student model with a fixed temperature in the softmax layer. As depicted in (Hinton, Vinyals, and Dean 2015;Liu et al. 2022;Chandrasegaran et al. 2022), the temperature controls the smoothness of distribution and can faithfully determine the difficulty level of the loss minimization process. Most existing works (Tung and Mori 2019; Chen et al. 2020;Ji et al. 2021) ignore the flexible role of the temperature and empirically set it to a fixed value (e.g., 4). Differently, MKD (Liu et al. 2022) proposes to learn the suitable temperature via meta-learning. However, it has certain limitations that require an additional validation set to train the temperature module, which complicates the training process. Besides, it mainly focuses on the strong data augmentation condition, neglecting that most existing KD methods work under normal augmentation. Directly combining MKD with existing distillation methods under strong augmentation may cause severe performance degradation (Das et al. 2020).
In human education, teachers always train students with simple curricula, which start from easier knowledge and gradually present more abstract and complex concepts when students grow up. This curriculum learning paradigm has inspired various machine learning algorithms (Caubrière et al. 2019;Duan et al. 2020). In knowledge distillation, LFME (Xiang, Ding, and Han 2020) adopt the classic curriculum strategy and propose to train the student gradually using samples ordered in an easy-to-hard sequence. RCO  propose to utilize the sequence of the teacher's intermediate states as the curriculum to gradually guide the learning of a smaller student. The progressive curricula based on data samples and models can help students learn better representations during distillation, but it requires a careful curriculum design and complex computational process, making it hard to deploy into existing methods.
In this paper, we propose a simple and elegant curriculumbased approach, called Curriculum Temperature for Knowledge Distillation (CTKD), which enhances the distillation performance by progressively increasing the learning difficulty level of the student through a dynamic and learnable temperature. The temperature is learned during the student's training process with a reversed gradient that aims to maximize the distillation loss (i.e., increasing the learning difficulty) between teacher and student in an adversarial manner. Specifically, the student is trained under a designed curriculum via the learnable temperature: following the easy-to-hard principle, we gradually increase the distillation loss w.r.t. the temperature, resulting in increased learning difficulty through simply adjusting the temperature dynamically. This operation can be easily implemented by a non-parametric gradient reversal layer (Ganin and Lempitsky 2015) to reverse the gradients of the temperature, which hardly introduces extra computation budgets. Furthermore, based on the curriculum principle, we explore two (global and instance-wise) versions of the learnable temperature, namely Global-T and Instance-T respectively. As an easyto-use plug-in technique, CTKD can be seamlessly integrated into most existing state-of-the-art KD frameworks and achieves comprehensive improvement at a negligible additional computation cost.
In summary, our contributions are as follows: • We propose to adversarially learn a dynamic temperature hyperparameter during the student's training process with a reversed gradient that aims to maximize the distillation loss between teacher and student.
• We introduce simple and effective curricula which organize the distillation task from easy to hard through a dynamic and learnable temperature parameter.
• Extensive experiment results demonstrate that CTKD is a simple yet effective plug-in technique, which consistently improves existing state-of-the-art distillation approaches with a substantial margin on CIFAR-100 and ImageNet.

Related Work
Curriculum Learning. Originally proposed by (Bengio et al. 2009), curriculum learning  is a way to train networks by organizing the order in which tasks are learned and incrementally increasing the learning difficulty (Morerio et al. 2017;Caubrière et al. 2019). This training strategy has been widely applied in various domains, such as computer vision (Wu et al. 2018;Sinha, Garg, and Larochelle 2020) and natural language processing (Platanios et al. 2019;Tay et al. 2019). Curriculum Dropout (Morerio et al. 2017) dynamically increases the dropout ratios in order to improve the generalization ability of the model. PG-GANs (Karras et al. 2017) learn to sequentially generate images from low-resolution to high-resolution, and also grew both generator and discriminator simultaneously. In knowledge distillation, various works (Xiang, Ding, and Han 2020;Zhao et al. 2021) adopt the curriculum learning strategy to train the student model. LFME (Xiang, Ding, and Han 2020) proposes to use the teacher as a difficulty measure and organize the training samples from easy to hard so that the model can receive a less challenging schedule. RCO  proposes to utilize the sequence of teachers' intermediate states as a curriculum to supervise the student at different learning stages. Knowledge Distillation. KD (Hinton, Vinyals, and Dean 2015) aims at effectively transferring the knowledge from a pretrained teacher model to a compact and comparable student model. Traditional methods propose to match the output distributions of two models by minimizing the Kullback-Leibler divergence loss with a fixed temperature hyperparameter. To improve distillation performance, existing methods have designed various forms of knowledge transfer. It can be roughly divided into three types, logit-based (Chen et al. 2020;Li et al. 2020b;Zhao et al. 2022), representationbased (Yim et al. 2017;Chen et al. 2021) and relationshipbased (Park et al. 2019;Peng et al. 2019) methods. The temperature controls the smoothness of probability distributions and can faithfully determine the difficulty level of the distillation process. As discussed in (Chandrasegaran et al. 2022;Liu et al. 2022), a lower temperature will make the distillation pays more attention to the maximal logits of teacher output. On the contrary, a higher value will flatten the distribution, making the distillation focus on the logits. Most works ignore the effectiveness of the temperature on distillation and fix it as a hyperparameter that can be decided by an inefficient grid search. However, keeping a constant value, i.e., a fixed level of distillation difficulty, is sub-optimal for a growing student during its progressive learning stages.
Recently, MKD (Liu et al. 2022) proposes to learn the temperature by performing meta-learning on the extra validation set. It mainly works on the ViT (Dosovitskiy et al. 2020) backbone with strong data augmentation while most existing KD methods work under normal augmentation. Directly applying MKD to other distillation methods may weaken the effect of distillation (Das et al. 2020). Our proposed CTKD is more efficient than MKD since we don't need to pay the effort to split and preserve an extra validation set. Besides, CTKD works under normal augmentation, so it can be seamlessly integrated into existing KD frameworks. The detailed comparison and discussion are attached in the supplement.

Method
In this section, we first review the concept of knowledge distillation and then introduce our proposed curriculum temperature knowledge distillation technique.

Background
Knowledge distillation (Hinton, Vinyals, and Dean 2015), as one of the main network compression techniques, has been widely used in many vision tasks Ye et al. 2019;Li et al. 2021b. The traditional two-stage distillation process usually starts with a pre-trained cumbersome teacher network. Then a compact student network will be trained under the supervision of the teacher network in the form of soft predictions or intermediate representations (Romero et al. 2014;Yim et al. 2017). After the distillation, the student can master the expertise of the teacher and use it for final deployment. Given the labeled classification dataset D = {(x i , y i )} I i=1 , the Kullback-Leibler (KL) divergence loss is used to minimize the discrepancy between the soft output probabilities of the student and teacher model: where q t and q s denote the logits produced by teacher and student, σ(·) is the softmax function, and τ is the temperature to scale the smoothness of two distributions. As discussed in previous works (Hinton, Vinyals, and Dean 2015;Liu et al. 2022), a lower τ will sharpen the distribution, enlarge the difference between two distributions and make distillation focus on the maximal logits of teacher prediction. While a higher τ will flatten the distribution, narrow the gap between two models and make the distillation focus on whole logits. Therefore, the temperature value τ can

Loss Loss
Figure 1: An overview of our proposed Curriculum Temperature for Knowledge Distillation (CTKD). (a) We introduce a learnable temperature module that predicts a suitable temperature τ for distillation. The gradient reversal layer is proposed to reverse the gradient of the temperature module during the backpropagation. (b) Following the easy-to-hard curriculum, we gradually increase the parameter λ, leading to increased learning difficulty w.r.t. temperature for the student.
faithfully determine the difficulty level of the KD loss minimization process by affecting the probability distribution.

Adversarial Distillation
For a vanilla distillation task, the student θ stu is optimized to minimize the task-specific loss and distillation loss. The objective of the distillation process can be formulated as follows: (2) where L task is the regular cross-entropy loss for the image classification task, f t (·) and f s (·) denotes the function of teacher and student. α 1 and α 2 are balancing weights. In order to control the learning difficulty of the student via dynamic temperature, inspired by GANs (Goodfellow et al. 2014), we propose to adversarially learn a dynamic temperature module θ temp that predicts a suitable temperature value τ for the current training. This module is optimized in the opposite direction of the student, intending to maximize the distillation loss between the student and teacher. Different from vanilla distillation, the student θ stu and temperature module θ temp play the two-player mini-max game with the following value function L(θ stu , θ temp ): We apply the alternating algorithm to solve the problem in Eqn. (3), fixing one set of variables and solving for the other set. Formally, we can alternate between solving these two subproblems: The optimization process for Eqn. (4) and Eqn. (5) can be conducted via stochastic gradient descent (SGD). The student θ stu and temperature module θ temp parameters are updated as follows: where µ is the learning rate.
In practice, we implement the above adversarial process (i.e., Eqn. (7)) by a non-parametric Gradient Reversal Layer (GRL) (Ganin and Lempitsky 2015). The GRL is inserted between the softmax layer and the learnable temperature module, as shown in Fig. 1(a).

Curriculum Temperature
Keeping a constant learning difficulty is sub-optimal for a growing student during its progressive learning stages. In school, human teachers always teach students with curricula, which start with basic (easy) concepts, and then gradually present more advanced (difficult) concepts when students T pred T pred Prob Figure 2: The illustrations of global and instance-wise temperature modules. B denotes the batch size, C denotes the number of classes. τ is the final temperature. grow up. Humans will learn much better when the tasks are organized in a meaningful order. Inspired by curriculum learning (Bengio et al. 2009), we further introduce a simple and effective curriculum which organizes the distillation task from easy to hard via directly scaling the loss L by magnitude λ w.r.t. the temperature, i.e., L → λL. Consequently, the θ temp would be updated by: At the beginning of training, the junior student has limited representation ability and requires to learn basic knowledge. We set the initial λ value to 0 so that the junior student can focus on the learning task without any constraints. By gradually increasing λ, the student learns more advanced knowledge as the distillation difficulty increases. Specifically, following the basic concept of curriculum learning, our proposed curriculum satisfies the following two conditions: (1) Given the unique variable τ , the distillation loss w.r.t. the temperature module (simplified as L kd (τ )) gradually increases, i.e., (2) The value of λ increases, i.e., where n represents the n-th step of training. In our method, when training at E n epoch, we gradually increase λ with a cosine schedule as follows: (11) where λ max and λ min are ranges for λ. E loops is the hyperparameter that gradually varies the difficulty scale λ. In our method, we default to set λ max , λ min and E loops to 1, 0 and 10, respectively. This curriculum indicates that the parameter λ increases from 0 to 1 during 10 epochs of training and keeps 1 until the end. Detailed ablation studies are conducted in Table 6 and Table 8.

Learnable Temperature Module
In this section, we introduce two versions of the learnable temperature module, namely Global-T and Instance-T.
Global-T. The global version consists of only one learnable parameter, predicting one value T pred for all instances, for data batch x in D do 3: Forward propagation through θ tea and θ stu to obtain predictions f t (x; θ tea ), f s (x; θ stu );

4:
Obtain temperature τ by θ temp in Eqn. (12) and parameter λ n in Eqn. (11); 5: Calculate the loss L and update θ stu and θ temp by backward propagation as Eqn. (6) Fig. 2(a). This efficient version does not bring additional computational costs to the distillation process since it only involves a single learnable parameter.
Instance-T. To achieve a better distillation performance, one global temperature is not accurate enough for all instances. We further explore the instance-wise variant, termed Instance-T, which predicts a temperature for all instances individually, e.g., for a batch of 128 samples, we predict 128 corresponding temperature values. Inspired by GFLv2 (Li et al. 2020a(Li et al. , 2021a, we propose to utilize the statistical information of probability distribution to control the smoothness of itself. Specifically, a 2-layer MLP is introduced in our work, which takes two predictions as input and outputs predicted value T pred , as shown in Fig. 2(b). During training, the module will automatically learn the implicit relationship between original and smoothed distribution.
To ensure the non-negativity of the temperature parameter and keep its value within a proper range, we scale the predicted T pred with the following equation: where τ init denotes the initial value, τ range denotes the range for τ , δ(·) is the sigmoid function, T pred is the predicted value. We default to set τ init and τ range to 1 and 20, so that all normal values can be included. Compared to Global-T, Instance-T can achieve better distillation performance due to its better representation ability.  In the following experiments, we mainly use the global version as the default scheme. We demonstrate the effectiveness of the instance-wise temperature method in Table 2. To get a better understanding of our method, we describe the training procedure in Algorithm 1.

Experiments
We evaluate our CTKD on various popular neural networks e.g., VGG (Simonyan and Zisserman  Our adversarial distillation technique makes the optimization process harder than the vanilla method as expected.
tection dataset. The train2017 split contains 118k images, and the val2017 split contains 5k images. Implementation details. All details are attached in supplement due to the page limit.

Main Results
CIFAR-100 classification. Table 1 shows the top-1 classification accuracy on CIFAR-100 based on eleven different teacher-student pairs. We can observe that all different student networks benefit from our method and the improvement are quite significant in some cases. Fig. 3 shows the loss curves of vanilla KD and CTKD. During training, the temperature module is optimized to maximize the distillation loss, which satisfies the condition in Eqn. (9). While the student is optimized to minimize the distillation loss, which plays a leading role in this mini-max game. So the overall losses still show a downward trend. As shown in Fig. 3, the distillation loss of CTKD is higher than the vanilla method, proving the effectiveness of adversarial temperature distillation. We can observe that the distillation loss of CTKD is higher than that of vanilla KD, proving the effect of the adversarial operation. Fig. 4 demonstrates that representations of our method are more separable than vanilla KD, proving that CTKD benefits the discriminability of deep features. Fig. 5 shows the learning curves of temperature during training. Compared to fixed temperature distillation, our curriculum temperature method achieves better results via an effective dynamic mechanism.
Global and instance-wise temperature. Table 2 shows the top-1 classification accuracy and computational efficiency (MACs, Time) of the global and instance-wise versions. Since the instance-wise method introduces an addi-    tional network (i.e., 2-layer MLP) to obtain stronger representation ability, it requires more computational cost than the global version. From this table, we can see that both versions can improve student performance at a negligible additional computational cost. We mainly use the global version in the following experiments. Applied to existing distillation works. As an easy-to-use plug-in technique, CTKD can be seamlessly integrated into existing distillation works. As shown in Table 3, our method brings comprehensive improvements to six state-of-the-art methods based on seven teacher-student pairs. More importantly, CTKD does not incur additional computational costs to the methods since it only contains a lightweight learnable temperature module and a non-parameterized GRL.
ImageNet-2012 classification. Table 4 reports the top-1/-5 accuracy of image classification on ImageNet-2012. As a plug-in technique, we also applied our CTKD to four existing state-of-the-art distillation works. The result shows that CTKD can still works on the large-scale dataset effectively.   MS-COCO object detection. We also apply our method to the object detection task. We follow the object detection implementation of DKD. As shown in Table 5, our CTKD can further boost the detection performance.

Ablation Study
In the following experiments, we evaluate the effectiveness of hyper-parameters and components on CIFAR-100. We set ResNet-110 as the teacher and ResNet-32 as the student.
Curriculum parameters. Table 6 reports the student accuracy with different λ min , λ max , and E loops . Table 7 reports the distillation results with different fixed λ. The training of students needs to gradually increase the learning difficulty. Directly starting with a fixed high-difficulty task will significantly reduce the performance of students, especially when λ is greater than 4. Besides, as shown in the sixth and seventh columns of Table 6, rapidly increasing parameter λ in a short time can also be detrimental to student training.    Table 7: Training with fixed λ. Compared with curriculum λ, directly training the student with the fixed high-difficulty task (e.g., λ>4) will reduce distillation performance.
When we smooth the learning difficulty of the student and increase E loops , the performance can be further improved. Curriculum strategy. In Table 8, we compare the performance of different curriculum strategies. "None τ =1 , 10 Epoch" means that during the first 10 epochs of training, we only use vanilla distillation, and set the temperature τ = 1. After 10 epochs, we start to train the student with CTKD, and λ is fixed to 1. "Lin [0,1] , 10 Epoch" means that we use CTKD to train the student with a linear increasing strategy. The parameter λ is gradually increased from 0 to 1 in 10 epochs of training. The value of λ remains 1 until the end. From this table, we can see that the cosine curriculum strategy works the best.
Adversarial temperature and curriculum distillation. We evaluate the effectiveness of these two elements as shown in Table 9. The second row means that we only adopt the adversarial temperature technique and use the fixed learning difficulty (i.e., fix λ to 1) to train the student. The results demonstrate that learning the temperature parameter in an adversarial manner can also improve distillation performance. The third row shows that the coopera-   Table 9: Ablation of Adversarial Temperature (AT) module and Curriculum Distillation (CD) strategy. The first row indicates the vanilla distillation performance.
tion of two elements can achieve better results than a single element.

Conclusion
In this paper, we propose a curriculum-based distillation approach, termed Curriculum Temperature for Knowledge Distillation, which organizes the distillation task from easy to hard through a dynamic and learnable temperature. The temperature is learned during the student's training process with a reversed gradient that aims to maximize the distillation loss (i.e., increase the learning difficulty) between teacher and student in an adversarial manner. As an easy-touse plug-in technique, CTKD can be seamlessly integrated into existing state-of-the-art knowledge distillation frameworks and brings general improvements at a negligible additional computation cost.

Supplementary Materials Implementation Details
All the methods are implemented by PyTorch (Paszke et al. 2019). CIFAR-100: We follow the standard data augmentation scheme for all training images as in Zhao et al. 2022), i.e. random cropping and horizontal flipping. We use the stochastic gradient descents (SGD) as the optimizer with momentum 0.9 and weight decay 5e-4. The learning rate is divided by 10 at 150, 180, and 210 epochs, for a total of 240 epochs. The mini-batch size is 64. The initial learning rate is set to 0.01 for MobileNet/ShuffleNet architectures and 0.05 for other architectures. We set λ max , λ min and E loops to 1, 0 and 10. We use the Global-T version as the default scheme in all experiments unless we specified. The temperature T in existing KD methods is default set to 4 in all experiments. In Table 1, we directly cite the vanilla KD results reported in Zhao et al. 2022) since we use the same experimental settings. In Table 3, we combine existing methods (i.e., PKT, SP, VID, CRD, SRRL) with KD and rerun the experiments according to their official implementations 123 for fair comparison. ImageNet: We set the initial learning rate to 0.1 and divide the learning rate by 10 at 30, 60, and 90 epochs. We follow the standard training process but train for 20 more epochs (i.e., 120 epochs in total). Weight decay is set to 1e-4. E loops is set to 5. The mini-batch size is 256. In Table 4, all teacher and student models are retrained based on 120 epochs.
MS-COCO: Our implementation for MS-COCO follows the settings in Zhao et al. 2022). We adopt the two-stage object detection method Faster-RCNN (Ren et al. 2015) with FPN (Lin et al. 2017) as the basic network. In our CTKD implementation, we set λ max , λ min and E loops to 0.25, 0 and 6, respectively.
Training details: Existing methods have designed various forms of knowledge transfer in addition to the traditional soft labels. It also contains representation-based (Yim et al. 2017;Chen et al. 2021) and relationship-based (Park et al. 2019;Peng et al. 2019) methods. For a modern distillation task, the training objective for different methods can be simply formulated as follows: L total = αL cross−entropy + βL kl−divergence + γL distill−f orms .
where α, β and γ are balancing hyper-parameters. For CIFAR-100, we default set α = 0.1 and β = 0.9. For ImageNet, we default set α = 1 and β = 1. The results in Table 3 and Table 4 are reproduced according to the official implementation with the following hyper-parameters:  Figure 6: The detailed architecture of our instance-wise temperature method. B denotes the batch size, C denotes the number of classes, and C inter denotes the number of intermediate channels in the MLP.  • SRRL (Yang et al. 2021): γ = 1.