190

# Continuous Depth models and a brief overview of Neural ODE

Talked about the basics of continuous depth models and a primer of Neural ODE at an internal presentation of SketchX group, CVSSP, University of Surrey. ## Ayan Das

September 02, 2021

## Transcript

1. Foundation of Countinuous Depth ODE, Interpretation & Applications
Foundation of continuous-depth models
.. and a brief introduction to Neural ODE
Ayan Das
PhD Student
SketchX, CVSSP
SketchX meetup talk
September 2, 2021
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 1 / 15

2. Foundation of Countinuous Depth ODE, Interpretation & Applications
Presentation Overview
1 Foundation of Countinuous Depth
2 ODE, Interpretation & Applications
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 1 / 15

3. Foundation of Countinuous Depth ODE, Interpretation & Applications
Foundation of Countinuous Depth
1 Foundation of Countinuous Depth
Motivation
Revisiting ResNet
Modifying ResNet
2 ODE, Interpretation & Applications
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 1 / 15

4. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Deep architectures so far
Deep models so far
1 Notion of depth is discrete
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 2 / 15

5. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Deep architectures so far
Deep models so far
1 Notion of depth is discrete
2 Parameter count increases with number of layers
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 2 / 15

6. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Deep architectures so far
Deep models so far
1 Notion of depth is discrete
2 Parameter count increases with number of layers
3 Backpropagation takes O(L) memory
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 2 / 15

7. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Deep architectures so far
Deep models so far
1 Notion of depth is discrete
2 Parameter count increases with number of layers
3 Backpropagation takes O(L) memory
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 2 / 15

8. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Design Motivation
Can we have ?
1 Continuous notion of Depth
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 3 / 15

9. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Design Motivation
Can we have ?
1 Continuous notion of Depth
2 Input “evolves” along depth
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 3 / 15

10. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Design Motivation
Can we have ?
1 Continuous notion of Depth
2 Input “evolves” along depth
3 Low cost Backpropagation
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 3 / 15

11. Foundation of Countinuous Depth ODE, Interpretation & Applications
Motivation
Design Motivation
Can we have ?
1 Continuous notion of Depth
2 Input “evolves” along depth
3 Low cost Backpropagation
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 3 / 15

12. Foundation of Countinuous Depth ODE, Interpretation & Applications
Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 4 / 15

13. Foundation of Countinuous Depth ODE, Interpretation & Applications
Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)
For each layer l
xl+1 ← xl + g(xl)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 4 / 15

14. Foundation of Countinuous Depth ODE, Interpretation & Applications
Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 4 / 15

15. Foundation of Countinuous Depth ODE, Interpretation & Applications
Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···
xl+1 ← xl + g(xl )
xl+2 ← xl+1 + g(xl+1 )
xl+3 ← xl+2 + g(xl+2 )
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 4 / 15

16. Foundation of Countinuous Depth ODE, Interpretation & Applications
Revisiting ResNet
Generic ResNet
ResNet motivated the development of Neural ODEs
ResNet Block (Isolated)
For each layer l
xl+1 ← xl + g(xl)
xl ∈ RM×N×···
xl+1 ∈ RM×N×···
xl+1 ← xl + g(xl; Θ1)
xl+2 ← xl+1 + g(xl+1; Θ2)
xl+3 ← xl+2 + g(xl+2; Θ3)
Remeber !
They have different parameters, where majority of the
ResNet’s modelling capacity lies
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 4 / 15

17. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure
.. for countinuous depth models
Precisely, three changes .. x1 ← x0 + g(x0; Θ1)
x2 ← x1 + g(x1; Θ2)
x3 ← x2 + g(x2; Θ3)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 5 / 15

18. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure
.. for countinuous depth models
Precisely, three changes ..
1 Scale g(·) with a scalar ∆l
→ Quite harmless, g(·) can always adjust by learning to
scale it up
x1 ← x0 + g(x0; Θ1)·∆l
x2 ← x1 + g(x1; Θ2)·∆l
x3 ← x2 + g(x2; Θ3)·∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 5 / 15

19. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure
.. for countinuous depth models
Precisely, three changes ..
1 Scale g(·) with a scalar ∆l
→ Quite harmless, g(·) can always adjust by learning to
scale it up
2 Share parameters accross blocks
→ Reduces number of parameters, hence modelling
capacity
x1 ← x0 + g(x0; Θ)·∆l
x2 ← x1 + g(x1; Θ)·∆l
x3 ← x2 + g(x2; Θ)·∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 5 / 15

20. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure
.. for countinuous depth models
Precisely, three changes ..
1 Scale g(·) with a scalar ∆l
→ Quite harmless, g(·) can always adjust by learning to
scale it up
2 Share parameters accross blocks
→ Reduces number of parameters, hence modelling
capacity
3 much more blocks than usual
→ To compensate model capacity, we want more blocks
x1 ← x0 + g(x0; Θ)·∆l
x2 ← x1 + g(x1; Θ)·∆l
x3 ← x2 + g(x2; Θ)·∆l
.
.
.
x4 ← x3 + g(x3; Θ) · ∆l
x5 ← x4 + g(x4; Θ) · ∆l
x6 ← x5 + g(x5; Θ) · ∆l
.
.
.
xL ← xL−1 + g(xL−1; Θ) · ∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 5 / 15

21. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 + g(x0; Θ) · ∆l + g(x1; Θ) · ∆l + · · · + g(xL−1; Θ) · ∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 6 / 15

22. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 6 / 15

23. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
Can we have infinitely long ResNet ?
i.e., L → ∞
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 6 / 15

24. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
Can we have infinitely long ResNet ?
i.e., L → ∞
The summation might blow up (in either direction)
−∞ ← xL → +∞
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 6 / 15

25. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Exploiting ResNet structure (Continued)
With input x0, we want to compute output at Lth block:
xL ← x0 +
L−1
l=0
g(xl; Θ) · ∆l
Can we have infinitely long ResNet ?
i.e., L → ∞
The summation might blow up (in either direction)
−∞ ← xL → +∞
Solution
Simulteneously do the following:
L → ∞
∆l → 0
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 6 / 15

26. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Emergence of Continuous Depth
In limiting case:
xL = x0+ lim
L→∞
∆l→0
L−1
l=0
g(xl; Θ) · ∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 7 / 15

27. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Emergence of Continuous Depth
In limiting case:
xL = x0+ lim
L→∞
∆l→0
L−1
l=0
g(xl; Θ) · ∆l
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 7 / 15

28. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Emergence of Continuous Depth
In limiting case:
xL = x0+ lim
L→∞
∆l→0
L−1
l=0
g(xl; Θ) · ∆l
xL = x0+
L
0
g(xl; Θ)dl
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 7 / 15

29. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth ResNet
A new kind of ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 8 / 15

30. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth ResNet
A new kind of ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 8 / 15

31. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth ResNet
A new kind of ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Parameterized by two things
1 Internal function g, along with Θ
2 Integration limits, i.e.
l2
l1
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 8 / 15

32. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth ResNet
A new kind of ResNet
x → x +
L
0
g(x; Θ)dl → y
F : x → y
Parameterized by two things
1 Internal function g, along with Θ
2 Integration limits, i.e.
l2
l1
Surprised ?
L can be fraction. So, L = 1.234 layer network (?)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 8 / 15

33. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth RNN
Intermediate values matter
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 9 / 15

34. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth RNN
Intermediate values matter
x +
l1 or l2 or l3
0
g(x; Θ)dl
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 9 / 15

35. Foundation of Countinuous Depth ODE, Interpretation & Applications
Modifying ResNet
Continuous Depth RNN
Intermediate values matter
x +
l1 or l2 or l3
0
g(x; Θ)dl
Unlike RNN, we can produce intermediate values at any (non-uniform) interval
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 9 / 15

36. Foundation of Countinuous Depth ODE, Interpretation & Applications
ODE, Interpretation & Applications
1 Foundation of Countinuous Depth
2 ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Interpretation
Potential applications
Practical Usage
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 9 / 15

37. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Where is the Differential Equation?
Neural Ordinary Differential Equation1
y = x +
L
0
g(x; Θ)dl
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 10 / 15

38. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Where is the Differential Equation?
Neural Ordinary Differential Equation1
y = F(l = 0) +
L
0
g(x; Θ)dl
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 10 / 15

39. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Where is the Differential Equation?
Neural Ordinary Differential Equation1
F(l = L) = F(l = 0) +
L
0
g(x; Θ)dl
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 10 / 15

40. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Where is the Differential Equation?
Neural Ordinary Differential Equation1
F(l = L) = F(l = 0) +
L
0
∂F
∂l
dl
A popular ODE solver called Eular Integration
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 10 / 15

41. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Where is the Differential Equation?
Neural Ordinary Differential Equation1
F(l = L) = F(l = 0) +
L
0
∂F
∂l
dl
A popular ODE solver called Eular Integration
Proposed first in NeurIPS 2018 (best) paper titled ”Neural ODE”1.
Parameterize the derivative as a Neural Network:
∂F
∂l
= NeuralNet(x; Θ)
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 10 / 15

42. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Backpropagation
Naive Backpropagation
How to we backpropagate ?
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 11 / 15

43. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Backpropagation
Naive Backpropagation
How to we backpropagate ?
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y
Why not just discretize and backpropagate trivially ?
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 11 / 15

44. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Backpropagation
Naive Backpropagation
How to we backpropagate ?
∂(loss)
∂x
,
∂(loss)
∂Θ
← x +
L
0
∂F
∂l
dl ←
∂(loss)
∂y
Why not just discretize and backpropagate trivially ?
Nope, don’t do it!
1 Introduces discretization error
2 Memory consumption blows up
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 11 / 15

45. Foundation of Countinuous Depth ODE, Interpretation & Applications
Neural Ordinary Differential Equation
Backpropagation
x → x +
L
0
∂F
∂l
dl → y
∂(loss)
∂x
,
∂(loss)
∂Θ
∂(loss)
∂y
1“Neural Ordinary Differential Equation” by Ricky T. Q. Chen et al. (https://arxiv.org/pdf/1806.07366.pdf)
2https://ayandas.me/blog-tut/2020/03/20/neural-ode.html
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 12 / 15

46. Foundation of Countinuous Depth ODE, Interpretation & Applications
Interpretation
Standard Interpretation
.. from calculus
Initial Value Problem
1
∂F
∂l
= NeuralNet(x; Θ) is a vector field
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 13 / 15

47. Foundation of Countinuous Depth ODE, Interpretation & Applications
Interpretation
Standard Interpretation
.. from calculus
Initial Value Problem
1
∂F
∂l
= NeuralNet(x; Θ) is a vector field
2 x → y is its solution trajectory by solving the IVP
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 13 / 15

48. Foundation of Countinuous Depth ODE, Interpretation & Applications
Interpretation
Standard Interpretation
.. from calculus
Initial Value Problem
1
∂F
∂l
= NeuralNet(x; Θ) is a vector field
2 x → y is its solution trajectory by solving the IVP
Fitting
By changing Θ, we can fit a given x0 & Y
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 13 / 15

49. Foundation of Countinuous Depth ODE, Interpretation & Applications
Interpretation
Standard Interpretation
.. from calculus
Initial Value Problem
1
∂F
∂l
= NeuralNet(x; Θ) is a vector field
2 x → y is its solution trajectory by solving the IVP
Fitting
By changing Θ, we can fit a given x0 & Y
Supervise intermediate values to fit a whole
trajectory
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 13 / 15

50. Foundation of Countinuous Depth ODE, Interpretation & Applications
Potential applications
Lifespan synthesis1
A possible appraoch
1https://grail.cs.washington.edu/projects/lifespan age transformation synthesis/
Exploring the notion of continuous-depth & a brief overview of Neural ODE Copyright © 2021 Ayan Das (@dasayan05) 14 / 15

51. Foundation of Countinuous Depth ODE, Interpretation & Applications
Practical Usage
Practical Implementation (torchdiffeq1)
.. in PyTorch code
# libraries like ’torchdiffeq’
{"ft_lang_label":"__label__en","ft_lang_prob":0.69586045,"math_prob":0.82796204,"size":20094,"snap":"2023-40-2023-50","text_gpt3_token_len":6001,"char_repetition_ratio":0.21936287,"word_repetition_ratio":0.82713413,"special_character_ratio":0.26604956,"punctuation_ratio":0.08558823,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.98309565,"pos_list":[0,1,2],"im_url_duplicate_count":[null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-12-09T15:16:09Z\",\"WARC-Record-ID\":\"<urn:uuid:61dd2395-ccdb-4bf5-a425-f6a13827ca18>\",\"Content-Length\":\"189567\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:25186a0d-67f8-4e0e-98ff-0d4a0cdb2119>\",\"WARC-Concurrent-To\":\"<urn:uuid:8634c0f4-7e88-463e-b343-fd7caa03cecd>\",\"WARC-IP-Address\":\"104.22.39.239\",\"WARC-Target-URI\":\"https://speakerdeck.com/dasayan05/continuous-depth-models-and-a-brief-overview-of-neural-ode\",\"WARC-Payload-Digest\":\"sha1:MJ2N24YZOGA5GYJT7BEYOLOY2AQJUTWA\",\"WARC-Block-Digest\":\"sha1:SJ6EPGJQM5NIN26K4K3ODRSDZXX443WM\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-50/CC-MAIN-2023-50_segments_1700679100912.91_warc_CC-MAIN-20231209134916-20231209164916-00039.warc.gz\"}"}