# Chapter 2 Linear Algebra

broadcasting: We allow the addition of a matrix and a vector, yielding another matrix: $\boldsymbol{C} = \boldsymbol{A} + \boldsymbol{b}$ where $C_{i,j} = A_{i,j} + b_j$. In other words, the vector $\textbf{b}$ is added to each row of the matrix. This shorthand eliminates the need to define a matrix with $\textbf{b}​$ copied into each row before doing the addition.

Multiplying Matrices: We can think of the matrix product $\boldsymbol{C} = \boldsymbol{AB}​$ as computing $\it{C_{i,j}}​$ as the dot product between row $i​$ of $\boldsymbol{A}​$ and column $j​$ of$\boldsymbol{B}​$.

Range of A: Determining whether Ax=b has a solution thus amounts to testing whether b is in the span of the columns of A.

$$\boldsymbol{Ax} = [\sum_i(\sum_j \boldsymbol{A}_{i,j}x_j)] = \sum_i\boldsymbol{A}_{:,i}x_i$$

# Chapter 3 概率与信息论

Multinoulli Distribution: If you perform an experiment that can have K outcomes and you denote by $X_i$ a random variable that takes value 1 if you obtain the i-th outcome and 0 otherwise.

$$I(x) = -\log P(x)$$

$$H(x) = \mathbb{E}_{x\sim P}[I(x)] = -\mathbb E {x\sim P}[\log P(x)]$$
KL散度：如果对于同一个随机变量x有两个单独的概率分布P(x) 和 Q(x) ，那么可以使用KL散度来衡量这两个分布的差异:
$$D {KL}(P||Q) = \mathbb E _{x\sim P} \Big[\log \frac{P(x)}{Q(x)} \Big] = \mathbb E {x\sim P}[\log P(x) - \log Q(x)]$$

$$H(P,Q)= H(P) + D_{KL}(P||Q)$$

# chapter 4 数值计算

Optimization: refers to the task of either maximizing or minimizing some function $f(\boldsymbol{x})$ by altering $\boldsymbol x$

Partial derivative: for functions with multiple inputs, the partial derivative measures how $f$ changes as only the variable $x_i$ increases at point $\boldsymbol x$.

Gradient: the gradient of $f$ is the vector containing all the partial derivatives, denoted $\nabla _\boldsymbol x f(\boldsymbol x)$.

0%
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 1, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.9608737826347351, "perplexity": 302.89960171724414}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2019-43/segments/1570986673538.21/warc/CC-MAIN-20191017095726-20191017123226-00083.warc.gz"}