Mathematical Background Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj
Outline Norms Analysis Functions Derivatives Linear Algebra
Inner product � Inner product on � � � � � ��� Euclidean norm, or � -norm � � �/� � �/� � � � � Cauchy-Schwartz inequality � � � � � Angle between nonzero vectors 𝑦 � 𝑧 ∠�𝑦, 𝑧� � cos �� , 𝑦, 𝑧 ∈ 𝐒 � 𝑦 � 𝑧 �
Inner product ��� , ��� Inner product on � � ⟨𝑌, 𝑍⟩ � tr 𝑌 � 𝑍 � � � 𝑌 �� 𝑍 �� ��� ��� Here tr� � denotes trace of a matrix. ��� Frobenius norm of a matrix �/� � � �/� � � 𝑌 � � tr 𝑌 � 𝑌 � � 𝑌 �� ��� ��� � Inner product on � � � ⟨𝑌, 𝑍⟩ � tr�𝑌𝑍� � � � 𝑌 �� 𝑍 �� � � 𝑌 �� 𝑍 �� � 2 � 𝑌 �� 𝑍 �� ��� ��� ��� ���
Norms A function 𝑔: 𝐒 � → 𝐒 with dom 𝑔 � 𝐒 � is called a norm if 𝑔 is nonnegative: 𝑔�𝑦� � 0 for all 𝑦 ∈ 𝐒 � 𝑔 is definite: 𝑔�𝑦� � 0 only if 𝑦 � 0 𝑔 is homogeneous: 𝑔�𝑢𝑦� � |𝑢|𝑔�𝑦�, for all 𝑦 ∈ 𝐒 � and 𝑢 ∈ 𝐒 𝑔 satisfies the triangle inequality: 𝑔�𝑦 � 𝑧� � 𝑔�𝑦� � 𝑔�𝑧�, for all 𝑦, 𝑧 ∈ 𝐒 � Distance Between vectors 𝑦 and 𝑧 as the length of their difference, i.e., dist�𝑦, 𝑧� � 𝑦 � 𝑧
Norms Unit ball The set of all vectors with norm less than or equal to one, ℬ � 𝑦 ∈ 𝐒 � | 𝑦 � 1 is called the unit ball of the norm ∥⋅∥ . The unit ball satisfies the following properties: ℬ is symmetric about the origin, i.e., 𝑦 ∈ ℬ if and only if � 𝑦 ∈ ℬ ℬ is convex ℬ is closed, bounded, and has nonempty interior Conversely, if 𝐷 ⊆ 𝐒 � is any set satisfying these three conditions, the it is the unit ball of a norm: � �sup 𝑢 � 0 𝑢𝑦 ∈ 𝐷 � �� 𝑦
Norms � Some common norms on Sum-absolute-value, or � -norm � � � � Chebyshev or � -norm � � � � -norm � � � � �/� � � , -quadratic norm is For �� � �/� �/� � �
Norms ��� Some common norms on Sum-absolute-value norm � � ��� �� ��� ��� Maximum-absolute-value norm ��� ��
Norms Equivalence of norms Suppose that � and � are norms � , there exist positive constants on � and , for all � � � � , then there If is any norm on exists a quadratic norm � for which � � holds for all .
Norms Operator norms � Suppose � and � are norms on � , respectively. Operator norm of and ��� induced by � and � is �,� � � When � and � are Euclidean norms, the operator norm of is its maximum singular value, and is denoted � �/� 𝑌 � � 𝜏 ��� �𝑌� � 𝜇 ��� 𝑌 � 𝑌 Spectral norm or ℓ � -norm
Norms Operator norms The norm induced by the ℓ � -norm on 𝐒 � and 𝐒 � , denoted 𝑌 � , is the max-row-sum norm, � 𝑌 � � sup 𝑌𝑣 � | 𝑣 � � 1 � max ���,…,� � 𝑌 �� ��� The norm induced by the ℓ � -norm on 𝐒 � and 𝐒 � , denoted 𝑌 � , is the max-column-sum norm, � 𝑌 � � max ���,…,� ∑ 𝑌 �� ���
Norms Dual norm � . Let be a norm on The associated dual norm, denoted ∗ , is defined as � ∗ We have the inequality � ∗ The dual of Euclidean norm sup 𝑨 � 𝑦| 𝑦 � � 1 � 𝑨 � The dual of the � -norm sup 𝑨 � 𝑦| 𝑦 � � 1 � 𝑨 �
Norms Dual Norm The dual of � -norm is the � -norm such that ��� is the The dual of the � -norm on nuclear norm � �∗ � � �/� � �
Outline Norms Analysis Functions Derivatives Linear Algebra
Analysis Interior and Open Set An element 𝑦 ∈ 𝐷 ⊆ 𝐒 � is called an interior point of 𝐷 if there exists an 𝜗 � 0 for which 𝑧 𝑧 � 𝑦 � � 𝜗� ⊆ 𝐷 i.e., there exists a ball centered at 𝑦 that lies entirely in 𝐷 . The set of all points interior to 𝐷 is called the interior of 𝐷 and is denoted int 𝐷 . A set is open if
Analysis Closed Set and Boundary � A set is closed if its complement is open 𝐒 � ∖ 𝐷 � 𝑦 ∈ 𝐒 � |𝑦 ∉ 𝐷 The closure of a set 𝐷 is defined as cl 𝐷 � 𝐒 � ∖ int�𝐒 𝐨 ∖ 𝐷� The boundary of the set 𝐷 is defined as bd 𝐷 � cl 𝐷 ∖ int 𝐷 𝐷 is closed if it contains its boundary. It is open if it contains no boundary points.
Analysis Supremum and infimum The least upper bound or supremum of the set is denoted . The greatest lower bound or infimum of the set is denoted .
Outline Norms Analysis Functions Derivatives Linear Algebra
� An example �� � Functions Notation
Functions Continuity � is continuous at � A function if for all there exists a with , such that � � Closed functions � A function is closed if, for each , the sublevel set is closed. This is equivalent to ���
Outline Norms Analysis Functions Derivatives Linear Algebra
Derivatives Definition � and � Suppose . The function is differentiable at if there ��� that satisfies exists a matrix 𝑔 𝑨 � 𝑔 𝑦 � 𝐸𝑔 𝑦 𝑨 � 𝑦 � lim � 0 𝑨 � 𝑦 � �∈��� �, ���, �→� in which case we refer to as the derivative (or Jacobian) of at .
Derivatives Definition The affine function of given by is called the first-order approximation of at (or near) . � �� �
Derivatives Gradient When 𝑔 is real-valued (i.e., 𝑔: 𝐒 � → 𝐒 ) the derivative 𝐸𝑔�𝑦� is a 1 � 𝑜 matrix (it is a row vector). Its transpose is called the gradient of the function: 𝛼𝑔�𝑦� � 𝐸𝑔�𝑦� � which is a column vector (in 𝐒 � ). Its components are the partial derivatives of 𝑔 : 𝛼𝑔�𝑦� � � 𝜖𝑔�𝑦� , 𝑗 � 1, ⋯ , 𝑜 𝜖𝑦 � The first-order approximation of 𝑔 at a point 𝑦 ∈ int dom 𝑔 can be expressed as (the affine function of 𝑨 ) 𝑔�𝑦� � 𝛼𝑔�𝑦� � �𝑨 � 𝑦�
Derivatives Examples 𝑔 𝑦 � 1 2 𝑦 � 𝑄𝑦 � 𝑟 � 𝑦 � 𝑠 𝛼𝑔 𝑦 � 𝑄𝑦 � 𝑟 � 𝑔 𝑌 � log det 𝑌 , dom 𝑔 � 𝐓 �� 𝛼𝑔 𝑌 � 𝑌 ��
Derivatives Chain rule Suppose 𝑔: 𝐒 � → 𝐒 � is differentiable at 𝑦 ∈ int dom 𝑔 and : 𝐒 � → 𝐒 � is differentiable at 𝑔�𝑦� ∈ int dom . Define the composition ℎ: 𝐒 � → 𝐒 � by ℎ�𝑨� � �𝑔�𝑨�� . Then ℎ is differentiable at 𝑦 , with derivate 𝐸ℎ�𝑦� � 𝐸�𝑔�𝑦��𝐸𝑔�𝑦� Suppose 𝑔: 𝐒 � → 𝐒 , : 𝐒 → 𝐒 , and ℎ 𝑦 � �𝑔 𝑦 � 𝛼ℎ 𝑦 � � 𝑔 𝑦 𝛼𝑔�𝑦�
Derivatives Composition of Affine Function 𝑦 � 𝑔�𝐵𝑦 � 𝑐� 𝛼 𝑦 � 𝐵 � 𝛼𝑔�𝐵𝑦 � 𝑐� 𝑔: 𝐒 � → 𝐒, : 𝐒 → 𝐒 𝑦, 𝑤 ∈ 𝐒 � 𝑢 � 𝑔 𝑦 � 𝑢𝑤 , ′ 𝑢 � 𝑤 � 𝛼𝑔 𝑦 � 𝑢𝑤
Example 1 � Consider the function � � 𝑦 � 𝑐 � � 𝑔 𝑦 � log � exp �𝑏 � ��� � where � � � 𝑧 � log � exp �𝑧 � � ��� exp 𝑧 � 1 ⋮ 𝛼 𝑧 � � ∑ exp 𝑧 � exp 𝑧 � ���
Example 1 � Consider the function � � 𝑦 � 𝑐 � � 𝑔 𝑦 � log � exp �𝑏 � ��� � where � � 1 𝛼𝑔 𝑦 � 𝐵 � 𝛼 𝐵𝑦 � 𝑐 � 1 � 𝑨 𝐵 � 𝑨 � 𝑦 � 𝑐 � exp 𝑏 � 𝑨 � ⋮ � 𝑦 � 𝑐 � exp 𝑏 �
Example 2 Consider the function � � � � � � where � � � � � � � �� � � � �� � �� �
Second Derivative Definition Suppose 𝑔: 𝐒 � → 𝐒 . The second derivative or Hessian matrix of 𝑔 at 𝑦 ∈ int dom 𝑔 , denoted 𝛼 � 𝑔�𝑦� , is given by 𝛼 � 𝑔�𝑦� �� � 𝜖 � 𝑔�𝑦� , 𝑗 � 1, ⋯ , 𝑜, 𝑘 � 1, ⋯ , 𝑜. 𝜖𝑦 � 𝜖𝑦 � Second-order Approximation 𝑔�𝑦� � 𝛼𝑔�𝑦� � 𝑨 � 𝑦 � 1 2 𝑨 � 𝑦 � 𝛼 � 𝑔�𝑦��𝑨 � 𝑦�
Recommend
More recommend
