DeepLearning-Notations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Notations\n",
    "by [Elvis Saravia](http://elvissaravia.com/)\n",
    "\n",
    "---\n",
    "**Aim:** This notebook contains useful notations widely used in deep learning papers and educational materials found online. I used similar notations used in the Deep Learning book written by Ian Goodfellow, Yoshua Bengio and Aaron Courville. I will also provide sample code using PyTorch to show the type of data structures and concepts these notation may represent.\n",
    "\n",
    "**Uses:** You can reuse the notations in this notebook as a cheatsheet to assist you in writing your research papers, presentations, and blogs. It's also good resource for reviewing important mathematical notations used widely in deep learning research and other related fields. I provide example code in PyTorch but as an exercise, you can try generating similar code using Numpy or Tensorflow. (The code shouldn't be too different.)\n",
    "\n",
    "**Requirements:** [PyTorch](http://pytorch.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Number and Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$a$ - a scalar (integer or real)   \n",
    "Latex: `$a$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "a = 2\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol a$ - a vector  \n",
    "Latex: `$\\boldsymbol a$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 1\n",
      " 2\n",
      "[torch.FloatTensor of size 2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### 1D vector (column vector)\n",
    "a = torch.Tensor([1,2]) \n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 1  2\n",
      "[torch.FloatTensor of size 1x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### 1D vector (row form)\n",
    "a = torch.Tensor([[1,2]])\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A$ - a matrix  \n",
    "Latex: `$\\boldsymbol A$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 1  2  4\n",
      " 4  5  6\n",
      "[torch.FloatTensor of size 2x3]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.Tensor([[1,2,4],[4,5,6]])\n",
    "print(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathsf A$ - a tensor  \n",
    "Latex: `$\\mathsf A$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(0 ,.,.) = \n",
      "  1  2\n",
      "  3  4\n",
      "\n",
      "(1 ,.,.) = \n",
      "  5  6\n",
      "  7  8\n",
      "[torch.FloatTensor of size 2x2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.Tensor([[[1., 2.], [3., 4.]],\n",
    "                  [[5., 6.], [7., 8.]]])\n",
    "print(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identity matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol I_n$ - identity matrix with $n$ rows and $n$ columns  \n",
    "Latex: `$\\boldsymbol I_n$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 1  0  0  0\n",
      " 0  1  0  0\n",
      " 0  0  1  0\n",
      " 0  0  0  1\n",
      "[torch.FloatTensor of size 4x4]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "I = torch.eye(4)\n",
    "print(I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard Basis Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol e^{(i)}$ - standard basic vector $[0,...,0,1,0,...,0]$ with a 1 at position $i$  \n",
    "Latex: `$\\boldsymbol e^{(i)}$ `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 0\n",
      " 1\n",
      " 0\n",
      " 0\n",
      " 0\n",
      "[torch.FloatTensor of size 9]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "i = 5 # index\n",
    "e = torch.zeros(9)\n",
    "e[i]=1\n",
    "print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Diagonal Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\text{diag}(\\boldsymbol a)$ - A square, diagonal matrix with diagonal entries given by $\\boldsymbol a$  \n",
    "Latex: `$\\text{diag}(\\boldsymbol a)$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "-0.1878  0.0000  0.0000  0.0000\n",
       " 0.0000 -0.6244  0.0000  0.0000\n",
       " 0.0000  0.0000 -0.5755  0.0000\n",
       " 0.0000  0.0000  0.0000 -0.4487\n",
       "[torch.FloatTensor of size 4x4]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.diag(torch.randn(4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\rm a$ - a scalar random variable  \n",
    "Latex: `$\\rm a$`\n",
    "\n",
    "$\\bf a$ - a vector-valued random variable  \n",
    "Latex: `$\\bf a$`\n",
    "\n",
    "$\\rm {a_i}$ - element $i$ of the random vector $\\bf a$  \n",
    "Latex: `$\\rm {a_i}$`\n",
    "\n",
    "$\\bf A$ - a matrix-valued random variable  \n",
    "Latex: `$\\bf A$`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sets and Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathbb{A}$ - a set  \n",
    "Latex: `$\\mathbb{A}$`\n",
    "\n",
    "$\\mathbb{R}$ - the set of real numbers  \n",
    "Latex: `$\\mathbb{R}$`\n",
    "\n",
    "$\\{ 0,1\\}$ - the set containing $0$ and $1$  \n",
    "Latex: `$\\{ 0,1\\}$`\n",
    "\n",
    "$\\{ 0,1,...,n\\}$ - the set of all integers between $0$ and $n$  \n",
    "Latex: `$\\{ 0,1,...,n\\}$`\n",
    "\n",
    "$\\left[ a, b\\right]$ - the real interval including $a$ and $b$  \n",
    "Latex:  `$\\left[ a, b\\right]$`\n",
    "\n",
    "$(a,b ]$ - the real interval excluding $a$ but including $b$  \n",
    "Latex: `$(a,b ]$`\n",
    "\n",
    "$\\mathbb{A} \\backslash \\mathbb{B}$ - set substraction, i.e., the set containing the elements of $\\mathbb{A}$ that are not in $\\mathbb{B}$  \n",
    "Latex: `$\\mathbb{A} \\backslash \\mathbb{B}$`\n",
    "\n",
    "$\\mathcal{G}$ - a graph  \n",
    "Latex: `$\\mathcal{G}$`\n",
    "\n",
    "$Pa_{\\mathcal{G}}(\\rm x_{i})$ - the parents of $\\rm x_{i}$ in $\\mathcal{G}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$a_i$ - the i-th element of a vector (indexing starting at 0)   \n",
    "Latex: `$a_i$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0\n"
     ]
    }
   ],
   "source": [
    "i = 1\n",
    "a = torch.Tensor([1,2,3,4,5])\n",
    "print(a[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$a_{-i}$ - all elements of vector $\\boldsymbol a$ except for element $i$  \n",
    "Latex: `$a_{-i}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.0, 2.0, 4.0, 5.0]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 2 # element 3\n",
    "[b for b in a if b != a[i]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$A_{ij}$ - element $i,j$ of a matrix $\\boldsymbol A$  \n",
    "Latex: `$A_{ij}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-1.7500  1.1511 -0.5790  1.2555\n",
      "-0.1847  0.1199 -2.0554 -0.1277\n",
      " 0.7025  0.2841  0.4543  0.2571\n",
      " 1.0044  0.3005 -0.3800 -0.9919\n",
      "[torch.FloatTensor of size 4x4]\n",
      " 0.454309344291687\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn((4,4))\n",
    "i, j = 2,2\n",
    "print(A, A[i][j])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A_{i,:}$ - row $i$ of matrix $\\boldsymbol A$  \n",
    "Latex: `$A_{i,:}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 0.7025\n",
       " 0.2841\n",
       " 0.4543\n",
       " 0.2571\n",
       "[torch.FloatTensor of size 4]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 2 # i.e., row 3\n",
    "A[2,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A_{:,i}$ - column $i$ of matrix $\\boldsymbol A$  \n",
    "Latex: `$\\boldsymbol A_{:,i}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "-0.5790\n",
       "-2.0554\n",
       " 0.4543\n",
       "-0.3800\n",
       "[torch.FloatTensor of size 4]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 2 # i.e., column 3\n",
    "A[:,i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathsf A_{i,j,k}$ - element $(i,j,k)$ of a 3-D tensor $\\mathsf A$  \n",
    "Latex: `$\\mathsf A_{i,j,k}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "(0 ,.,.) = \n",
      " -0.5824  1.4592 -0.7219\n",
      "  1.8931  2.7500 -0.9853\n",
      "\n",
      "(1 ,.,.) = \n",
      "  0.5265 -0.5071 -0.8086\n",
      " -0.1803  0.2604 -0.3469\n",
      "[torch.FloatTensor of size 2x2x3]\n",
      "\n",
      "-0.3468764126300812\n"
     ]
    }
   ],
   "source": [
    "i, j , k = 1,1,2 \n",
    "A = torch.randn((2,2,3))\n",
    "print(A)\n",
    "print(A[i, j ,k])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\mathsf A_{:,:,i}$ - 2-D slice of a 3-D tensor  \n",
    "Latex: `$\\mathsf A_{:,:,i}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "-0.7219 -0.9853\n",
       "-0.8086 -0.3469\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = 2 \n",
    "A[:,:,i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Algebra Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A^\\top $ - transpose of matrix $\\boldsymbol A$  \n",
    "Latex: `$\\boldsymbol A^\\top $`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-0.5358  0.0759\n",
      " 0.1394 -0.8398\n",
      "-1.1586 -0.2389\n",
      "[torch.FloatTensor of size 3x2]\n",
      "\n",
      "\n",
      "-0.5358  0.1394 -1.1586\n",
      " 0.0759 -0.8398 -0.2389\n",
      "[torch.FloatTensor of size 2x3]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn((3,2))\n",
    "print(A)\n",
    "print(A.t())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A^+$ - the  Moore-Penrose pseudoinverse pseudoinverse of matrix $\\boldsymbol A$  \n",
    "Latex: `$\\boldsymbol A^+$`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A^{-1}$ - the inverse matrix of the square matrix $\\boldsymbol A$  \n",
    "Latex: `$\\boldsymbol A^{-1}$`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-1.5843 -1.2536\n",
      "-1.6495  1.2179\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n",
      "\n",
      "-0.3047 -0.3136\n",
      "-0.4127  0.3963\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn((2,2))\n",
    "print(A)\n",
    "print(torch.inverse(A))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\boldsymbol A \\bigodot \\boldsymbol B$ - element-wise (Hadamard) product of $\\boldsymbol A$ and $\\boldsymbol B$  \n",
    "Latex: `$\\boldsymbol A \\bigodot \\boldsymbol B$ `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-0.0398 -0.7343\n",
      " 0.5065 -0.2811\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "A = torch.randn((2,2))\n",
    "B = torch.randn((2,2))\n",
    "print(A.mul(B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\text{det}(\\boldsymbol A)$ - determinant of $\\boldsymbol A$  \n",
    "Latex: `$\\text{det}(\\boldsymbol A)$ `"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "- [Latex commands for MathJax](http://www.onemathematicalcat.org/MathJaxDocumentation/TeXSyntax.htm)\n",
    "- [Deep Learning Book](http://www.deeplearningbook.org/)\n",
    "- [Deep Learning Book (Official Notations)](https://github.com/goodfeli/dlbook_notation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}