q_learning_fa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-Learning with Kernel Function Approximation\n",
    "\n",
    "This notebook shows the function approximation reinforcement learning algorithm, Q-Learning applied to the MountainCar problem (it also works on Acrobat and CartPole) from Open AI's gym. This algorithm is designed to work on state spaces larger than the tabular Q-Learning algorithm. It achieves this by representing the states by feature vectors, $x(s) = (x_{1}(s), x_{2}(s), ..., x_{n}(s))^T$, which is a vector with smaller size than the set of all states. \n",
    "Now the action value function can be estimated with function approximation, given by:\n",
    "$$\\hat{q}(s, a, \\bold{w}) \\approx q_{\\pi}(s, a)$$\n",
    ", where $\\bold{w}$ is the parameter vector.\n",
    "\n",
    "The parameter vector can then be updated using gradient descent:\n",
    "$$\\bold{w}_{t+1} = \\bold{w}_{t} + \\alpha [R_{t+1} + \\gamma \\max_{a}\\hat{q}(S_{t + 1}, a, \\bold{w}_{t}) - \\hat{q}(S_{t}, A_{t}, \\bold{w}_{t})] \\nabla \\hat{q}(S_{t}, A_{t}, \\bold{w}_{t})$$\n",
    ", where $\\bold{w}$ is the parameter vector, $\\alpha$ is the step size, $\\gamma$ is the discount rate, $R$ is the reward, $\\nabla$ is the gradient operator, $S$ is the state, $A$ is the action and $\\hat{q}$ is the approximate action-value function.\n",
    "\n",
    "For more information on function approximation see section 10 of [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2018.pdf). For alternative implementations of Q-learning function approximation see: [Q-Learning Value Function Approximation Example by Denny Britz](https://github.com/dennybritz/reinforcement-learning/blob/master/FA/Q-Learning%20with%20Value%20Function%20Approximation%20Solution.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import gymnasium as gym\n",
    "import numpy as np\n",
    "import sklearn\n",
    "import sklearn.pipeline\n",
    "from sklearn.linear_model import SGDRegressor\n",
    "from sklearn.kernel_approximation import RBFSampler\n",
    "from IPython.display import Video\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type of environment, options include:\n",
    "# MountainCar-v0, Acrobot-v1, CartPole-v1\n",
    "ENV_TYPE = \"MountainCar-v0\"\n",
    "\n",
    "# Save videos from training\n",
    "SAVE_VIDEOS = False\n",
    "VIDEOS_ROOT_DIR = './videos/q_learning_fa/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the environment\n",
    "env = gym.make(ENV_TYPE, render_mode=\"rgb_array\")\n",
    "\n",
    "# Number of possible actions\n",
    "num_actions = env.action_space.n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Features Using Radial Basis Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>FeatureUnion(transformer_list=[(&#x27;rbf1&#x27;, RBFSampler(gamma=5.0)),\n",
       "                               (&#x27;rbf2&#x27;, RBFSampler(gamma=2.0)),\n",
       "                               (&#x27;rbf3&#x27;, RBFSampler()),\n",
       "                               (&#x27;rbf4&#x27;, RBFSampler(gamma=0.5))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;FeatureUnion<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.FeatureUnion.html\">?<span>Documentation for FeatureUnion</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>FeatureUnion(transformer_list=[(&#x27;rbf1&#x27;, RBFSampler(gamma=5.0)),\n",
       "                               (&#x27;rbf2&#x27;, RBFSampler(gamma=2.0)),\n",
       "                               (&#x27;rbf3&#x27;, RBFSampler()),\n",
       "                               (&#x27;rbf4&#x27;, RBFSampler(gamma=0.5))])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf1</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=5.0)</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf2</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=2.0)</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf3</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler()</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf4</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=0.5)</pre></div> </div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "FeatureUnion(transformer_list=[('rbf1', RBFSampler(gamma=5.0)),\n",
       "                               ('rbf2', RBFSampler(gamma=2.0)),\n",
       "                               ('rbf3', RBFSampler()),\n",
       "                               ('rbf4', RBFSampler(gamma=0.5))])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feature Preprocessing: Normalize to zero mean and unit variance\n",
    "# We use a few samples from the observation space to do this\n",
    "observation_examples = np.array([np.float64(env.observation_space.sample()) for x in range(10000)])\n",
    "scaler = sklearn.preprocessing.StandardScaler()\n",
    "scaler.fit(observation_examples)\n",
    "\n",
    "# Used to convert a state to a featurizes represenation.\n",
    "# We use RBF kernels with different variances to cover different parts of the space\n",
    "featurizer = sklearn.pipeline.FeatureUnion([\n",
    "        (\"rbf1\", RBFSampler(gamma=5.0, n_components=100)),\n",
    "        (\"rbf2\", RBFSampler(gamma=2.0, n_components=100)),\n",
    "        (\"rbf3\", RBFSampler(gamma=1.0, n_components=100)),\n",
    "        (\"rbf4\", RBFSampler(gamma=0.5, n_components=100))\n",
    "        ])\n",
    "featurizer.fit(scaler.transform(observation_examples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Action-Value Function Approximator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_features(state, scaler):\n",
    "    \"\"\"\n",
    "    Converts the state to a feature representation.\n",
    "    \"\"\"\n",
    "    scaled = scaler.transform([state])\n",
    "    featurized = featurizer.transform(scaled)\n",
    "    return featurized[0]    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of linear action-value fuction approximators (one for each action)\n",
    "Q = []\n",
    "\n",
    "for _ in range(num_actions):\n",
    "    linear_model = SGDRegressor(learning_rate=\"constant\")\n",
    "    # We need to call partial_fit once to initialize the model\n",
    "    # or we get a NotFittedError when trying to make a prediction\n",
    "    # This is quite hacky.\n",
    "    state, info = env.reset()\n",
    "    linear_model.partial_fit([convert_to_features(state, scaler)], [0])\n",
    "    Q.append(linear_model)    \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_returns(state, Q):\n",
    "    features = convert_to_features(state, scaler)\n",
    "    estimated_returns = []\n",
    "    for linear_model in Q:\n",
    "        estimated_returns.append(linear_model.predict([features])[0])\n",
    "\n",
    "    return np.array(estimated_returns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create An $\\epsilon$-Greedy Policy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_fn(state, Q, epsilon, num_actions):\n",
    "    # Create a distribution of actions and divide the epsilon probability between all actions\n",
    "    action_dist = np.ones(num_actions, dtype=float) * epsilon / num_actions\n",
    "    # Use function approximation to get the estimated returns for the actions in the current state\n",
    "    estimated_returns = predict_returns(state, Q)\n",
    "    # Find the best action\n",
    "    best_action = np.argmax(estimated_returns)\n",
    "    # Set probability for the best action to (1 - epsilon)\n",
    "    action_dist[best_action] += (1.0 - epsilon)\n",
    "    return action_dist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn The Optimal Action-Value Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu22/repos/reinforcement-learning-algorithms/venv-pre/lib/python3.10/site-packages/gymnasium/wrappers/rendering.py:282: UserWarning: \u001b[33mWARN: Overwriting existing videos at /home/ubuntu22/repos/reinforcement-learning-algorithms/videos/q_learning_fa folder (try specifying a different `video_folder` for the `RecordVideo` wrapper if this is not desired)\u001b[0m\n",
      "  logger.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Return: -3548.0\n",
      "Episode: 2 Return: -878.0\n",
      "Episode: 3 Return: -740.0\n",
      "Episode: 4 Return: -856.0\n",
      "Episode: 5 Return: -410.0\n",
      "Episode: 6 Return: -561.0\n",
      "Episode: 7 Return: -627.0\n",
      "Episode: 8 Return: -283.0\n",
      "Episode: 9 Return: -543.0\n",
      "Episode: 10 Return: -251.0\n",
      "Episode: 11 Return: -352.0\n",
      "Episode: 12 Return: -241.0\n",
      "Episode: 13 Return: -291.0\n",
      "Episode: 14 Return: -204.0\n",
      "Episode: 15 Return: -321.0\n",
      "Episode: 16 Return: -259.0\n",
      "Episode: 17 Return: -230.0\n",
      "Episode: 18 Return: -165.0\n",
      "Episode: 19 Return: -198.0\n",
      "Episode: 20 Return: -165.0\n",
      "Episode: 21 Return: -159.0\n",
      "Episode: 22 Return: -162.0\n",
      "Episode: 23 Return: -193.0\n",
      "Episode: 24 Return: -191.0\n",
      "Episode: 25 Return: -182.0\n",
      "Episode: 26 Return: -164.0\n",
      "Episode: 27 Return: -180.0\n",
      "Episode: 28 Return: -190.0\n",
      "Episode: 29 Return: -196.0\n",
      "Episode: 30 Return: -159.0\n",
      "Episode: 31 Return: -186.0\n",
      "Episode: 32 Return: -174.0\n",
      "Episode: 33 Return: -166.0\n",
      "Episode: 34 Return: -126.0\n",
      "Episode: 35 Return: -161.0\n",
      "Episode: 36 Return: -214.0\n",
      "Episode: 37 Return: -111.0\n",
      "Episode: 38 Return: -116.0\n",
      "Episode: 39 Return: -170.0\n",
      "Episode: 40 Return: -116.0\n",
      "Episode: 41 Return: -118.0\n",
      "Episode: 42 Return: -117.0\n",
      "Episode: 43 Return: -184.0\n"
     ]
    }
   ],
   "source": [
    "# Number of episodes to train on\n",
    "episodes = 43\n",
    "# Return (accumulation of all rewards over an episode)\n",
    "G = 0\n",
    "returns = []\n",
    "# Discount ratio\n",
    "gamma = 0.99\n",
    "# Probability of taking a non-greedy action\n",
    "epsilon = 0.001\n",
    "\n",
    "env = gym.make(ENV_TYPE, render_mode=\"rgb_array\")\n",
    "env = gym.wrappers.RecordEpisodeStatistics(env, buffer_length=episodes)\n",
    "\n",
    "if SAVE_VIDEOS:\n",
    "    os.makedirs(VIDEOS_ROOT_DIR, exist_ok=True)\n",
    "    env = gym.wrappers.RecordVideo(\n",
    "        env=env, \n",
    "        video_folder=VIDEOS_ROOT_DIR,\n",
    "        episode_trigger=lambda episode: episode % 10 == 0,\n",
    "        video_length=0,\n",
    "        name_prefix=ENV_TYPE,\n",
    "        fps=10\n",
    "        )\n",
    "\n",
    "\n",
    "for episode in range(1, episodes+1):\n",
    "    terminated, truncated = False, False\n",
    "    G, reward = 0, 0\n",
    "    state, info = env.reset()\n",
    "    while not terminated:\n",
    "        # Select the action following the epsilon-greedy policy\n",
    "        action_dist = policy_fn(state, Q, epsilon, num_actions)\n",
    "        action = np.random.choice(np.arange(num_actions), p=action_dist)\n",
    "        # Take the action and observe reward and next state\n",
    "        next_state, reward, terminated, truncated, info = env.step(action)\n",
    "\n",
    "        # Update the expected return for the action-value function\n",
    "        next_estimated_returns = predict_returns(next_state, Q)\n",
    "        td_target = reward + gamma * np.max(next_estimated_returns)\n",
    "        Q[action].partial_fit([convert_to_features(state, scaler)], [td_target])\n",
    " \n",
    "        G += reward\n",
    "        state = next_state\n",
    "\n",
    "    returns.append(G)\n",
    "\n",
    "    if episode % 1 == 0:\n",
    "        print(f'Episode: {episode} Return: {G}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot The Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAHHCAYAAABwaWYjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABi+UlEQVR4nO3deVxUVeMG8GdYZoZ1ANkFFcV9QzERxZ1EX1soLbVM3NJcyi1NfRVt+1GapWal1Ztab+aWmqmpSGaviZQLbgmJ+8LmwgyyM3N+f+DcnEBkkJlx8Pl+Pvcjc++5957L1ebp3HPPkQkhBIiIiIioymwsXQEiIiIia8MARURERGQkBigiIiIiIzFAERERERmJAYqIiIjISAxQREREREZigCIiIiIyEgMUERERkZEYoIiIiIiMxABFRFSLyWQyzJ8/36znHD58OBo0aGDWcxKZGwMUkZVbtWoVZDKZtNjZ2aFu3boYPnw4rl69Wq1j/vnnn5g/fz4uXLhQs5U1sxs3bmD69Olo2rQplEolPDw8EBUVhW3btlm6ahVq0KCBwb28e+nbt6+lq0dEd7GzdAWIqGa89dZbCAoKQmFhIQ4ePIhVq1Zh//79OHnyJJRKpVHH+vPPP/Hmm2+iR48eVtuSkJqait69eyM7OxsjRoxAhw4dkJOTg2+//RZPPvkkXn/9dSxcuNDS1SwnJCQE06ZNK7fe39+/WscrKCiAnR3/U09U0/iviqiW6NevHzp06AAAGD16NDw9PfH+++9j69ateP755y1cuzJ5eXlwcnIy+XlKSkowcOBA3Lp1C7/++ivCwsKkbVOmTMGLL76IDz74AB06dMCgQYNMXh+90tJS6HQ6yOXye5apW7cuhg4dWmPnNDY8E1HV8BEeUS3VtWtXAMDZs2cN1qekpGDgwIHw8PCAUqlEhw4dsHXrVmn7qlWr8NxzzwEAevbsKT1C+uWXXwDcu09NgwYNMHz4cIPjyGQy7Nu3D+PHj4e3tzcCAgIAAD169ECrVq3w559/omfPnnB0dETdunWxYMGCcsf9+OOP0bJlSzg6OsLd3R0dOnTAmjVrKr3277//HidPnsTMmTMNwhMA2NraYsWKFXBzc5OuIzMzE3Z2dnjzzTfLHSs1NRUymQzLli2T1uXk5GDy5MkIDAyEQqFAcHAw3n//feh0OqnMhQsXIJPJ8MEHH2Dx4sVo1KgRFAoF/vzzz0rrXhXDhw+Hs7Mzzp07h6ioKDg5OcHf3x9vvfUWhBAGZf95v3JzczF58mQ0aNAACoUC3t7eePzxx3HkyBGD/TZs2IDQ0FA4ODjA09MTQ4cOrfCR8JYtW9CqVSsolUq0atUKmzdvrrDOOp0OixcvRsuWLaFUKuHj44OxY8fi1q1bBuUOHTqEqKgoeHp6wsHBAUFBQRg5cmQ1f1NEpsMWKKJaSt9/yd3dXVp36tQpdOnSBXXr1sXMmTPh5OSE9evXIzo6Gt9//z2eeeYZdOvWDa+99hqWLl2K2bNno3nz5gAg/Wms8ePHw8vLC7GxscjLy5PW37p1C3379sWzzz6L559/Hhs3bsQbb7yB1q1bo1+/fgCAL774Aq+99hoGDhyISZMmobCwEMePH0dSUhJeeOGFe57zxx9/BAAMGzaswu0qlQpPP/00Vq9ejbS0NAQHB6N79+5Yv3495s2bZ1B23bp1sLW1lUJlfn4+unfvjqtXr2Ls2LGoV68eDhw4gFmzZiE9PR2LFy822H/lypUoLCzEmDFjoFAo4OHhUenvq6SkBNevXy+33snJCQ4ODtJnrVaLvn37olOnTliwYAF27tyJefPmobS0FG+99dY9j//KK69g48aNmDhxIlq0aIEbN25g//79OH36NNq3bw+gLPyOGDECjz32GOLi4pCZmYklS5bgt99+w9GjR+Hm5gYA2L17NwYMGIAWLVogLi4ON27cwIgRI6SgfLexY8dKx33ttddw/vx5LFu2DEePHsVvv/0Ge3t7ZGVloU+fPvDy8sLMmTPh5uaGCxcuYNOmTZX+zogsQhCRVVu5cqUAIPbs2SOys7PF5cuXxcaNG4WXl5dQKBTi8uXLUtnevXuL1q1bi8LCQmmdTqcTnTt3Fo0bN5bWbdiwQQAQe/fuLXc+AGLevHnl1tevX1/ExMSUq1dERIQoLS01KNu9e3cBQHz99dfSuqKiIuHr6ysGDBggrXv66adFy5Ytjfl1CCGECAkJESqVqtIyH374oQAgtm7dKoQQYsWKFQKAOHHihEG5Fi1aiF69ekmf3377beHk5CT++usvg3IzZ84Utra24tKlS0IIIc6fPy8ACFdXV5GVlVWletevX18AqHCJi4uTysXExAgA4tVXX5XW6XQ60b9/fyGXy0V2dra0/p/3S6VSiQkTJtyzDsXFxcLb21u0atVKFBQUSOu3bdsmAIjY2FhpXUhIiPDz8xM5OTnSut27dwsAon79+tK6//3vfwKA+Pbbbw3OtXPnToP1mzdvFgDEH3/8UYXfFpFl8REeUS0RGRkJLy8vBAYGYuDAgXBycsLWrVul1oCbN2/i559/xvPPP4/c3Fxcv34d169fx40bNxAVFYUzZ85U+629yrz88suwtbUtt97Z2dmgr49cLkfHjh1x7tw5aZ2bmxuuXLmCP/74w6hz5ubmwsXFpdIy+u0ajQYA8Oyzz8LOzg7r1q2Typw8eRJ//vmnQT+pDRs2oGvXrnB3d5d+h9evX0dkZCS0Wi1+/fVXg/MMGDAAXl5eVa57WFgY4uPjyy1DhgwpV3bixInSzzKZDBMnTkRxcTH27Nlzz+O7ubkhKSkJ165dq3D7oUOHkJWVhfHjxxv0n+rfvz+aNWuG7du3AwDS09ORnJyMmJgYqFQqqdzjjz+OFi1aGBxzw4YNUKlUePzxxw1+Z6GhoXB2dsbevXulugHAtm3bUFJScp/fFJFl8REeUS3xySefoEmTJlCr1fjqq6/w66+/QqFQSNvT0tIghMDcuXMxd+7cCo+RlZWFunXr1mi9goKCKlwfEBAAmUxmsM7d3R3Hjx+XPr/xxhvYs2cPOnbsiODgYPTp0wcvvPACunTpUuk5XVxcKnwMdrfc3FypLAB4enqid+/eWL9+Pd5++20AZY/v7Ozs8Oyzz0r7nTlzBsePH79nKMrKyjL4fK/rvxdPT09ERkbet5yNjQ0aNmxosK5JkyYAUOnwEwsWLEBMTAwCAwMRGhqKf/3rXxg2bJh0rIsXLwIAmjZtWm7fZs2aYf/+/QblGjduXK5c06ZNDfpUnTlzBmq1Gt7e3hXWSf876969OwYMGIA333wTH330EXr06IHo6Gi88MILBn+XiR4GDFBEtUTHjh2lt/Cio6MRERGBF154AampqXB2dpY6OL/++uuIioqq8BjBwcHVPr9Wq61w/d39du5WUasUAINO0M2bN0dqaiq2bduGnTt34vvvv8enn36K2NjYCjt8371fcnIyLl26hHr16lVYRh/U7m4tGTx4MEaMGIHk5GSEhIRg/fr16N27Nzw9PaUyOp0Ojz/+OGbMmFHhcfUhRu9e128pzz//PLp27YrNmzdj9+7dWLhwId5//31s2rRJ6ntW03Q6Hby9vfHtt99WuF0fRmUyGTZu3IiDBw/ixx9/xK5duzBy5EgsWrQIBw8ehLOzs0nqR1QdDFBEtZCtrS3i4uLQs2dPLFu2DDNnzpRaGOzt7e/bwvHPlqG7ubu7Iycnx2BdcXEx0tPTH7jeFXFycsKgQYMwaNAgFBcX49lnn8W7776LWbNm3fMV/SeeeALfffcdvv76a8yZM6fcdo1Ggx9++AHNmjUzCI3R0dEYO3as9Bjvr7/+wqxZswz2bdSoEW7fvl2lViJT0ul0OHfunEFg++uvvwDgvmN3+fn5Yfz48Rg/fjyysrLQvn17vPvuu+jXrx/q168PoOztw169ehnsl5qaKm3X/3nmzJlyx09NTTX43KhRI+zZswddunSpUqDs1KkTOnXqhHfffRdr1qzBiy++iLVr12L06NH33ZfIXNgHiqiW6tGjBzp27IjFixejsLAQ3t7e6NGjB1asWFFh2MnOzpZ+1o/V9M+gBJR9Gf6zn8/nn39+zxaoB3Hjxg2Dz3K5HC1atIAQotI+MgMHDkSLFi3w3nvv4dChQwbbdDodxo0bh1u3bpV7487NzQ1RUVFYv3491q5dC7lcjujoaIMyzz//PBITE7Fr165y583JyUFpaamRV1l9dw+tIITAsmXLYG9vj969e1dYXqvVQq1WG6zz9vaGv78/ioqKAAAdOnSAt7c3li9fLq0DgJ9++gmnT59G//79AZSFsJCQEKxevdrgmPHx8eWGanj++eeh1WqlR6N3Ky0tlf6e3bp1q9wwDCEhIQBgUBeihwFboIhqsenTp+O5557DqlWr8Morr+CTTz5BREQEWrdujZdffhkNGzZEZmYmEhMTceXKFRw7dgxA2ZeWra0t3n//fajVaigUCvTq1Qve3t4YPXo0XnnlFQwYMACPP/44jh07hl27dhk85qopffr0ga+vL7p06QIfHx+cPn0ay5YtQ//+/SvtJC6Xy7Fx40b07t0bERERBiORr1mzBkeOHMG0adMwePDgcvsOGjQIQ4cOxaeffoqoqCipY7Pe9OnTsXXrVjzxxBMYPnw4QkNDkZeXhxMnTmDjxo24cOHCA/0url69iv/+97/l1js7OxuEOaVSiZ07dyImJgZhYWH46aefsH37dsyePfue/bNyc3MREBCAgQMHom3btnB2dsaePXvwxx9/YNGiRQDKWijff/99jBgxAt27d8eQIUOkYQwaNGiAKVOmSMeLi4tD//79ERERgZEjR+LmzZvSuF23b9+WynXv3h1jx45FXFwckpOT0adPH9jb2+PMmTPYsGEDlixZgoEDB2L16tX49NNP8cwzz6BRo0bIzc3FF198AVdXV/zrX/+q9u+UyCQs+g4gET0w/XABFb36rdVqRaNGjUSjRo2koQTOnj0rhg0bJnx9fYW9vb2oW7eueOKJJ8TGjRsN9v3iiy9Ew4YNha2trcGQBlqtVrzxxhvC09NTODo6iqioKJGWlnbPYQwqqlf37t0rHJ4gJibG4PX3FStWiG7duok6deoIhUIhGjVqJKZPny7UanWVfjdZWVli6tSpIjg4WCgUCuHm5iYiIyOloQsqotFohIODgwAg/vvf/1ZYJjc3V8yaNUsEBwcLuVwuPD09RefOncUHH3wgiouLhRB/D2OwcOHCKtVViMqHMbj79xITEyOcnJzE2bNnRZ8+fYSjo6Pw8fER8+bNE1qt1uCYuGsYg6KiIjF9+nTRtm1b4eLiIpycnETbtm3Fp59+Wq4u69atE+3atRMKhUJ4eHiIF198UVy5cqVcue+//140b95cKBQK0aJFC7Fp06Zy91Hv888/F6GhocLBwUG4uLiI1q1bixkzZohr164JIYQ4cuSIGDJkiKhXr55QKBTC29tbPPHEE+LQoUNV/h0SmYtMiH+0lxIR0UNt+PDh2Lhxo0ErDxGZF/tAERERERmJAYqIiIjISAxQREREREZiHygiIiIiI7EFioiIiMhIDFBERERERuJAmiag0+lw7do1uLi4VDolBhERET08hBDIzc2Fv78/bGwqb2NigDKBa9euITAw0NLVICIiomq4fPkyAgICKi3DAGUC+ikmLl++DFdXVwvXhoiIiKpCo9EgMDCw0qmi9Big7uGTTz7BwoULkZGRgbZt2+Ljjz9Gx44dq7Sv/rGdq6srAxQREZGVqUr3G3Yir8C6deswdepUzJs3D0eOHEHbtm0RFRWFrKwsS1eNiIiIHgIMUBX48MMP8fLLL2PEiBFo0aIFli9fDkdHR3z11VeWrhoRERE9BBig/qG4uBiHDx9GZGSktM7GxgaRkZFITEyscJ+ioiJoNBqDhYiIiGovBqh/uH79OrRaLXx8fAzW+/j4ICMjo8J94uLioFKppIVv4BEREdVuDFA1YNasWVCr1dJy+fJlS1eJiIiITIhv4f2Dp6cnbG1tkZmZabA+MzMTvr6+Fe6jUCigUCjMUT0iIiJ6CLAF6h/kcjlCQ0ORkJAgrdPpdEhISEB4eLgFa0ZEREQPC7ZAVWDq1KmIiYlBhw4d0LFjRyxevBh5eXkYMWKEpatGREREDwEGqAoMGjQI2dnZiI2NRUZGBkJCQrBz585yHcuJiIjo0SQTQghLV6K20Wg0UKlUUKvVHImciIjIShjz/c0+UERERERGYoAiIiIiMhIDFBERkRXR6gSyNIXQFJagVKuzdHUeWexETkREZCUu3sjDiJV/4Nz1PGmdws4GjnJbOMrt4KSwhYPcDk53fe4Y5IGBoQFQ2Nk+8PlLtTr8kHwNB8/dgI1MBltbGextZLCztYGdjQx2tjLY2eh/toG9rQwAIASgEwJaIcp+1pX9rBOAEAJaXVl37O5NvBDWsM4D19Mc2IncBNiJnIio9hFC4MilW/jpRAacFHbo09IHLfxcIZPJzHL+lAwNXvrP78jOLTJ6X19XJV7p3hCDO9aD0t74IFVcqsPmo1fwyd6zuHQz3+j9jREd4o/Z/ZvD20Vp0vNUxJjvbwYoE2CAIiKqPS5cz8Pmo1exJfkqLt4wDA+BHg7o29IXUS190b6eO2xsTBOmjl66heEr/4C6oATNfF3w9aiOcFXao6BYi7ziUuQXa8uWorKf9euyc4uwJukSMjSFAAAvFwXGdmuIF8Pqw0F+/yBVVKrFhkNX8NkvZ3E1pwAAUMdJjkGPBcJJYYdSrUCpTodSnUCpVocSbVlrUqnu7591QsBWJoNMJoONDLC1MfzZRiaDTAbcuF2MH49fgxCAi8IOr0c1xdBO9WFrot9pRRigLIwBiojIuuXkF+PH4+nYfOQKjlzKkdY7ym3Rt6UvbheVYt9f2Sgq/bsPkpeLAn1a+CCqpS86NawDuV3NdDM+kHYdo78+hPxiLdrXc8PK4R2hcrSv8v4VhSBPZzlGd22IlzrVh5OifG+ewhIt1v1xGZ/9crZc+HohrB4c5abpAXT8Sg7mbDmJ41fUAIBWdV3xTnRrhAS6meR8/8QAZWEMUEQPH61O4HZRadlSWIrbRSXILfz7c1GpDva2NpDb3VlsbaCwM/ys/9nbRQEXZdW/wKhyeUWl+N+ZbAAydAzygIeT3CL1KCrVYm9KNjYfvYKfU7JQoi37erSRARGNvfBsu7ro09JHCg/5xaXYl5qNXacykHA6C7lFpdKxXJV26N28LEz1bOZV7f5Hu09lYOJ3R1FcqkNEsCdWvBRaYeCpCv1juGV703D5ZlmQcne0x+iuDTEsvD5c7rRofZt0EZ//eg5Zdx4V+rgqMK57o2o//jOWView5vdLWLAzBbmFpZDJgCEd62FGVFO4OZr27wYDlIUxQBFV3dWcAnwU/xfS1QWICPZCZHNvBHs7V7tfSVGpFknnbiLhdCYOnruJW/nFuH3nsUZNsbWRoX09N/Ro6o3uTbzQws/VZI9uqkpdUIK0rNtIy8rFmczbOJN1G2lZtyGTAUGeTmhQxwkNPJ0Q5OmIIE9nBLg7wN7Wci9i5xeX4ueULGw/no6fU7IMWnKa+DgjLKgOwhp6ICyoDrxcqj9ZuxAC+cVa3MovRk5+CXLyS+78XHzn5xLk5BfjVn4xjlzKgbqgRNq3uZ8rnm1XF0+H+MPbtfL+OMWlOiSeu4GdJzMQ/2cGrt8ulrb5qZQY060hBj9Wr0qPzfQ2H72C1zcch1YnENXSB0uHtKuRjuAldzqCL/v5DC7ceSTpqrRD/zb+BnX3VykxrmcwngsNMEtw+qfs3CLE7TiNTUevAih7dDjrX80xoH1dk/U7Y4CyMAYoovsrLNHii1/P4ZNf0lBYYvgqdj0PR/Ru7o3I5j54rIHHfR+F3LhdhL2p2Ug4nYlf/8pGXiVhyd5WBhelPZwVdmWL0g4uCjso7W1RrNWhuPTOUsHPRaU6FJVqkVtYanBMT2c5ujX2QvemXuja2MtkLSj5xaW4cbsY6epCnLkTlNKybuNMVi4yNcZ1LLa1kSHQ3eFOqCpbfF2V8HRRwMtZAU9nhVFf9lVRUKzFL6lZ2HY8HQkpmQb3vUEdR9jb2uBM1u1y+zX0ckJYUB10uhOofFVlYUarE8jOLUK6ugAZ6kKkqwuRobnzp7oA6epCZGmKUGzEq/4+rgpEh9TFM+3roplv9f77rdUJHL54C7tOZeDHY9eklhxPZzlGRTTE0E717tuC+U3iBcz94RQAYED7ALw/oDXsajjwlmp12HY8HR//fAZns/9+qy/A3QETewbj2fYBNfYY8kEcPHcDc7eclP5udGzggbejW6Gpr0uNn4sBysIYoIgql3A6E2/++Kf0Nk/HBh6IauWLX//KRuLZGwZfeC4KO3RrWtYy1aOJN9yd5BBC4EzWbew5nYmE01k4cukW7v4vmZeLAr2beaNHU28EuDvARfl3WKqJ/4O/fDMfv57Jxi+p2TiQdt0gsMlkQJsAN3Rv4oXuTbzg46qAVidQou9sqxUo1Qlo73Sy1XfCLS7V4VZ+MW7kFePm7WLczLvzc57+56JyQfOffF2VaOzjjGBvZzT2dkGwtzNsZMC563m4cD0PF27k4Vx2Hi7eyEdByf1b5JwVdvB0lsPLpSxQSYuLHC5Ke7jc+Z3qw6iL0g5OCjuDlq3CEi1+Sc3G9hPpSDidadASWM/DEf3b+OGJNn7S22w3bhfh9/M3kXT+Jg6eu4HUzFz881sqwN2hbCyk3CLp9ff7kdvawM3RHu6Ocqgc7eFu8LMcbg72qFfHEWFBdWq003JhiRYbD1/B8n1nceVW2WMzlYM9hndugBFdGpR7JCWEwKe/nMXCXakAgOGdGyD2iRYmbeHU6gR2nEjHL6nZCGvogWfa1bVo62RFikt1+Oq381iy5wwKSrSwtZHh1V7BmBzZpEbPwwBlYQxQ9LAQQpjtFeuqOH89D2/9eAp7U7MBlP3f/ux/NcdTbf2lepb1h7mOhNOZ2JuaZfAoxEYGtKvnjqzcQqkPh14LP1dENvdG7+Y+aF1XZbZHasWlOhy+eAu//JWFfanZSMnINen59H2wykLSnaB0JzS5VrFflhACmZoinLt+Gxeu5+PCjTycv56HrNwiXM8tQvbtIhSXVn+ARqW9DZwV9nBR2iFLU2gQMAPcHcpCU2t/tKp7/yEAcvKL8fv5m1KoOnVNjbszk52NDD6uSviqyha/Oz/7qRzgq1LCx1UBDyc5HOxtLfpvQf/Y7NNf0nDuTmuPk9wWQ8PrY3REQ3i5KCCEwHs7U7Bi3zkAwGu9G2NKZOOH6t+wpV3NKcBbP57CrlOZmPdkC4zoElSjx2eAsjAGKLK0dHUBpqxLxtWcAvwn5jE08an5pm5j5BeXYtnPafjyf+dRrNXB3laGURENMbFXMJwr6RCr0wkkX8lBwp2WprvDidzOBp0b1UHv5j7o3cwb/m4O5riU+8pQF+LXv7Kx769s/Hb2OgqKtdKgggYDDdrKYGcjg72tDWzv/OnuaA8PJwXqOMvh4VS21JH+VMDDWQ4nuemDgBACuUWlZWEqtwjXbxfj+u0iXL9d9vlGXvGdjvilyC0sufNnqUE/prv5q5To38YP/dv4o22A6oHqrykswcmrajjJ7eCnUqKOs8Ksr7k/KK1O4KeT6Vj2c5r091lhZ4MhHeuhoFiLdYcuAwDm9G+O0V0bWrKqD7Xf0q4jLMijxh9rMkBZGAMUWdLBczcwcc0RqeXG20WBja90Rr06jmavixAC246n4/92nEa6uuxV6G5NvDDvyRZo5OVs9PGu3MrHb2nX4eYoR9fGniZ7lZqqp7hUh7yiUilQ3S4qhaPc9qHoZP+wEULg55QsfPxzGpIv50jrbWRA3LOtMeixepar3COMAcrCGKDIEoQQWPnbBby74zS0OoHmfq7Q6nT4K/M26nk4YsMr4fC5z5tENSW3sARJ527iy/3ncPDcTQBlAw7O7d8Cj7fw4SMJojuEEPgt7QaW7T2DU9c0eO/ZNujfxs/S1XpkMUBZGAMUGaNUq8Oe05lo6uuKIE+nah2joFiLWZuOY0vyNQBlUyHEPdsGuYUlGLg8EZdu5qOJjzPWjQmHuwneECsq1eLopRwcSLuO/WnXceyKWurcq7CzwfgewRjbvaFFXoUmshYPW5/FRxEDlIUxQNVe56/n4Uxmbo21opRqdZi8LhnbjqdDJgP6tPDBmG4NEVrfo8rHuHQjH2P/exin0zWwtZHh3/9qjhFdGkj1u3wzHwOXH0CmpghtAlT4dnTYAw8CqdMJnM7Q4Le069ifdgN/nL9Z7q2uBnUc0bWxF8Z0a4hAD/M/PiQiMhYDlIUxQNVOOp1Az0W/4OKNfAx+LBDvPtP6gTqvlmp1mLr+GLYeuwZbG5nB69jt67lhTLdGeLyFT6Xn2PdXNl777ijUBSXwdJZj2Qvt0amCmczPZObi+RWJuJVfgrAgD6we2bFarUGnrqmxfN85/JZ2HTfzig22eTrL0bmRJyKCPdE5uA4C3BmaiMi6MEBZGANU7XTg7HW88EWS9LlvS18sHhxSrSCi1QlMW5+MLcnXYGcjw6cvtkdDLyd88et5bD56VRoHKcjTCaMigjDwHyMB68eK+WB3KoQAQgLd8NnQ9vBT3ftNtBNX1BjyxUHcLipF72beWP5SaJXHesnSFOKD3anYcPiKNCaPo9wWnRrWQZdgT3QJroOmPi58/EBEVo0BysIYoB4Op9M1dwZRrJk5y6auT8amI1fRNkCF0+m5KNbqEN6wDj4fFmrUObQ6gekbjmHT0auws5Fh2Qvt0beVr7Q9S1OI1YkX8E3iRWjujHjt4STHsPD6GBbeAHI7G0xbn4xdpzIBAEM6BmL+Uy2rNEBk0rkbGPbV7ygq1eGptv74aFBIpS1cBcVafPG/c1i+76w0AOJTbf3xUnh9hAS6PXSD7RERPQgGKAtjgLI8fWtR18ae+GZU2AMf73ZRKR57Zw8KSrT4flxnFJVo8fLXh5BXrEXruiqsHPEYPJ3vP1+XTicw4/vj2Hj4CmxtZFg2pB36ta74jZu8olKsP3QZX/7vvDSDutLeBnWcFLiaUwC5rQ3efLolhnQ07nXnvSlZePnrQyjVCbwQVg/vRrcq13Kk0wlsPXYN7+9MkYYfaF/PDXOeaIH29dyNOh8RkbUw5vub//tItdLmI2WTT/7vzHWcTtc88PF2HE9HQYkWDb2c0L6eGzoHe2LtmHB4OMlx4qoazy9PxJVb+ZUeQ6cTmLnp7/C0dPC9wxMAOCnsMKJLEPZN74GlQ9qhVV1XFJbocDWnAL6uSqwb28no8AQAPZt5Y/HgEMhkwJqkS3h/Z6rB9kMXbuKZzw5g8rpkpKsLUdfNAR8PaYfvx3VmeCIiuoOj0FGtox8WQO/rxIuIe7b1Ax1z4+ErAICBoQFSa03rABU2vBKOYf/5Heeu52HgZ4n4ZlRHNK5g1G+dTmD25hNYf+gKbGTA4kEhVR7rxc7WBk+19ceTbfyQeO4GDl24hSEd6z3QDPVPtPHH7cJSzNx0Asv3nYWL0g5PtfXHez+lYPuJdABl86CN79kII7sEcfgBIqJ/4CM8E+AjPMvSP76zt5WhRCvgYG+Lg7N6Q+VYvb5QF67noccHv8BGBhyY2VuaCV4vXV2AYf/5HWeybsPN0R5fDX/MoKVGpxOY88NJrEm6BBsZ8NGgEDwdUveBrrGmfPHrOby74zQASL8vGxkw6LFATH286QOFNCIia8NHePRI23UyAwAQHVIXzXxdUFCixYbDl6t9vE1HylqfIhp7lQtPAOCncsD6seEICXRDTn4JXvwiCfv+KpssVwiB2K1l4UkmAxY93/ahCU8A8HK3hnitVzAAoEQrEBHsie2vdUXcs20YnoiIKsFHeFSr6HRCejutX2tftKvnjtmbT+CbgxcxskuQ0fNx6XQC39/pT/VcaMA9y7k7yfHt6DC88t/D+N+Z6xi9+g8sej4Ehy/cxH8PloWnDwa2xTPt7n0MS5nyeBM08XWBm4McXYLrcCgCIqIqYAsU1SrHr6qRoSmEk9wWnRt5IrqdP1yVdrh4I19qFTJG4rkbuJpTABelHR5v4VNpWSeFHf4T8xieaOOHEq3Aa98dxerEi5DJgAUD2mBAJQHMkmQyGZ5o44+Ixp4MT0REVcQARbXKzjuP73o284bS3haOcjs83yEQALA68YLRx9N3Hn+qrX+VOlLL7WywZHA7vNSpvrTu/Wfb4Lk7dSAiotqBj/Co1hBCYNepsgB198CUQzvVx39+O49fUrNx4XoeGlRxwl5NYQl+Oln2RtpAI1qPbG1keOvplugY5AF3RzkiGnsacRVERGQN2AJFtcaZrNs4fz0Pcjsb9GjqLa1v4OmEHk28AADfHLxY5ePtOJ6OwhIdgr2dERLoZlRdZDIZnmzrz/BERFRLMUBRraF/fNc12BPOCsPG1WGdGwAA1h+6jPzi0iodr6Kxn4iIiAAGKKpF9AEq6q7Hd3rdG3uhfh1H5BaWYvPRq/c91vnreTh08RZsZMAz7R6eYQeIiOjhwABFJnH44i08+fF+7E3NMsv5Lt/Mx5/pGtjayBDZvPzbcjY2Mqlj99cHLuJ+48duvDNuVPcmXvBxLT/2ExERPdoYoMgkVh+4gBNX1Xjtu6O4fLPyOeJqgr7zeMcGHvBwkldY5rkOgXCwt0VqZi6Szt+857G0OoFNd8Z+GhjKt+eIiKg8BiiqcUIIJJ2/AQDILSzFpLVHUarVmfSc+sd3fSt4fKencrDHM+3LHsd9XcmQBgfOXke6uhAqB3v0bu59z3JERPToYoCiGnfxRj4yNUWwt5XBRWGHI5dysCThjMnOl5VbiMOXbgEA+rSsfLDLYeFlj/F2ncpEurqgwjLGjv1ERESPHgYoqnH61qe2AW74v2dbAwCW7U1D4tkbJjlf/J+ZEAJoG+gGP5VDpWWb+boiLMgDWp3AmqRL5barC0qk1qznOjycI4cTEZHlMUBRjUs6V9a/KKyhB55s64/nOwRACGDyuqO4mVdc4+eTHt+1vPfju7vF3BnS4LvfL6GoVGuwbfvxdBSV6tDExxmt66pqtJ5ERFR71KoA1aBBA8hkMoPlvffeMyhz/PhxdO3aFUqlEoGBgViwYEG542zYsAHNmjWDUqlE69atsWPHDnNdQq2g76AdFlQHADD/qZZo6OWETE0RZmw8dt834Iyhzi+RWrai7vP4Tu/xFj7wdVXi+u1i7DiRbrBN//Ydx34iIqLK1KoABQBvvfUW0tPTpeXVV1+Vtmk0GvTp0wf169fH4cOHsXDhQsyfPx+ff/65VObAgQMYMmQIRo0ahaNHjyI6OhrR0dE4efKkJS7H6ly+mY+rOQWwtZEhtL47AMBRboePh7SD3NYGe05n4evEqo8Gfj8/p2aiVCfQ1McFDb2cq7SPva0NXgyrBwBYfeDvuqRl3caRSzmwtZEhmmM/ERFRJWpdgHJxcYGvr6+0ODn9Pe/Zt99+i+LiYnz11Vdo2bIlBg8ejNdeew0ffvihVGbJkiXo27cvpk+fjubNm+Ptt99G+/btsWzZMktcjtXRtz61rquC012jgbf0V2HWv5oBAN7dcRp/XtPUyPmkwTOr2PqkN7hjPdjbypB8OQfHLucAAL4/UtZ5vEcTL3i7cOwnIiK6t1oXoN577z3UqVMH7dq1w8KFC1Fa+ve0HYmJiejWrRvk8r/HCYqKikJqaipu3bollYmMjDQ4ZlRUFBITE+95zqKiImg0GoPlUZV0ruxxWlhDj3LbhndugF7NvFFcqsOr3x2p8pQq91JQrMW+v7IBVDz6eGW8XBTo39oPAPB14sU7Yz/9PXULERFRZWpVgHrttdewdu1a7N27F2PHjsX//d//YcaMGdL2jIwM+PgYtlToP2dkZFRaRr+9InFxcVCpVNISGPjoDr6ob4HqdKf/091kMhkWDmwDbxcFzmbn4a0f/3ygc+37KxuFJToEejighZ+r0fvrO5P/ePwath67ikxNEdwc7dGLYz8REdF9PPQBaubMmeU6hv9zSUlJAQBMnToVPXr0QJs2bfDKK69g0aJF+Pjjj1FUVGTSOs6aNQtqtVpaLl++bNLzPazS1QW4dDMfNjKgQwP3CsvUcVZg8aAQyGTA2j8uY9vxa9U+n3708agWvtXq8B0S6IY2ASoUl+rw781lfdyiQ+pCYcexn4iIqHJ29y9iWdOmTcPw4cMrLdOwYcMK14eFhaG0tBQXLlxA06ZN4evri8zMTIMy+s++vr7SnxWV0W+viEKhgEKhuN+l1Hr64Qta+qvgorS/Z7nOwZ4Y36MRPtl7FrM2nUDbADcEejgada7iUh32nC67T5WNPl4ZmUyGYeEN8PqGY8gvLhvOgI/viIioKh76FigvLy80a9as0uXuPk13S05Oho2NDby9yx7JhIeH49dff0VJSYlUJj4+Hk2bNoW7u7tUJiEhweA48fHxCA8PN9EV1h76ATTDgsr3f/qnyZFN0K6eW7Wnekk8dwO5haXwclGgfb2KW7uq4ok2fnB3LAt7zXxd0NLf+EeBRET06HnoA1RVJSYmYvHixTh27BjOnTuHb7/9FlOmTMHQoUOlcPTCCy9ALpdj1KhROHXqFNatW4clS5Zg6tSp0nEmTZqEnTt3YtGiRUhJScH8+fNx6NAhTJw40VKXZjX+HkCzfP+nf7K3tcHSwe2qPdWL/vFdnxY+sLGp/nhNSntbvNytrAVzZEQQx34iIqIqeegf4VWVQqHA2rVrMX/+fBQVFSEoKAhTpkwxCEcqlQq7d+/GhAkTEBoaCk9PT8TGxmLMmDFSmc6dO2PNmjWYM2cOZs+ejcaNG2PLli1o1aqVJS7LamRpCnHueh5kMqBjg/u3QAFAoIcj/u/Z1nj1u6NYtjcN+cVaTI5sXOnjPwDQ6gR2nyp7fBdVxdHHKzOueyM82y4AvioOXUBERFUjEzU5LDQBKBuwU6VSQa1Ww9X10Xgk9OOxa3j1u6No7ueKnyZ1NWrfN388hZW/XQBQNrzAnP7N8VRb/3u2Bv1x4SaeW54IV6UdDs15HHK7WtOQSkREFmTM9ze/eahGGNP/6Z/mPdkSK0c8hgZ1HJGdW4RJa5Mx+POD+Cszt8Lyu+4MnhnZ3IfhiYiILILfPlQj9P2fOlWh/1NFejb1xs7J3fB6nyZQ2tsg6fxN9FvyP7yz7U/kFv7d6V8IgZ36/k818PiOiIioOhig6IHduF2EM1m3AQAdq9ECpae0t8XEXo2xZ2p3RLX0gVYn8OX+8+i9aB9+SL4KIQROXdPgyq0CKO1t0L2JV01dAhERkVFqTSdyspzf74w+3tTHBR5OFQ8pYYwAd0eseKkD9qZmYf7WU7h4Ix+T1iZj7e+X4Xeno3ePJt5wkHPASyIisgwGKHpg+ulbKpr/7kH0bOqN8Ml18MWv57BsbxoS78yzB1R/8EwiIqKawEd49MAO6icQrmD+uweltLfFq73LHuv1aVE2R6Gzwg49m3G+OiIishy2QNEDyckvRuqdt+UepP/T/QR6OOLzYR2QfDkHDva2UDlUPlYUERGRKTFA0QP5/fxNCAE08nKCl4vp5wMMCXQz+TmIiIjuh4/w6IH83f+p5h/fERERPawYoOiBPMgAmkRERNaKAYqqTVNYgj+vaQBUfwBNIiIia8QARdV26MJN6ATQoI4jfFw5ES8RET06GKCo2vTTt5hi+AIiIqKHGQMUVdtBEw2gSURE9LBjgKJquV1UipNX1QD4Bh4RET16GKAIW45exTeJF6DTiSrvc/jiLWh1AgHuDqjr5mDC2hERET18OJDmI27z0SuYsu4YAODc9TzEPtECMpnsvvslmXD6FiIioocdW6AeYSevqjHz+xPS55W/XcBH8X9VaV9TTSBMRERkDRigHlE384ox9pvDKCrVoWdTL8x/sgUAYOnPaVix72yl+xYUa3H8Sg4AoBNboIiI6BHER3iPoFKtDq9+dwRXcwrQoI4jFg9uB5WDPfJLtFiwMxVxP6XASWGHoZ3qV7j/kUu3UKIV8FMpEejB/k9ERPToYQvUI2jhrlT8lnYDjnJbrHipA1QO9gCA8T2CMb5HIwDA3B9OYvPRKxXu/3f/J48q9ZciIiKqbRigHjE/HruGFb+eAwAsHNgWTX1dDLZPj2qKmPD6EAJ4fcNx7DqVUe4YBzmBMBERPeIYoB4hp9M1mLHxOADgle6N0L+NX7kyMpkM855siQHtA6DVCby65ij+dyZb2l5YokXy5RwAnECYiIgeXQxQj4ic/LJO4wUlWnRt7InpUU3vWdbGRob3B7RGv1a+KNbq8PLXh/DHhbJWp+TLOSgu1cHLRYEgTydzVZ+IiOihwgD1CNDqBF5bm4xLN/MR6OGAj4e0g61N5X2X7GxtsGRwO3Rv4oXCEh1GrvwDJ6+q75r/jv2fiIjo0cUA9QhYtDsVv/6VDaW9DVYM7QA3R3mV9pPb2WD50FB0DPJAblEpXvpPErafuAaA/Z+IiOjRxgBVy/10Ih2f/lI2rtP7A9qghb+rUfs7yG3xn5gOaBugwq38EvyVeRsA0In9n4iI6BHGAFWL/ZWZi2kbyqZpGR0RhKdD6lbrOC5Ke6wa0RFNfcre2KvjJEewt3ON1ZOIiMjacCDNWkpdUIKx3xxGfrEWnRvVwcx+zR7oeO5OcnwzuiPe3Ponujf1Yv8nIiJ6pDFA1VIf7ErF+et5qOtW1mnczvbBGxu9XZT45MX2NVA7IiIi68ZHeLVUSoYGAPB6VBPUcVZYuDZERES1CwNULVVQogWAKr9xR0RERFXHAFVL5ReXBShHe1sL14SIiKj2YYCqpQrvBCgHOQMUERFRTWOAqqXy7zzCc2SAIiIiqnEMULVUwZ0WKCUf4REREdU4BqhaSKsTKCrVAQAc5RypgoiIqKYxQNVChXce3wGAA1ugiIiIapzVBKh3330XnTt3hqOjI9zc3Cosc+nSJfTv3x+Ojo7w9vbG9OnTUVpaalDml19+Qfv27aFQKBAcHIxVq1aVO84nn3yCBg0aQKlUIiwsDL///rsJrsh09G/gAYDS3mpuMRERkdWwmm/X4uJiPPfccxg3blyF27VaLfr374/i4mIcOHAAq1evxqpVqxAbGyuVOX/+PPr374+ePXsiOTkZkydPxujRo7Fr1y6pzLp16zB16lTMmzcPR44cQdu2bREVFYWsrCyTX2NN0bdAOdjbcsoVIiIiE5AJIYSlK2GMVatWYfLkycjJyTFY/9NPP+GJJ57AtWvX4OPjAwBYvnw53njjDWRnZ0Mul+ONN97A9u3bcfLkSWm/wYMHIycnBzt37gQAhIWF4bHHHsOyZcsAADqdDoGBgXj11Vcxc+bMKtVRo9FApVJBrVbD1dW1Bq7aOKkZuYha/CvqOMlxeO7jZj8/ERGRNTLm+9tqWqDuJzExEa1bt5bCEwBERUVBo9Hg1KlTUpnIyEiD/aKiopCYmAigrJXr8OHDBmVsbGwQGRkplalIUVERNBqNwWJJ+lHI+QYeERGRadSaAJWRkWEQngBInzMyMioto9FoUFBQgOvXr0Or1VZYRn+MisTFxUGlUklLYGBgTVxSteUXl/X74hhQREREpmHRADVz5kzIZLJKl5SUFEtWsUpmzZoFtVotLZcvX7ZofaQ+UAxQREREJmHRQYKmTZuG4cOHV1qmYcOGVTqWr69vubflMjMzpW36P/Xr7i7j6uoKBwcH2NrawtbWtsIy+mNURKFQQKFQVKme5qB/C49DGBAREZmGRQOUl5cXvLy8auRY4eHhePfdd5GVlQVvb28AQHx8PFxdXdGiRQupzI4dOwz2i4+PR3h4OABALpcjNDQUCQkJiI6OBlDWiTwhIQETJ06skXqaQwHnwSMiIjIpq+kDdenSJSQnJ+PSpUvQarVITk5GcnIybt++DQDo06cPWrRogZdeegnHjh3Drl27MGfOHEyYMEFqHXrllVdw7tw5zJgxAykpKfj000+xfv16TJkyRTrP1KlT8cUXX2D16tU4ffo0xo0bh7y8PIwYMcIi110dBZwHj4iIyKSsZp6P2NhYrF69Wvrcrl07AMDevXvRo0cP2NraYtu2bRg3bhzCw8Ph5OSEmJgYvPXWW9I+QUFB2L59O6ZMmYIlS5YgICAAX375JaKioqQygwYNQnZ2NmJjY5GRkYGQkBDs3LmzXMfyh9nfj/Cs5vYSERFZFasbB8oaWHocqI/i/8KShDMY2qke3olubfbzExERWaNHchwo+tvfj/DYAkVERGQKDFC1kL4TOQfSJCIiMg0GqFpI3weKnciJiIhMgwGqFrp7MmEiIiKqeQxQtZB+KheOA0VERGQaDFC1UAFboIiIiEyKAaoWKmAfKCIiIpNigKqF2AJFRERkWgxQtVA+58IjIiIyKQaoWkh6C48BioiIyCQYoGohaRwozoVHRERkEgxQtYwQQuoDpZTz9hIREZkCv2FrmaJSHfTTQ3MuPCIiItNggKpl9I/vAL6FR0REZCoMULWM/vGd3M4GtjYyC9eGiIiodmKAqmUK7kzjwkE0iYiITIcBqpYpKNYB4OM7IiIiU2KAqmU4kTAREZHpMUDVMpzGhYiIyPQYoGoZTiRMRERkegxQtYw0iCZboIiIiEyGAaqWyWcLFBERkckxQNUyhewDRUREZHIMULWMvgXKgdO4EBERmQwDVC3Dt/CIiIhMjwGqluFbeERERKbHAFXLFEiP8BigiIiITIUBqpbJ5yM8IiIik2OAqmUKOJULERGRyTFA1TL6TuTsA0VERGQ6DFC1jH4YA45ETkREZDoMULUM38IjIiIyPQaoWoaP8IiIiEyPAaqWKeAjPCIiIpNjgKpl/n6Ex6lciIiITIUBqpbhVC5ERESmxwBVixSX6lCqEwA4DhQREZEpWU2Aevfdd9G5c2c4OjrCzc2twjIymazcsnbtWoMyv/zyC9q3bw+FQoHg4GCsWrWq3HE++eQTNGjQAEqlEmFhYfj9999NcEU1T9/6BLAFioiIyJSsJkAVFxfjueeew7hx4yott3LlSqSnp0tLdHS0tO38+fPo378/evbsieTkZEyePBmjR4/Grl27pDLr1q3D1KlTMW/ePBw5cgRt27ZFVFQUsrKyTHVpNUbf/8nORga5ndXcWiIiIqtjNT2N33zzTQCosMXobm5ubvD19a1w2/LlyxEUFIRFixYBAJo3b479+/fjo48+QlRUFADgww8/xMsvv4wRI0ZI+2zfvh1fffUVZs6cWUNXYxrs/0RERGQeta6ZYsKECfD09ETHjh3x1VdfQQghbUtMTERkZKRB+aioKCQmJgIoa+U6fPiwQRkbGxtERkZKZR5m+ZwHj4iIyCyspgWqKt566y306tULjo6O2L17N8aPH4/bt2/jtddeAwBkZGTAx8fHYB8fHx9oNBoUFBTg1q1b0Gq1FZZJSUm553mLiopQVFQkfdZoNDV4VVVXqG+BYoAiIiIyKYu2QM2cObPCjt93L5UFl3+aO3cuunTpgnbt2uGNN97AjBkzsHDhQhNeQZm4uDioVCppCQwMNPk5K6KfB4+P8IiIiEzLoi1Q06ZNw/Dhwyst07Bhw2ofPywsDG+//TaKioqgUCjg6+uLzMxMgzKZmZlwdXWFg4MDbG1tYWtrW2GZe/WrAoBZs2Zh6tSp0meNRmOREKXvRM4WKCIiItOyaIDy8vKCl5eXyY6fnJwMd3d3KBQKAEB4eDh27NhhUCY+Ph7h4eEAALlcjtDQUCQkJEhv7+l0OiQkJGDixIn3PI9CoZDOYUmcB4+IiMg8rKYP1KVLl3Dz5k1cunQJWq0WycnJAIDg4GA4Ozvjxx9/RGZmJjp16gSlUon4+Hj83//9H15//XXpGK+88gqWLVuGGTNmYOTIkfj555+xfv16bN++XSozdepUxMTEoEOHDujYsSMWL16MvLw86a28hxkf4REREZmH1QSo2NhYrF69Wvrcrl07AMDevXvRo0cP2Nvb45NPPsGUKVMghEBwcLA0JIFeUFAQtm/fjilTpmDJkiUICAjAl19+KQ1hAACDBg1CdnY2YmNjkZGRgZCQEOzcubNcx/KH0d+P8KzmthIREVklmbj7PX+qERqNBiqVCmq1Gq6urmY77yd707BwVyoGdQjE+wPbmO28REREtYEx39+1bhyoRxk7kRMREZkHA1Qtks8ARUREZBYMULUIp3IhIiIyj2oFqIKCAuTn50ufL168iMWLF2P37t01VjEyXsGdqVw4jAEREZFpVStAPf300/j6668BADk5OQgLC8OiRYvw9NNP47PPPqvRClLV6VuglGyBIiIiMqlqBagjR46ga9euAICNGzfCx8cHFy9exNdff42lS5fWaAWp6vR9oNgCRUREZFrVClD5+flwcXEBAOzevRvPPvssbGxs0KlTJ1y8eLFGK0hVV8g+UERERGZRrQAVHByMLVu24PLly9i1axf69OkDAMjKyjLruEdkiG/hERERmUe1AlRsbCxef/11NGjQAGFhYdJccrt375ZGCCfz41t4RERE5lGtOT8GDhyIiIgIpKeno23bttL63r1745lnnqmxypFxCqQ+UJzKhYiIyJSq/U3r6+sLX19fg3UdO3Z84ApR9UktUHIO70VERGRK1QpQeXl5eO+995CQkICsrCzodDqD7efOnauRypFx8jmZMBERkVlU65t29OjR2LdvH1566SX4+flBJpPVdL3ISFqdQHFpWZBlHygiIiLTqlaA+umnn7B9+3Z06dKlputD1aR/fAdwHCgiIiJTq1ZnGXd3d3h4eNR0XegB5N+ZxkUmAxR27ANFRERkStX6pn377bcRGxtrMB8eWVZh8d+P7/hIlYiIyLSq9Qhv0aJFOHv2LHx8fNCgQQPY29sbbD9y5EiNVI6qLr+EEwkTERGZS7UCVHR0dA1Xgx6UfgwoTiRMRERkekYHqNLSUshkMowcORIBAQGmqBNVQwEnEiYiIjIbo/tA2dnZYeHChSgtLTVFfaiaOI0LERGR+VSrE3mvXr2wb9++mq4LPQBOJExERGQ+1eoD1a9fP8ycORMnTpxAaGgonJycDLY/9dRTNVI5qjq2QBEREZlPtQLU+PHjAQAffvhhuW0ymQxarbbcejItTiRMRERkPtX6tv3n3HdkefoWKL6FR0REZHocsrqWyOdbeERERGZTrRaot956q9LtsbGx1aoMVV9hCTuRExERmUu1AtTmzZsNPpeUlOD8+fOws7NDo0aNGKAsQD8XHjuRExERmV61AtTRo0fLrdNoNBg+fDieeeaZB64UGY/DGBAREZlPjfWBcnV1xZtvvom5c+fW1CHJCPpHeOwDRUREZHo12olcrVZDrVbX5CGpivI5Fx4REZHZVOsR3tKlSw0+CyGQnp6Ob775Bv369auRipFxOBceERGR+VQrQH300UcGn21sbODl5YWYmBjMmjWrRipGxuFI5EREROZTrQB1/vz5mq4HPaACdiInIiIym2r1gRo5ciRyc3PLrc/Ly8PIkSMfuFJkvHxO5UJERGQ21QpQq1evRkFBQbn1BQUF+Prrrx+4UmS8Qj7CIyIiMhujmis0Gg2EEBBCIDc3F0qlUtqm1WqxY8cOeHt713gl6f44lQsREZH5GBWg3NzcIJPJIJPJ0KRJk3LbZTIZ3nzzzRqrHFWNEIKTCRMREZmRUQFq7969EEKgV69e+P777+Hh4SFtk8vlqF+/Pvz9/Wu8klS5whKd9DNboIiIiEzPqD5Q3bt3R48ePXD+/HlER0eje/fu0hIeHm6y8HThwgWMGjUKQUFBcHBwQKNGjTBv3jwUFxcblDt+/Di6du0KpVKJwMBALFiwoNyxNmzYgGbNmkGpVKJ169bYsWOHwXYhBGJjY+Hn5wcHBwdERkbizJkzJrmumqJvfQLYAkVERGQO1epEXr9+fezfvx9Dhw5F586dcfXqVQDAN998g/3799doBQEgJSUFOp0OK1aswKlTp/DRRx9h+fLlmD17tlRGo9GgT58+qF+/Pg4fPoyFCxdi/vz5+Pzzz6UyBw4cwJAhQzBq1CgcPXoU0dHRiI6OxsmTJ6UyCxYswNKlS7F8+XIkJSXByckJUVFRKCwsrPHrqin6iYQVdjawtZFZuDZERESPAFENGzduFA4ODmL06NFCoVCIs2fPCiGE+Pjjj0W/fv2qc0ijLViwQAQFBUmfP/30U+Hu7i6KioqkdW+88YZo2rSp9Pn5558X/fv3NzhOWFiYGDt2rBBCCJ1OJ3x9fcXChQul7Tk5OUKhUIjvvvuuynVTq9UCgFCr1UZfV3WcydSI+m9sE23f3GWW8xEREdVGxnx/V6sF6p133sHy5cvxxRdfwN7eXlrfpUsXHDlypIaiXeXUarVBH6zExER069YNcrlcWhcVFYXU1FTcunVLKhMZGWlwnKioKCQmJgIoGyA0IyPDoIxKpUJYWJhU5mEkvYHHx3dERERmUa0AlZqaim7dupVbr1KpkJOT86B1uq+0tDR8/PHHGDt2rLQuIyMDPj4+BuX0nzMyMiotc/f2u/erqExFioqKoNFoDBZz0o9CrmQHciIiIrOoVoDy9fVFWlpaufX79+9Hw4YNq3ycmTNnSsMi3GtJSUkx2Ofq1avo27cvnnvuObz88svVqX6Ni4uLg0qlkpbAwECznj+/hGNAERERmVO15v14+eWXMWnSJHz11VeQyWS4du0aEhMTMW3aNMTGxlb5ONOmTcPw4cMrLXN3ILt27Rp69uyJzp07G3QOB8pCXWZmpsE6/WdfX99Ky9y9Xb/Oz8/PoExISMg96zhr1ixMnTpV+qzRaMwaoqR58PgIj4iIyCyqFaBmzpwJnU6H3r17Iz8/H926dYNCocD06dMxevToKh/Hy8sLXl5eVSp79epV9OzZE6GhoVi5ciVsbAwbz8LDw/Hvf/8bJSUlUr+s+Ph4NG3aFO7u7lKZhIQETJ48WdovPj4e4eHhAICgoCD4+voiISFBCkwajQZJSUkYN27cPeumUCigUCiqetk17u+JhDkPHhERkTlU6xGeTCbDv//9b9y8eRMnT57EwYMHkZ2dDZVKhaCgoJquI65evYoePXqgXr16+OCDD5CdnY2MjAyDfkkvvPAC5HI5Ro0ahVOnTmHdunVYsmSJQcvQpEmTsHPnTixatAgpKSmYP38+Dh06hIkTJ0rXNXnyZLzzzjvYunUrTpw4gWHDhsHf3x/R0dE1fl01JV+aB69at5OIiIiMZFSTRVFREebPn4/4+HipxSk6OhorV67EM888A1tbW0yZMqXGKxkfH4+0tDSkpaUhICDAYJsQAkBZB/bdu3djwoQJCA0NhaenJ2JjYzFmzBipbOfOnbFmzRrMmTMHs2fPRuPGjbFlyxa0atVKKjNjxgzk5eVhzJgxyMnJQUREBHbu3Gkw79/DplCaB48tUEREROYgE/oEUgVvvPEGVqxYgcjISBw4cADZ2dkYMWIEDh48iNmzZ+O5556DrS374Wg0GqhUKqjVari6upr8fEv2nMFHe/7CkI71EPdsa5Ofj4iIqDYy5vvbqCaLDRs24Ouvv8ZTTz2FkydPok2bNigtLcWxY8cgk3EEbEsp4Ft4REREZmVUp5krV64gNDQUANCqVSsoFApMmTKF4cnCCu5M5cIARUREZB5GBSitVmsw0rednR2cnZ1rvFJkHH0LFCcSJiIiMg+jHuEJITB8+HDplf3CwkK88sorcHJyMii3adOmmqsh3Zc0lQtboIiIiMzCqAAVExNj8Hno0KE1WhmqnsISDqRJRERkTkYFqJUrV5qqHvQA8qWBNBmgiIiIzIEjL9YCBWyBIiIiMisGqFqggANpEhERmRUDVC0gtUDJeTuJiIjMgd+4tYDUB8qeLVBERETmwABVCxSwEzkREZFZMUBZOSEEp3IhIiIyMwYoK1es1UGrK5sPmiORExERmQcDlJUrLNZJP7MFioiIyDwYoKxcfknZRMJ2NjLY2/J2EhERmQO/ca0cO5ATERGZHwOUleNEwkRERObHAGXlOJEwERGR+TFAWbm/JxLmIJpERETmwgBl5f6eSJi3koiIyFz4rWvlOJEwERGR+TFAWTl9CxQH0SQiIjIfBigrx7fwiIiIzI8BysrxLTwiIiLzY4CycvnFZSORcyBNIiIi82GAsnL5HImciIjI7BigrJz+EZ4jH+ERERGZDQOUlWMLFBERkfkxQFk5TiZMRERkfgxQVq6Ab+ERERGZHQOUlSvgOFBERERmxwBl5fR9oDgSORERkfkwQFk56S08zoVHRERkNgxQVo5TuRAREZkfA5SV42TCRERE5scAZeXYiZyIiMj8GKCsWKlWh2KtDgCHMSAiIjInBigrpn98B3AgTSIiInOyigB14cIFjBo1CkFBQXBwcECjRo0wb948FBcXG5SRyWTlloMHDxoca8OGDWjWrBmUSiVat26NHTt2GGwXQiA2NhZ+fn5wcHBAZGQkzpw5Y5brNJY+QMlkgMLOKm4lERFRrWAV37opKSnQ6XRYsWIFTp06hY8++gjLly/H7Nmzy5Xds2cP0tPTpSU0NFTaduDAAQwZMgSjRo3C0aNHER0djejoaJw8eVIqs2DBAixduhTLly9HUlISnJycEBUVhcLCQrNcqzGk/k/2tpDJZBauDRER0aNDJoQQlq5EdSxcuBCfffYZzp07B6CsBSooKAhHjx5FSEhIhfsMGjQIeXl52LZtm7SuU6dOCAkJwfLlyyGEgL+/P6ZNm4bXX38dAKBWq+Hj44NVq1Zh8ODBVaqbRqOBSqWCWq2Gq6vrg11oJVIyNOi7+H/wdJbj0JzHTXYeIiKiR4Ex399W0QJVEbVaDQ8Pj3Lrn3rqKXh7eyMiIgJbt2412JaYmIjIyEiDdVFRUUhMTAQAnD9/HhkZGQZlVCoVwsLCpDIVKSoqgkajMVjMIZ8TCRMREVmEVQaotLQ0fPzxxxg7dqy0ztnZGYsWLcKGDRuwfft2REREIDo62iBEZWRkwMfHx+BYPj4+yMjIkLbr192rTEXi4uKgUqmkJTAw8IGvsSr0j/D4Bh4REZF5WTRAzZw5s8KO33cvKSkpBvtcvXoVffv2xXPPPYeXX35ZWu/p6YmpU6ciLCwMjz32GN577z0MHToUCxcuNPl1zJo1C2q1WlouX75s8nMCdwUoTuNCRERkVhb95p02bRqGDx9eaZmGDRtKP1+7dg09e/ZE586d8fnnn9/3+GFhYYiPj5c++/r6IjMz06BMZmYmfH19pe36dX5+fgZl7tWvCgAUCgUUCsV961PT8kv0LVBW2ZBIRERktSwaoLy8vODl5VWlslevXkXPnj0RGhqKlStXwsbm/qEhOTnZIAiFh4cjISEBkydPltbFx8cjPDwcABAUFARfX18kJCRIgUmj0SApKQnjxo2r+oWZSWExJxImIiKyBKv45r169Sp69OiB+vXr44MPPkB2dra0Td9qtHr1asjlcrRr1w4AsGnTJnz11Vf48ssvpbKTJk1C9+7dsWjRIvTv3x9r167FoUOHpNYsmUyGyZMn45133kHjxo0RFBSEuXPnwt/fH9HR0ea74CrKLy4FwD5QRERE5mYVASo+Ph5paWlIS0tDQECAwba7R2F4++23cfHiRdjZ2aFZs2ZYt24dBg4cKG3v3Lkz1qxZgzlz5mD27Nlo3LgxtmzZglatWkllZsyYgby8PIwZMwY5OTmIiIjAzp07oVQqTX+hRioouTONC9/CIyIiMiurHQfqYWaucaA+3J2KpT+n4aVO9fF2dKv770BERET39EiMA0V/T+XiyBYoIiIis2KAsmIcSJOIiMgyGKCsWEEJB9IkIiKyBAYoKyZNJswWKCIiIrNigLJi+hYoJVugiIiIzIoByorlcyBNIiIii2CAsmKF+j5Qct5GIiIic+I3rxWT3sKzZwsUERGROTFAWbECDmNARERkEQxQVowDaRIREVkGA5QV42TCRERElsEAZaV0OoFCTiZMRERkEQxQVqqwVCv9zBYoIiIi82KAslL6DuQAAxQREZG5MUBZKf0QBgo7G9jYyCxcGyIiokcLA5SVKuQbeERERBbDAGWl/h5EkwGKiIjI3BigrFRBCQfRJCIishQGKCtVwImEiYiILIYBykpJLVB8hEdERGR2DFBWKp/z4BEREVkMA5SVYgsUERGR5TBAWamCO/PgcRgDIiIi82OAslIFxWXz4CkZoIiIiMyOAcpK5ZfcaYHiIzwiIiKzY4CyUgXsRE5ERGQxDFBWigGKiIjIchigrFQ+38IjIiKyGAYoK1VYzMmEiYiILIUBykrpB9JUsgWKiIjI7BigrJR+IE3OhUdERGR+DFBWSupEzhYoIiIis2OAslLSVC7sA0VERGR2DFBWKp+dyImIiCyGAcpKFXIYAyIiIothgLJCQgjkczJhIiIii2GAskLFWh10ouxnTiZMRERkfgxQVkj/Bh7AR3hERESWYDUB6qmnnkK9evWgVCrh5+eHl156CdeuXTMoc/z4cXTt2hVKpRKBgYFYsGBBueNs2LABzZo1g1KpROvWrbFjxw6D7UIIxMbGws/PDw4ODoiMjMSZM2dMem3G0r+BZ28rg72t1dxCIiKiWsNqvn179uyJ9evXIzU1Fd9//z3Onj2LgQMHSts1Gg369OmD+vXr4/Dhw1i4cCHmz5+Pzz//XCpz4MABDBkyBKNGjcLRo0cRHR2N6OhonDx5UiqzYMECLF26FMuXL0dSUhKcnJwQFRWFwsJCs15vZfI5BhQREZFFyYQQwtKVqI6tW7ciOjoaRUVFsLe3x2effYZ///vfyMjIgFwuBwDMnDkTW7ZsQUpKCgBg0KBByMvLw7Zt26TjdOrUCSEhIVi+fDmEEPD398e0adPw+uuvAwDUajV8fHywatUqDB48uEp102g0UKlUUKvVcHV1reErB05eVeOJj/fDx1WBpNmRNX58IiKiR5Ex399W0wJ1t5s3b+Lbb79F586dYW9vDwBITExEt27dpPAEAFFRUUhNTcWtW7ekMpGRhoEjKioKiYmJAIDz588jIyPDoIxKpUJYWJhUpiJFRUXQaDQGiylxGhciIiLLsqoA9cYbb8DJyQl16tTBpUuX8MMPP0jbMjIy4OPjY1Be/zkjI6PSMndvv3u/ispUJC4uDiqVSloCAwOreYVVw4mEiYiILMuiAWrmzJmQyWSVLvrHbwAwffp0HD16FLt374atrS2GDRuGh+EJ5KxZs6BWq6Xl8uXLJj1fAUchJyIisiiLPgOaNm0ahg8fXmmZhg0bSj97enrC09MTTZo0QfPmzREYGIiDBw8iPDwcvr6+yMzMNNhX/9nX11f6s6Iyd2/Xr/Pz8zMoExIScs86KhQKKBSKyi+2BhWUlA2iyU7kRERElmHRAOXl5QUvL69q7avT6QCU9T8CgPDwcPz73/9GSUmJ1C8qPj4eTZs2hbu7u1QmISEBkydPlo4THx+P8PBwAEBQUBB8fX2RkJAgBSaNRoOkpCSMGzeuWvU0hYLismvnRMJERESWYRV9oJKSkrBs2TIkJyfj4sWL+PnnnzFkyBA0atRICj8vvPAC5HI5Ro0ahVOnTmHdunVYsmQJpk6dKh1n0qRJ2LlzJxYtWoSUlBTMnz8fhw4dwsSJEwEAMpkMkydPxjvvvIOtW7fixIkTGDZsGPz9/REdHW2JS6+QfhoXtkARERFZhlW8xuXo6IhNmzZh3rx5yMvLg5+fH/r27Ys5c+ZIj85UKhV2796NCRMmIDQ0FJ6enoiNjcWYMWOk43Tu3Blr1qzBnDlzMHv2bDRu3BhbtmxBq1atpDIzZsxAXl4exowZg5ycHERERGDnzp1QKpVmv+57KSxhHygiIiJLstpxoB5mph4HasHOFHz6y1kM79wA859qWePHJyIiehTV+nGgHnUFbIEiIiKyKAYoK8RhDIiIiCyLAcoK6VugOJAmERGRZTBAWaH8Yk7lQkREZEkMUFZI/xaeg5y3j4iIyBL4DWyF9C1QDvZsgSIiIrIEBigrJAUodiInIiKyCAYoK8SBNImIiCyLAcoKcSoXIiIiy2KAskIFfIRHRERkUQxQVkg/DhRboIiIiCyDAcrKlGh1KNGWTV/IPlBERESWwQBlZfStTwBHIiciIrIUBigrU3in/5ONDFDY8fYRERFZAr+Brczfg2jaQiaTWbg2REREjyYGKCsjdSDnPHhEREQWwwBlZf4ehZy3joiIyFL4LWxlpFHIOQ8eERGRxTBAWRnOg0dERGR5DFBWhoNoEhERWR4DlJUpuDMPHgfRJCIishwGKCujnwdPyQBFRERkMQxQViZf6kTOAEVERGQpDFBWpoCdyImIiCyOAcrKMEARERFZHgOUlcnnW3hEREQWxwBlZfSTCfMtPCIiIsthgLIyd08mTERERJbBAGVlOJkwERGR5TFAWZkCtkARERFZHAOUldG3QLEPFBERkeUwQFmZ/DtTuSjZAkVERGQxDFBWprBEB4AtUERERJbEAGVl8jmZMBERkcUxQFkZfR8oPsIjIiKyHAYoK6LTCT7CIyIieggwQFmRwlKt9DPnwiMiIrIcBigroh+FHACUdgxQRERElmI1Aeqpp55CvXr1oFQq4efnh5deegnXrl2Ttl+4cAEymazccvDgQYPjbNiwAc2aNYNSqUTr1q2xY8cOg+1CCMTGxsLPzw8ODg6IjIzEmTNnzHKN96MfRFNpbwMbG5mFa0NERPTospoA1bNnT6xfvx6pqan4/vvvcfbsWQwcOLBcuT179iA9PV1aQkNDpW0HDhzAkCFDMGrUKBw9ehTR0dGIjo7GyZMnpTILFizA0qVLsXz5ciQlJcHJyQlRUVEoLCw0y3VW5u9BNDmNCxERkSXJhBDC0pWojq1btyI6OhpFRUWwt7fHhQsXEBQUhKNHjyIkJKTCfQYNGoS8vDxs27ZNWtepUyeEhIRg+fLlEELA398f06ZNw+uvvw4AUKvV8PHxwapVqzB48OAq1U2j0UClUkGtVsPV1fWBr1Uv+XIOoj/5DXXdHPDbzF41dlwiIiIy7vvbalqg7nbz5k18++236Ny5M+zt7Q22PfXUU/D29kZERAS2bt1qsC0xMRGRkZEG66KiopCYmAgAOH/+PDIyMgzKqFQqhIWFSWUqUlRUBI1GY7CYgjQPHjuQExERWZRVBag33ngDTk5OqFOnDi5duoQffvhB2ubs7IxFixZhw4YN2L59OyIiIhAdHW0QojIyMuDj42NwTB8fH2RkZEjb9evuVaYicXFxUKlU0hIYGPjA11qRgpKyQTQ5kTAREZFlWTRAzZw5s8KO33cvKSkpUvnp06fj6NGj2L17N2xtbTFs2DDon0B6enpi6tSpCAsLw2OPPYb33nsPQ4cOxcKFC01+HbNmzYJarZaWy5cvm+Q8BcVlY0CxBYqIiMiyLNobedq0aRg+fHilZRo2bCj97OnpCU9PTzRp0gTNmzdHYGAgDh48iPDw8Ar3DQsLQ3x8vPTZ19cXmZmZBmUyMzPh6+srbdev8/PzMyhzr35VAKBQKKBQKCq9jpqgn8aFLVBERESWZdEA5eXlBS8vr2rtq9OVtcYUFRXds0xycrJBEAoPD0dCQgImT54srYuPj5cCWFBQEHx9fZGQkCAFJo1Gg6SkJIwbN65a9axJhdJbeAxQRERElmQV78MnJSXhjz/+QEREBNzd3XH27FnMnTsXjRo1ksLP6tWrIZfL0a5dOwDApk2b8NVXX+HLL7+UjjNp0iR0794dixYtQv/+/bF27VocOnQIn3/+OQBAJpNh8uTJeOedd9C4cWMEBQVh7ty58Pf3R3R0tNmv+5/0A2myBYqIiMiyrCJAOTo6YtOmTZg3bx7y8vLg5+eHvn37Ys6cOQaPzt5++21cvHgRdnZ2aNasGdatW2cwVlTnzp2xZs0azJkzB7Nnz0bjxo2xZcsWtGrVSiozY8YM5OXlYcyYMcjJyUFERAR27twJpVJp1muuiEDZIJqOCgYoIiIiS7LacaAeZqYaB0pPCAGZjCORExER1aRaPw7Uo47hiYiIyLIYoIiIiIiMxABFREREZCQGKCIiIiIjMUARERERGYkBioiIiMhIDFBERERERmKAIiIiIjISAxQRERGRkRigiIiIiIzEAEVERERkJAYoIiIiIiMxQBEREREZiQGKiIiIyEh2lq5AbSSEAABoNBoL14SIiIiqSv+9rf8erwwDlAnk5uYCAAIDAy1cEyIiIjJWbm4uVCpVpWVkoioxi4yi0+lw7do1uLi4QCaT1eixNRoNAgMDcfnyZbi6utbosalm8V5ZD94r68L7ZT2s7V4JIZCbmwt/f3/Y2FTey4ktUCZgY2ODgIAAk57D1dXVKv4yEu+VNeG9si68X9bDmu7V/Vqe9NiJnIiIiMhIDFBERERERmKAsjIKhQLz5s2DQqGwdFXoPnivrAfvlXXh/bIetflesRM5ERERkZHYAkVERERkJAYoIiIiIiMxQBEREREZiQGKiIiIyEgMUFbkk08+QYMGDaBUKhEWFobff//d0lUiAL/++iuefPJJ+Pv7QyaTYcuWLQbbhRCIjY2Fn58fHBwcEBkZiTNnzlimso+4uLg4PPbYY3BxcYG3tzeio6ORmppqUKawsBATJkxAnTp14OzsjAEDBiAzM9NCNX50ffbZZ2jTpo00AGN4eDh++uknaTvv08Prvffeg0wmw+TJk6V1tfF+MUBZiXXr1mHq1KmYN28ejhw5grZt2yIqKgpZWVmWrtojLy8vD23btsUnn3xS4fYFCxZg6dKlWL58OZKSkuDk5ISoqCgUFhaauaa0b98+TJgwAQcPHkR8fDxKSkrQp08f5OXlSWWmTJmCH3/8ERs2bMC+fftw7do1PPvssxas9aMpICAA7733Hg4fPoxDhw6hV69eePrpp3Hq1CkAvE8Pqz/++AMrVqxAmzZtDNbXyvslyCp07NhRTJgwQfqs1WqFv7+/iIuLs2Ct6J8AiM2bN0ufdTqd8PX1FQsXLpTW5eTkCIVCIb777jsL1JDulpWVJQCIffv2CSHK7o29vb3YsGGDVOb06dMCgEhMTLRUNekOd3d38eWXX/I+PaRyc3NF48aNRXx8vOjevbuYNGmSEKL2/rtiC5QVKC4uxuHDhxEZGSmts7GxQWRkJBITEy1YM7qf8+fPIyMjw+DeqVQqhIWF8d49BNRqNQDAw8MDAHD48GGUlJQY3K9mzZqhXr16vF8WpNVqsXbtWuTl5SE8PJz36SE1YcIE9O/f3+C+ALX33xUnE7YC169fh1arhY+Pj8F6Hx8fpKSkWKhWVBUZGRkAUOG9028jy9DpdJg8eTK6dOmCVq1aASi7X3K5HG5ubgZleb8s48SJEwgPD0dhYSGcnZ2xefNmtGjRAsnJybxPD5m1a9fiyJEj+OOPP8ptq63/rhigiOiRNGHCBJw8eRL79++3dFXoHpo2bYrk5GSo1Wps3LgRMTEx2Ldvn6WrRf9w+fJlTJo0CfHx8VAqlZaujtnwEZ4V8PT0hK2tbbk3FjIzM+Hr62uhWlFV6O8P793DZeLEidi2bRv27t2LgIAAab2vry+Ki4uRk5NjUJ73yzLkcjmCg4MRGhqKuLg4tG3bFkuWLOF9esgcPnwYWVlZaN++Pezs7GBnZ4d9+/Zh6dKlsLOzg4+PT628XwxQVkAulyM0NBQJCQnSOp1Oh4SEBISHh1uwZnQ/QUFB8PX1Nbh3Go0GSUlJvHcWIITAxIkTsXnzZvz8888ICgoy2B4aGgp7e3uD+5WamopLly7xfj0EdDodioqKeJ8eMr1798aJEyeQnJwsLR06dMCLL74o/Vwb7xcf4VmJqVOnIiYmBh06dEDHjh2xePFi5OXlYcSIEZau2iPv9u3bSEtLkz6fP38eycnJ8PDwQL169TB58mS88847aNy4MYKCgjB37lz4+/sjOjracpV+RE2YMAFr1qzBDz/8ABcXF6n/hUqlgoODA1QqFUaNGoWpU6fCw8MDrq6uePXVVxEeHo5OnTpZuPaPllmzZqFfv36oV68ecnNzsWbNGvzyyy/YtWsX79NDxsXFRepHqOfk5IQ6depI62vl/bL0a4BUdR9//LGoV6+ekMvlomPHjuLgwYOWrhIJIfbu3SsAlFtiYmKEEGVDGcydO1f4+PgIhUIhevfuLVJTUy1b6UdURfcJgFi5cqVUpqCgQIwfP164u7sLR0dH8cwzz4j09HTLVfoRNXLkSFG/fn0hl8uFl5eX6N27t9i9e7e0nffp4Xb3MAZC1M77JRNCCAtlNyIiIiKrxD5QREREREZigCIiIiIyEgMUERERkZEYoIiIiIiMxABFREREZCQGKCIiIiIjMUARERERGYkBiogeaRcuXIBMJkNycrLJzjF8+HCOPE9UyzBAEZFVGz58OGQyWbmlb9++Vdo/MDAQ6enp5aaiICKqDOfCIyKr17dvX6xcudJgnUKhqNK+tra2Vj0jPBFZBlugiMjqKRQK+Pr6Gizu7u4AAJlMhs8++wz9+vWDg4MDGjZsiI0bN0r7/vMR3q1bt/Diiy/Cy8sLDg4OaNy4sUE4O3HiBHr16gUHBwfUqVMHY8aMwe3bt6XtWq0WU6dOhZubG+rUqYMZM2bgnzNm6XQ6xMXFISgoCA4ODmjbtq1Bne5XByKyPAYoIqr15s6diwEDBuDYsWN48cUXMXjwYJw+ffqeZf/880/89NNPOH36ND777DN4enoCAPLy8hAVFQV3d3f88ccf2LBhA/bs2YOJEydK+y9atAirVq3CV199hf379+PmzZvYvHmzwTni4uLw9ddfY/ny5Th16hSmTJmCoUOHYt++ffetAxE9JCw8mTER0QOJiYkRtra2wsnJyWB59913hRBCABCvvPKKwT5hYWFi3LhxQgghzp8/LwCIo0ePCiGEePLJJ8WIESMqPNfnn38u3N3dxe3bt6V127dvFzY2NiIjI0MIIYSfn59YsGCBtL2kpEQEBASIp59+WgghRGFhoXB0dBQHDhwwOPaoUaPEkCFD7lsHIno4sA8UEVm9nj174rPPPjNY5+HhIf0cHh5usC08PPyeb92NGzcOAwYMwJEjR9CnTx9ER0ejc+fOAIDTp0+jbdu2cHJyksp36dIFOp0OqampUCqVSE9PR1hYmLTdzs4OHTp0kB7jpaWlIT8/H48//rjBeYuLi9GuXbv71oGIHg4MUERk9ZycnBAcHFwjx+rXrx8uXryIHTt2ID4+Hr1798aECRPwwQcf1Mjx9f2ltm/fjrp16xps03d8N3UdiOjBsQ8UEdV6Bw8eLPe5efPm9yzv5eWFmJgY/Pe//8XixYvx+eefAwCaN2+OY8eOIS8vTyr722+/wcbGBk2bNoVKpYKfnx+SkpKk7aWlpTh8+LD0uUWLFlAoFLh06RKCg4MNlsDAwPvWgYgeDmyBIiKrV1RUhIyMDIN1dnZ2UsfrDRs2oEOHDoiIiMC3336L33//Hf/5z38qPFZsbCxCQ0PRsmVLFBUVYdu2bVLYevHFFzFv3jzExMRg/vz5yM7OxquvvoqXXnoJPj4+AIBJkybhvffeQ+PGjdGsWTN8+OGHyMnJkY7v4uKC119/HVOmTIFOp0NERATUajV+++03uLq6IiYmptI6ENHDgQGKiKzezp074efnZ7CuadOmSElJAQC8+eabWLt2LcaPHw8/Pz989913aNGiRYXHksvlmDVrFi5cuAAHBwd07doVa9euBQA4Ojpi165dmDRpEh577DE4OjpiwIAB+PDDD6X9p02bhvT0dMTExMDGxgYjR47EM888A7VaLZV5++234eXlhbi4OJw7dw5ubm5o3749Zs+efd86ENHDQSbEPwYoISKqRWQyGTZv3sypVIioRrEPFBEREZGRGKCIiIiIjMQ+UERUq7GXAhGZAlugiIiIiIzEAEVERERkJAYoIiIiIiMxQBEREREZiQGKiIiIyEgMUERERERGYoAiIiIiMhIDFBEREZGRGKCIiIiIjPT/HdK0qh0eTLYAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(episodes), returns)\n",
    "plt.title(\"Returns Over Episodes\")\n",
    "plt.xlabel(\"Episodes\")\n",
    "plt.ylabel(\"Returns\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Episode Videos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/q_learning_fa//MountainCar-v0-episode-0.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-0.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/q_learning_fa//MountainCar-v0-episode-10.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-10.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/q_learning_fa//MountainCar-v0-episode-20.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-20.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/q_learning_fa//MountainCar-v0-episode-30.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-30.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/q_learning_fa//MountainCar-v0-episode-40.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-40.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Render Final Model On An Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Reward: -200.0, Steps Taken: 200\n"
     ]
    }
   ],
   "source": [
    "# Run the algorithm in inferencing mode and \n",
    "# observe its performance\n",
    "env = gym.make(ENV_TYPE, render_mode=\"human\")\n",
    "state, info = env.reset(seed=100)\n",
    "G = 0\n",
    "num_steps = 0\n",
    "terminated = False\n",
    "truncated = False\n",
    "\n",
    "while not terminated and not truncated:\n",
    "    estimated_returns = predict_returns(state, Q)\n",
    "    action = np.argmax(estimated_returns) \n",
    "    state, reward, terminated, truncated, info = env.step(action)\n",
    "    G += reward\n",
    "    num_steps += 1\n",
    "\n",
    "print(f'Total Reward: {G}, Steps Taken: {num_steps}')\n",
    "\n",
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "flax",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0a8d1f2580cdfde5c5829808ec6fccc81a351d243fb8b1925f7928e44ccf575b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}