sarsa_fa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SARSA with Kernel Function Approximation\n",
    "\n",
    "This notebook shows the function approximation reinforcement learning algorithm, SARSA applied to the MountainCar problem (it also works on Acrobat and CartPole) from Open AI's gym. This algorithm is designed to work on state spaces larger than the tabular SARSA algorithm. It achieves this by representing the states by feature vectors, $x(s) = (x_{1}(s), x_{2}(s), ..., x_{n}(s))^T$, which is a vector with smaller size than the set of all states. \n",
    "Now the action value function can be estimated with function approximation, given by:\n",
    "$$\\hat{q}(s, a, \\bold{w}) \\approx q_{\\pi}(s, a)$$\n",
    ", where $\\bold{w}$ is the parameter vector.\n",
    "\n",
    "The parameter vector can then be updated using gradient descent:\n",
    "$$\\bold{w}_{t+1} = \\bold{w}_{t} + \\alpha [R_{t+1} + \\gamma \\hat{q}(S_{t + 1}, A_{t + 1}, \\bold{w}_{t}) - \\hat{q}(S_{t}, A_{t}, \\bold{w}_{t})] \\nabla \\hat{q}(S_{t}, A_{t}, \\bold{w}_{t})$$\n",
    ", where $\\bold{w}$ is the parameter vector, $\\alpha$ is the step size, $\\gamma$ is the discount rate, $R$ is the reward, $\\nabla$ is the gradient operator, $S$ is the state, $A$ is the action and $\\hat{q}$ is the approximate action-value function.\n",
    "\n",
    "For more information on function approximation see section 10 of [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2018.pdf). For alternative implementations of SARSA function approximation see: [Q-Learning Value Function Approximation Example by Denny Britz](https://github.com/dennybritz/reinforcement-learning/blob/master/FA/Q-Learning%20with%20Value%20Function%20Approximation%20Solution.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import gymnasium as gym\n",
    "import numpy as np\n",
    "import sklearn\n",
    "import sklearn.pipeline\n",
    "from sklearn.linear_model import SGDRegressor\n",
    "from sklearn.kernel_approximation import RBFSampler\n",
    "from IPython.display import Video\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type of environment, options include:\n",
    "# MountainCar-v0, Acrobot-v1, CartPole-v1\n",
    "ENV_TYPE = \"MountainCar-v0\"\n",
    "\n",
    "# Save videos from training\n",
    "SAVE_VIDEOS = False\n",
    "VIDEOS_ROOT_DIR = './videos/sarsa_fa/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the environment\n",
    "env = gym.make(ENV_TYPE, render_mode=\"rgb_array\")\n",
    "\n",
    "# Number of possible actions\n",
    "num_actions = env.action_space.n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Features Using Radial Basis Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>FeatureUnion(transformer_list=[(&#x27;rbf1&#x27;, RBFSampler(gamma=5.0)),\n",
       "                               (&#x27;rbf2&#x27;, RBFSampler(gamma=2.0)),\n",
       "                               (&#x27;rbf3&#x27;, RBFSampler()),\n",
       "                               (&#x27;rbf4&#x27;, RBFSampler(gamma=0.5))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;&nbsp;FeatureUnion<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.FeatureUnion.html\">?<span>Documentation for FeatureUnion</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></label><div class=\"sk-toggleable__content fitted\"><pre>FeatureUnion(transformer_list=[(&#x27;rbf1&#x27;, RBFSampler(gamma=5.0)),\n",
       "                               (&#x27;rbf2&#x27;, RBFSampler(gamma=2.0)),\n",
       "                               (&#x27;rbf3&#x27;, RBFSampler()),\n",
       "                               (&#x27;rbf4&#x27;, RBFSampler(gamma=0.5))])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf1</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=5.0)</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf2</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=2.0)</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf3</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler()</pre></div> </div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label fitted sk-toggleable\"><label>rbf4</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow fitted\">&nbsp;RBFSampler<a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.kernel_approximation.RBFSampler.html\">?<span>Documentation for RBFSampler</span></a></label><div class=\"sk-toggleable__content fitted\"><pre>RBFSampler(gamma=0.5)</pre></div> </div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "FeatureUnion(transformer_list=[('rbf1', RBFSampler(gamma=5.0)),\n",
       "                               ('rbf2', RBFSampler(gamma=2.0)),\n",
       "                               ('rbf3', RBFSampler()),\n",
       "                               ('rbf4', RBFSampler(gamma=0.5))])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feature Preprocessing: Normalize to zero mean and unit variance\n",
    "# We use a few samples from the observation space to do this\n",
    "observation_examples = np.array([np.float64(env.observation_space.sample()) for x in range(10000)])\n",
    "scaler = sklearn.preprocessing.StandardScaler()\n",
    "scaler.fit(observation_examples)\n",
    "\n",
    "# Used to convert a state to a featurizes represenation.\n",
    "# We use RBF kernels with different variances to cover different parts of the space\n",
    "featurizer = sklearn.pipeline.FeatureUnion([\n",
    "        (\"rbf1\", RBFSampler(gamma=5.0, n_components=100)),\n",
    "        (\"rbf2\", RBFSampler(gamma=2.0, n_components=100)),\n",
    "        (\"rbf3\", RBFSampler(gamma=1.0, n_components=100)),\n",
    "        (\"rbf4\", RBFSampler(gamma=0.5, n_components=100))\n",
    "        ])\n",
    "featurizer.fit(scaler.transform(observation_examples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Action-Value Function Approximator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_to_features(state, scaler):\n",
    "    \"\"\"\n",
    "    Converts the state to a feature representation.\n",
    "    \"\"\"\n",
    "    scaled = scaler.transform([state])\n",
    "    featurized = featurizer.transform(scaled)\n",
    "    return featurized[0]    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of linear action-value fuction approximators (one for each action)\n",
    "Q = []\n",
    "\n",
    "for _ in range(num_actions):\n",
    "    linear_model = SGDRegressor(learning_rate=\"constant\")\n",
    "    # We need to call partial_fit once to initialize the model\n",
    "    # or we get a NotFittedError when trying to make a prediction\n",
    "    # This is quite hacky.\n",
    "    state, info = env.reset()\n",
    "    linear_model.partial_fit([convert_to_features(state, scaler)], [0])\n",
    "    Q.append(linear_model)    \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_returns(state, Q):\n",
    "    features = convert_to_features(state, scaler)\n",
    "    estimated_returns = []\n",
    "    for linear_model in Q:\n",
    "        estimated_returns.append(linear_model.predict([features])[0])\n",
    "\n",
    "    return np.array(estimated_returns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create An $\\epsilon$-Greedy Policy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_fn(state, Q, epsilon, num_actions):\n",
    "    # Create a distribution of actions and divide the epsilon probability between all actions\n",
    "    action_dist = np.ones(num_actions, dtype=float) * epsilon / num_actions\n",
    "    # Use function approximation to get the estimated returns for the actions in the current state\n",
    "    estimated_returns = predict_returns(state, Q)\n",
    "    # Find the best action\n",
    "    best_action = np.argmax(estimated_returns)\n",
    "    # Set probability for the best action to (1 - epsilon)\n",
    "    action_dist[best_action] += (1.0 - epsilon)\n",
    "    return action_dist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learn The Optimal Action-Value Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu22/repos/reinforcement-learning-algorithms/venv-pre/lib/python3.10/site-packages/gymnasium/wrappers/rendering.py:282: UserWarning: \u001b[33mWARN: Overwriting existing videos at /home/ubuntu22/repos/reinforcement-learning-algorithms/videos/sarsa_fa folder (try specifying a different `video_folder` for the `RecordVideo` wrapper if this is not desired)\u001b[0m\n",
      "  logger.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Return: -5109.0\n",
      "Episode: 2 Return: -1299.0\n",
      "Episode: 3 Return: -478.0\n",
      "Episode: 4 Return: -523.0\n",
      "Episode: 5 Return: -504.0\n",
      "Episode: 6 Return: -431.0\n",
      "Episode: 7 Return: -285.0\n",
      "Episode: 8 Return: -507.0\n",
      "Episode: 9 Return: -445.0\n",
      "Episode: 10 Return: -233.0\n",
      "Episode: 11 Return: -411.0\n",
      "Episode: 12 Return: -217.0\n",
      "Episode: 13 Return: -256.0\n",
      "Episode: 14 Return: -242.0\n",
      "Episode: 15 Return: -321.0\n",
      "Episode: 16 Return: -233.0\n",
      "Episode: 17 Return: -152.0\n",
      "Episode: 18 Return: -247.0\n",
      "Episode: 19 Return: -159.0\n",
      "Episode: 20 Return: -171.0\n",
      "Episode: 21 Return: -198.0\n",
      "Episode: 22 Return: -183.0\n",
      "Episode: 23 Return: -162.0\n",
      "Episode: 24 Return: -161.0\n",
      "Episode: 25 Return: -276.0\n",
      "Episode: 26 Return: -156.0\n",
      "Episode: 27 Return: -162.0\n",
      "Episode: 28 Return: -228.0\n",
      "Episode: 29 Return: -162.0\n",
      "Episode: 30 Return: -149.0\n",
      "Episode: 31 Return: -190.0\n",
      "Episode: 32 Return: -160.0\n",
      "Episode: 33 Return: -164.0\n",
      "Episode: 34 Return: -169.0\n",
      "Episode: 35 Return: -187.0\n",
      "Episode: 36 Return: -136.0\n",
      "Episode: 37 Return: -119.0\n",
      "Episode: 38 Return: -121.0\n",
      "Episode: 39 Return: -190.0\n",
      "Episode: 40 Return: -179.0\n",
      "Episode: 41 Return: -161.0\n",
      "Episode: 42 Return: -149.0\n"
     ]
    }
   ],
   "source": [
    "# Number of episodes to train on\n",
    "episodes = 42\n",
    "# Return (accumulation of all rewards over an episode)\n",
    "G = 0\n",
    "returns = []\n",
    "# Discount ratio\n",
    "gamma = 0.99\n",
    "# Probability of taking a non-greedy action\n",
    "epsilon = 0.001\n",
    "\n",
    "env = gym.make(ENV_TYPE, render_mode=\"rgb_array\")\n",
    "env = gym.wrappers.RecordEpisodeStatistics(env, buffer_length=episodes)\n",
    "\n",
    "if SAVE_VIDEOS:\n",
    "    os.makedirs(VIDEOS_ROOT_DIR, exist_ok=True)\n",
    "    env = gym.wrappers.RecordVideo(\n",
    "        env=env, \n",
    "        video_folder=VIDEOS_ROOT_DIR,\n",
    "        episode_trigger=lambda episode: episode % 10 == 0,\n",
    "        video_length=0,\n",
    "        name_prefix=ENV_TYPE,\n",
    "        fps=10\n",
    "        )\n",
    "\n",
    "        \n",
    "for episode in range(1, episodes+1):\n",
    "    terminated, truncated = False, False\n",
    "    G, reward = 0, 0\n",
    "    state, info = env.reset()\n",
    "    action_dist = policy_fn(state, Q, epsilon, num_actions)\n",
    "    action = np.random.choice(np.arange(num_actions), p=action_dist)\n",
    "\n",
    "    while not terminated:\n",
    "        # Take the action and observe reward and next state\n",
    "        next_state, reward, terminated, truncated, info = env.step(action)\n",
    "        # Select the action following the epsilon-greedy policy\n",
    "        next_action_dist = policy_fn(next_state, Q, epsilon, num_actions)\n",
    "        next_action = np.random.choice(np.arange(num_actions), p=next_action_dist)\n",
    "\n",
    "        # Update the expected return for the action-value function\n",
    "        next_estimated_returns = predict_returns(next_state, Q)\n",
    "        td_target = reward + gamma * next_estimated_returns[next_action]\n",
    "        Q[action].partial_fit([convert_to_features(state, scaler)], [td_target])\n",
    " \n",
    "        G += reward\n",
    "        state = next_state\n",
    "        action = next_action\n",
    "\n",
    "    returns.append(G)\n",
    "\n",
    "    if episode % 1 == 0:\n",
    "        print(f'Episode: {episode} Return: {G}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot The Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAHHCAYAAABwaWYjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYMklEQVR4nO3deVxU1f8/8NewzAzrALILKm4obiQk4r6Q6MdKSnOt0ExzK7fcylBbfpRmqVma9S2zNLfS/JipZGUfE8kNd8l9Y3OBGWQbmDm/P5CbE2gMMjPM+Ho+HvOAuffMnTPcal6d+77nyIQQAkRERERUZXaW7gARERGRtWGAIiIiIjISAxQRERGRkRigiIiIiIzEAEVERERkJAYoIiIiIiMxQBEREREZiQGKiIiIyEgMUERERERGYoAiIrJhMpkMc+fONet7Dh8+HA0aNDDrexKZGwMUkZVbuXIlZDKZ9HBwcEDdunUxfPhwXLt2rVrHPHnyJObOnYuLFy/WbGfN7ObNm5g2bRpCQ0OhVCrh5eWF2NhYbN261dJdq1SDBg0MzuXdj969e1u6e0R0FwdLd4CIasabb76JkJAQFBUVYd++fVi5ciX27NmD48ePQ6lUGnWskydPYt68eejWrZvVjiSkpaWhZ8+euH79OkaMGIHIyEjk5uZi9erVeOKJJ/Dqq69iwYIFlu5mBeHh4Zg6dWqF7YGBgdU6XmFhIRwc+J96oprGf6uIbESfPn0QGRkJAHjxxRfh7e2N9957D1u2bMHAgQMt3Lsy+fn5cHFxMfn7lJSUYMCAAcjJycHvv/+OqKgoad/kyZMxbNgwvP/++4iMjMSgQYNM3p9ypaWl0Ov1kMvl92xTt25dPPvsszX2nsaGZyKqGl7CI7JRnTt3BgCcO3fOYPvp06cxYMAAeHl5QalUIjIyElu2bJH2r1y5Es888wwAoHv37tIlpN9++w3AvWtqGjRogOHDhxscRyaTYffu3Rg3bhx8fX0RFBQEAOjWrRtatmyJkydPonv37nB2dkbdunUxf/78Csf96KOP0KJFCzg7O8PT0xORkZFYs2bNfT/7d999h+PHj2PmzJkG4QkA7O3t8emnn8LDw0P6HFlZWXBwcMC8efMqHCstLQ0ymQxLly6VtuXm5mLSpEkIDg6GQqFA48aN8d5770Gv10ttLl68CJlMhvfffx+LFi1Co0aNoFAocPLkyfv2vSqGDx8OV1dXnD9/HrGxsXBxcUFgYCDefPNNCCEM2v7zfOXl5WHSpElo0KABFAoFfH198dhjj+HQoUMGr9uwYQMiIiLg5OQEb29vPPvss5VeEt68eTNatmwJpVKJli1bYtOmTZX2Wa/XY9GiRWjRogWUSiX8/Pzw0ksvIScnx6DdgQMHEBsbC29vbzg5OSEkJAQvvPBCNf9SRKbDESgiG1Vev+Tp6SltO3HiBDp27Ii6deti5syZcHFxwfr16xEXF4fvvvsOTz31FLp06YJXXnkFS5YswWuvvYbmzZsDgPTTWOPGjYOPjw8SEhKQn58vbc/JyUHv3r3x9NNPY+DAgdi4cSNmzJiBVq1aoU+fPgCAzz77DK+88goGDBiAiRMnoqioCEePHkVKSgqGDh16z/f873//CwB4/vnnK92vUqnQr18/fPXVVzh79iwaN26Mrl27Yv369ZgzZ45B23Xr1sHe3l4KlQUFBejatSuuXbuGl156CfXq1cPevXsxa9YsZGRkYNGiRQav//LLL1FUVITRo0dDoVDAy8vrvn+vkpIS3Lhxo8J2FxcXODk5Sc91Oh169+6N9u3bY/78+di+fTvmzJmD0tJSvPnmm/c8/pgxY7Bx40ZMmDABYWFhuHnzJvbs2YNTp06hbdu2AMrC74gRI/Doo48iMTERWVlZWLx4Mf744w8cPnwYHh4eAICdO3eif//+CAsLQ2JiIm7evIkRI0ZIQfluL730knTcV155BRcuXMDSpUtx+PBh/PHHH3B0dER2djZ69eoFHx8fzJw5Ex4eHrh48SK+//77+/7NiCxCEJFV+/LLLwUA8fPPP4vr16+LK1euiI0bNwofHx+hUCjElStXpLY9e/YUrVq1EkVFRdI2vV4vOnToIJo0aSJt27BhgwAgfv311wrvB0DMmTOnwvb69euL+Pj4Cv3q1KmTKC0tNWjbtWtXAUCsWrVK2lZcXCz8/f1F//79pW39+vUTLVq0MObPIYQQIjw8XKhUqvu2+eCDDwQAsWXLFiGEEJ9++qkAII4dO2bQLiwsTPTo0UN6/tZbbwkXFxfx119/GbSbOXOmsLe3F5cvXxZCCHHhwgUBQLi7u4vs7Owq9bt+/foCQKWPxMREqV18fLwAIF5++WVpm16vF3379hVyuVxcv35d2v7P86VSqcT48ePv2QetVit8fX1Fy5YtRWFhobR969atAoBISEiQtoWHh4uAgACRm5srbdu5c6cAIOrXry9t+9///icAiNWrVxu81/bt2w22b9q0SQAQ+/fvr8Jfi8iyeAmPyEbExMTAx8cHwcHBGDBgAFxcXLBlyxZpNODWrVv45ZdfMHDgQOTl5eHGjRu4ceMGbt68idjYWJw5c6bad+3dz6hRo2Bvb19hu6urq0Gtj1wuR7t27XD+/Hlpm4eHB65evYr9+/cb9Z55eXlwc3O7b5vy/RqNBgDw9NNPw8HBAevWrZPaHD9+HCdPnjSok9qwYQM6d+4MT09P6W9448YNxMTEQKfT4ffffzd4n/79+8PHx6fKfY+KikJSUlKFx5AhQyq0nTBhgvS7TCbDhAkToNVq8fPPP9/z+B4eHkhJSUF6enql+w8cOIDs7GyMGzfOoH6qb9++aNasGX788UcAQEZGBlJTUxEfHw+VSiW1e+yxxxAWFmZwzA0bNkClUuGxxx4z+JtFRETA1dUVv/76q9Q3ANi6dStKSkr+5S9FZFm8hEdkIz7++GM0bdoUarUaX3zxBX7//XcoFApp/9mzZyGEwBtvvIE33nij0mNkZ2ejbt26NdqvkJCQSrcHBQVBJpMZbPP09MTRo0el5zNmzMDPP/+Mdu3aoXHjxujVqxeGDh2Kjh073vc93dzcKr0Mdre8vDypLQB4e3ujZ8+eWL9+Pd566y0AZZfvHBwc8PTTT0uvO3PmDI4ePXrPUJSdnW3w/F6f/168vb0RExPzr+3s7OzQsGFDg21NmzYFgPtOPzF//nzEx8cjODgYERER+M9//oPnn39eOtalS5cAAKGhoRVe26xZM+zZs8egXZMmTSq0Cw0NNaipOnPmDNRqNXx9fSvtU/nfrGvXrujfvz/mzZuHDz/8EN26dUNcXByGDh1q8M8yUW3AAEVkI9q1ayfdhRcXF4dOnTph6NChSEtLg6urq1Tg/OqrryI2NrbSYzRu3Lja76/T6Srdfnfdzt0qG5UCYFAE3bx5c6SlpWHr1q3Yvn07vvvuO3zyySdISEiotOD77telpqbi8uXLqFevXqVtyoPa3aMlgwcPxogRI5Camorw8HCsX78ePXv2hLe3t9RGr9fjsccew/Tp0ys9bnmIKXevz28pAwcOROfOnbFp0ybs3LkTCxYswHvvvYfvv/9eqj2raXq9Hr6+vli9enWl+8vDqEwmw8aNG7Fv3z7897//xY4dO/DCCy9g4cKF2LdvH1xdXU3SP6LqYIAiskH29vZITExE9+7dsXTpUsycOVMaYXB0dPzXEY5/jgzdzdPTE7m5uQbbtFotMjIyHrjflXFxccGgQYMwaNAgaLVaPP3003jnnXcwa9ase96i//jjj+Pbb7/FqlWrMHv27Ar7NRoNfvjhBzRr1swgNMbFxeGll16SLuP99ddfmDVrlsFrGzVqhNu3b1dplMiU9Ho9zp8/bxDY/vrrLwD417m7AgICMG7cOIwbNw7Z2dlo27Yt3nnnHfTp0wf169cHUHb3YY8ePQxel5aWJu0v/3nmzJkKx09LSzN43qhRI/z888/o2LFjlQJl+/bt0b59e7zzzjtYs2YNhg0bhrVr1+LFF1/819cSmQtroIhsVLdu3dCuXTssWrQIRUVF8PX1Rbdu3fDpp59WGnauX78u/V4+V9M/gxJQ9mX4zzqfFStW3HME6kHcvHnT4LlcLkdYWBiEEPetkRkwYADCwsLw7rvv4sCBAwb79Ho9xo4di5ycnAp33Hl4eCA2Nhbr16/H2rVrIZfLERcXZ9Bm4MCBSE5Oxo4dOyq8b25uLkpLS438lNV399QKQggsXboUjo6O6NmzZ6XtdTod1Gq1wTZfX18EBgaiuLgYABAZGQlfX18sX75c2gYAP/30E06dOoW+ffsCKAth4eHh+OqrrwyOmZSUVGGqhoEDB0Kn00mXRu9WWloq/XOWk5NTYRqG8PBwADDoC1FtwBEoIhs2bdo0PPPMM1i5ciXGjBmDjz/+GJ06dUKrVq0watQoNGzYEFlZWUhOTsbVq1dx5MgRAGVfWvb29njvvfegVquhUCjQo0cP+Pr64sUXX8SYMWPQv39/PPbYYzhy5Ah27NhhcJmrpvTq1Qv+/v7o2LEj/Pz8cOrUKSxduhR9+/a9b5G4XC7Hxo0b0bNnT3Tq1MlgJvI1a9bg0KFDmDp1KgYPHlzhtYMGDcKzzz6LTz75BLGxsVJhc7lp06Zhy5YtePzxxzF8+HBEREQgPz8fx44dw8aNG3Hx4sUH+ltcu3YN33zzTYXtrq6uBmFOqVRi+/btiI+PR1RUFH766Sf8+OOPeO211+5Zn5WXl4egoCAMGDAAbdq0gaurK37++Wfs378fCxcuBFA2Qvnee+9hxIgR6Nq1K4YMGSJNY9CgQQNMnjxZOl5iYiL69u2LTp064YUXXsCtW7ekebtu374ttevatSteeuklJCYmIjU1Fb169YKjoyPOnDmDDRs2YPHixRgwYAC++uorfPLJJ3jqqafQqFEj5OXl4bPPPoO7uzv+85//VPtvSmQSFr0HkIgeWPl0AZXd+q3T6USjRo1Eo0aNpKkEzp07J55//nnh7+8vHB0dRd26dcXjjz8uNm7caPDazz77TDRs2FDY29sbTGmg0+nEjBkzhLe3t3B2dhaxsbHi7Nmz95zGoLJ+de3atdLpCeLj4w1uf//0009Fly5dRJ06dYRCoRCNGjUS06ZNE2q1ukp/m+zsbDFlyhTRuHFjoVAohIeHh4iJiZGmLqiMRqMRTk5OAoD45ptvKm2Tl5cnZs2aJRo3bizkcrnw9vYWHTp0EO+//77QarVCiL+nMViwYEGV+irE/acxuPvvEh8fL1xcXMS5c+dEr169hLOzs/Dz8xNz5swROp3O4Ji4axqD4uJiMW3aNNGmTRvh5uYmXFxcRJs2bcQnn3xSoS/r1q0TjzzyiFAoFMLLy0sMGzZMXL16tUK77777TjRv3lwoFAoRFhYmvv/++wrnsdyKFStERESEcHJyEm5ubqJVq1Zi+vTpIj09XQghxKFDh8SQIUNEvXr1hEKhEL6+vuLxxx8XBw4cqPLfkMhcZEL8Y7yUiIhqteHDh2Pjxo0GozxEZF6sgSIiIiIyEgMUERERkZEYoIiIiIiMxBooIiIiIiNxBIqIiIjISAxQREREREbiRJomoNfrkZ6eDjc3t/suiUFERES1hxACeXl5CAwMhJ3d/ceYGKBMID09HcHBwZbuBhEREVXDlStXEBQUdN82DFAmUL7ExJUrV+Du7m7h3hAREVFVaDQaBAcH33epqHIMUCZQftnO3d2dAYqIiMjKVKX8hkXkREREREZigCIiIiIyEgPUPXz88cdo0KABlEoloqKi8Oeff1q6S0RERFRLMEBVYt26dZgyZQrmzJmDQ4cOoU2bNoiNjUV2dralu0ZERES1AANUJT744AOMGjUKI0aMQFhYGJYvXw5nZ2d88cUXlu4aERER1QIMUP+g1Wpx8OBBxMTESNvs7OwQExOD5OTkSl9TXFwMjUZj8CAiIiLbxQD1Dzdu3IBOp4Ofn5/Bdj8/P2RmZlb6msTERKhUKunBSTSJiIhsGwNUDZg1axbUarX0uHLliqW7RERERCbEiTT/wdvbG/b29sjKyjLYnpWVBX9//0pfo1AooFAozNE9IiIiqgU4AvUPcrkcERER2LVrl7RNr9dj165diI6OtmDPiIiIqLbgCFQlpkyZgvj4eERGRqJdu3ZYtGgR8vPzMWLECEt3jYiIiGoBBqhKDBo0CNevX0dCQgIyMzMRHh6O7du3VygsJyIiooeTTAghLN0JW6PRaKBSqaBWq7mYMBERmV1xqQ638rUQArC3k5U9ZDLY3fV7+XY7WdUWz30YGPP9zREoIiIiK6HXC9zM1yJLU3TnUYxMTRGy1EXIyitCprpse05BiVHHVTraoVeYP+I71Efbep4MVFXAAEVERLVacakOBcU6eDg7PpRf7NmaIiSdysLOE1lIPn8T2lJ9lV7nYCeDTAbo9AL6f7nWVFSix5Yj6dhyJB0tAt0RH90AT4YHQuloXwOfoGYVleiQW1AChYMdPF3kFusHL+GZAC/hERnKLdDiiz0XsO14Jv7TKgBjuzaCk7xm/sOck69F0qksONrL4OEsh4eTIzyd5fB0lsNN6QA7u9rzhavXC2TnlY0YlOr00AtALwSEAIQQECh7ri9/LgCFgx0iG3hB7lBzN03/lZWHb/ZdQolOwMnRHkpHuzs/y35X3vm9fJur0gEtA93hYF9zfSjV6XHsmhrX84pxK1+LWwVa3Lpd9jMnX2uwLV+rAwAEqpRo37AO2jesg+hGdRDk6VTtQFVUosPZ7Ns4fyMfMgByBzvI7e3gaG8HuYMdHO1l0u9yezs43tmmcLCHwsEOCgc7k4a5s9m3kXQyCztPZuLw5VyDfTIZUMdFAX+VAn5uSviplPB3V8LPXQE/dyX8VUr4uSkNAqe4889VqV4PvR7QCVEWrPQCOiFwNacQq/ddwg9H0qWA5uHsiEGPBuPZqPoI9nI2yecUQuBWvhYZd0bObuVroS4sQU6BFjkFJcgt0CK3oET6PadAi6KSsv5NjmmKiTFNarQ/xnx/M0CZAAMUUZmcfC3+b88FrNx7EbeLS6XtASolZv2nOZ5oHVDtL6ECbSm+/OMilv92Dnl3HftudjJAdSdQeTg7wsNZDi8XOUK8XdDE1xVN/NxQz8sZ9jUUsoQQyCkowdWcAly5VYgrOQW4cqsAV3IKcfVWAa7mFlZ59OBuzQPcsXhwOJr6uT1w/7798wrm/fcEio3sh5+7AoMigzGoXT3U9XCqdh8y1IVYt/8K1v55BZmaomofBwDqejghqqFXWaBqWOeeX/LZeUU4lZGHUxka6XHuej50/zYs8y/kd4JU2aMsWMkd7KBwtIeTox183ZQIUJUFmrKfTghUKVHHVVHhnzm9XiD1ai52nigLTeev5xvsDw/2QK8Wfohp7ocQbxc41mCYvVtOvhbrDlzB18mXcC23EEBZYOvZzA/xHeqjU2PvKv87W6LTI6dAiyx1MTLUhcjUFCE9twiZ6kJkqIuQoS5CpqaoWv9O2NvJ8FKXhpjeu5nRr70fBigLY4Cih92tfC0+/995fLX3ojR60MzfDf3C6+KbfX//h7ldAy8kPBGGlnVVVT52qU6P9QeuYtHPfyE7rxgA0NTPFb5uSuRI/7eqRcGd9/03cgc7NPJxLQtUd0JVEz9X1PdylkZc9HqBnAItbtzW4npeMW7cLjb4ef3Oz6s5hQZBsTL2djL4uSmgcLSHDGVfTnYymfQTKPtpZwfIIMPlWwVQF5ZA7mCHWX2aIT66QbVG1dSFJXjt+2P48VgGAKBzE29E1vdCUakOhVodikt1KCrRo1Crk7YVlepRXKJDhroI6sKSO30Duof6YmhUPXQL9a1S+NTpBX4/cx1rUi5j16ks6XKSyskRId4u8HIpGzGs41r208vFEV4uCni5lIXfOi4KONjLcPhyLvadv4nk8zdx5EouSv8RgOp6OKF9wzpoHaTCtdxCKSzduK2ttF8ezo5o4usKezsZtKV6lOgESnR6aHX6O8/vbCvVo/jOtprgYCeTRor8VUooHOzwvzM3cP3OP88A4GgvQ4dG3ngszA+PhfnBz11ZI+9dVTq9wC+ns7Eq+SL+d+aGtL2hjwsGPxoMJ7kDNIVlo0LqwhLpkVtQAs2d3/Or+O8gAHi7lo2olY8ee975H57ynx7Of48se7g4wk3hYJIRQAYoC2OAoofVzdvF+Ox/F7Aq+aIUYJoHuGNiz8boFeYPOzsZikp0+HT3eSzbfRZFJXrIZMDgR+vh1V5NUcf13jP6CyGw40Qm5u9Ik/7vPMjTCdNiQ/FE68AKoaK4VAf1naH/smBVdkngRl4xzl2/jTPZt3E2+/Y9R2Lk9nYI8nTC7eJS3MzXGjVa4eumQLCXM4I9ne78dEbQnd8DVEqjLoVl5xVh+saj+C3tOgCgS1MfvD+gNXyN+EI9fDkHL397GFdzCuFgJ8P03qF4sVPDKgcxbakeO05kYk3KZSSfvyltr+vhhMGPBmPgo8GVfsFn5xVhw4GrWJNyWQrNANAuxAvDouqhd0t/KByqdym3QFuKg5dyygLVuZs4elVdIVCVk8mAEG8XNA9wR1iAO5oHuKF5gDv83ZVGfQkLIVCiE9DqyoJlcan+zkOH4pKy37V3nudrdcjW3BllURchXV0oFXjf6x8lV4UDujfzRa8wP3QN9YG70rE6f5oadzb7Nr5OvoiNB68aFYqAsr+9j6sCAR5OCHC/ezROiUAPpzuXHpU1eon6QTBAWRgDFFVXboEWa/dfQX0vZ0Q28IKPm3UsEXTjdjFW/H4eXydfQmFJ2X9gWwS6Y2LPJngszK/SL6lruYVI3HYKW4+WjYi4Kx0wKaYpnouuX+HyRMr5m0j86TRSr+QCALxc5Hi5R2MMjapX7S9goOz/sq/mFOBMVlmgOpOVJwWr8s9xN09nR/i4KeDtqqj0Z10PJwR5OtV44a0QAt/su4S3fzyF4lI9PJ0dkfh0a/RuWfnyUuX0eoEV/zuP93ekoVQvEOzlhI+GtEV4sEe1+3Lu+m18m3IZGw9dRe6dO73s7WSIae6LYVH10bGxN5LP3cTqlEtIOpklhRp3pQP6RwRhWFQ9NPZ9sEuRlckv/jtQncrQINjLGc0D3NE8wB2hfm41VnP3oEp1ety4rUXGXZex1AVaRDTwQvuGXg/0z7Op5RWVYNPha/jldDYUDnbwcJJD5ewIlZMj3J0c4eFU9nv5w8PZEW5Kxxq7RG4ODFAWxgBl+4pKdPjldDY6NvaGyqlm/i+xqESHYZ+n4OClHGlbQ28XPNrAC4+GeKFdAy8Ee1W/aNYUbt4uxvLd5/D1vktSYWeruipM7NkEPZv7Vqmvf164hblbTuBkhgYA0MTXFQlPhKFzEx+cztRg/vY0/HI6GwDg5GiPUZ1DMKpLQ7iZ8P/O9XqBa7mFuHKrAO5OjvB2VaCOq9xkdSdVdTY7D5PWpeL4tbK/1cDIICQ80QKuioo3VF/PK8aU9anS5Ze+rQOQ+HSrGhvVKCrR4afjGViTchn7L/79z6yTo71B+GxbzwPDouqjb+uAWnlHF9HdGKAsjAHKtgkhMPrrg0g6mYVm/m5Y91L0A4coIQSmrD+CTYevwU3pgLoeTkjLysM//+30c1fg0QZeaBfihUcbeCHUzw0yWVl9y43bZXcu3bxdjBt3ft68rcXN/GLcuK1FgbYUjzX3x/AODaByfrD+5heX4v/2XMCK389LNT9tglSYGNME3UOrFpzuptMLrNt/Be/vTMOt/LJ6lVZ1VTierpYmAhzSLhiv9GwCXzfz1oLUNtpSPT78+S8s330OQgD16zjjw0HhaFvPU2qz58wNTFqXihu3i6F0tMPcJ1pg0KPBJgvff2XlYU3KZXx36CryikrhpnDAU23rYmhUPTTz538DyXowQFkYA5RtW777HN796bT0vF0DL6wa2e6B/u966S9n8P7Ov2BvJ8NXI9qhUxNvqAtKcODSLfx58Rb2X7iFY9fUKNEZ/uvqLLeHtlR/z9qPyrgqHPB8dH2M7BRy35qjypTo9Fi7/woW/3wGN26XFby2qqvClF5N0a2pzwN/QasLSrBo119YlXxJqjnq2yoAU3s1RUMf1wc6tq1JOX8TU9YfwbXcQtjbyTChe2OM7dYIS3adwbI74aqpnyuWDm37wHfvVVWhVoeTGRo0D3CDs5zTDJL1YYCyMAYo25V87iaGfb4PegGM7tIQ3/55GXlFpYhp7ovlz0ZUa56cbccyMG71IQDA23Et8Wz7+pW2K9TqkHolF39euIX9F2/h0OUcgzvN3JQOZZeaXMruZqrjqoC3S9nPOq5yFGp1+L89F3A6Mw9A2czDw6LqY3SXhv96h48QAj8ey8D7O9Jw8WYBgLKRj1d7haJvq4Aan2vpTFYefkhNR0yY3wPV69g6TVEJ5vxwApsOXwMAuCkcpCkdhkbVQ8LjYbxsRmQEBigLY4AyrR9Sr+GjX87i5R6N0S+8rtneN1tThP8s2YMbt4vxdNu6WPhMG+y/mIPn/i8FxaV69G8bhAUDWhsVJo5cycWgFckoKtFjRMcGmPNEiyq/tkSnx6WbBXBR2MPLRV6l4lO9XmDX6Wws/eUMjlxVAyi72+yZyCCM6dqo0nl09p69gXe3n8bRO+29XeV4pWcTDH60Xq25c+Zht+VIOl7fdKzs8pnSAe8+3Rp9WwdYultEVocBysIYoExDCIFPfz8vXT6TO9jhh/Ed0TzA9H/jEp0ewz5LwZ8Xb6GZvxs2jeso3dXz88ksvPTNQej0AqM6h+C1/zSv0qWsDHUh+i39A9l5xege6oPP4x81290qQgj878wNLP3lLP68eAtA2dw0cY/UxbhujdDQxxUn0tV4b3safv+r7PZ5F7k9RnVpiBc7N6y0aJksKz23EFuPpqNPywCTzRpNZOsYoCyMAarm6fUCb249iZV7LwIoW9IhXV2ERj4u+O/LnUxeb/H/tp3Cit/Pw1XhgC0TOlaox9l48Cpe3XAEADCzTzOM6drovsfLLy7FM8uTcTJDg1A/N2wcG23Su8ruJ+X8TSz99ax0t5adrGzW40N3lo9wtJdhWFR9TOjRGN5G1kwREVkTY76/Of5OtV5RiQ4vf3tYCk+z+zbH1lc6w89dgXPX8zHnhxMmff/txzOx4vfzAID3n2ldaTHzgIggvP6f5gCAd386jfX7r9zzeHq9wOR1qTiZoUEdFzk+j4+0WHgCgKiGdfD1yChsHt8RMc39oBeQwtOTbQLx85SumPtkC4YnIqK7cByeajV1YQlGrzqAlAu34Ggvw8KB4XiyTSAAYPHgRzD0s33YcPAqOjb2RtwjNV8PdeFGPqbdGVka1TkEvVveu65kVJeGuJmvxfLd5zDz+6NQOTsitkXFiQ7n70jDzpNZkDvYYcXzEbXmckt4sAc+j4/EqQwNfjqWgcfC/NEqqOpLrBARPUw4AkW1Vqa6CAOXJyPlwi24Khzw1Yh2UngCgPYN6+DlHmUrcb++6Rgu3Mi/16GqpVCrw9hvDiKvuBSPNvCs0qKVM3qHYlBkMPQCePnbw0g+d9Ng//oDV7B89zkAwPz+rRFR36tG+1wTmge4Y0qvUIYnIqL7YICiWumvrDw8/ckfSMvKg6+bAutfikaHxt4V2r3cozHahXghX6vDy98eQnGpces03YsQArM3H8fpzDx4uyqwdGjbKs1CLZPJ8M5TLdErzA/aUj1GrTqA49fK7l7bd/4mXt90DADwSo/GJhkxIyIi82CAolpn/8VbGLBsL9LVRWjo44Lvx3VAWGDlxXwO9nZYPDgcns6OOH5NYzDB5YNYu/8Kvjt0FXYy4KMhjxi1ErqDvR2WDHkE7Rt64XZxKYZ/+Sd+S8vGmG8OokQn0Ld1ACbFNK2RfhIRkWUwQFGtsv14BoZ9ngJNUSna1vPAd2M6IMjz/jVCASonvP9MGwDAl39cRNLJrAfqw/FraszZUlaYPi22GaIb1TH6GEpHe3z2fCRaBLrjxm0thn+5H7kFJWgTpMLCZ9rU+MSTRERkXgxQVGusSr6IsasPQVuqR0xzP6x+sT08XeRVem3P5n4Y2SkEADBt4xFkqAur1YfcAi3GfHNQ6sOYrg2rdRwAcFM6YuWIdmhQpywABqiU+Oz5SM4MTURkA3gXHlmUXi/w21/Z+Px/F7D3TsH10Kh6ePPJFkYvizK9dyj+vLNm3MRvU7FmVJRRx9Dryxb0vZpTiHpezlg4sM0Dr+3m46bAmlHt8e2fl/F02yD4GnEpkIiIai9OpGkCnEjz3xWV6PD9oWv4vz3nce562d1z9nYyTOrZBBN6NK52cLl4Ix+Pf7QHt4tL8UrPJpjy2L/XGmVpivD9oWvYcPAKzl/Ph9zBDt+P7YCWdXkXGhHRw8SY72+OQJFZXc8rxtf7LuGbfZdwK18LoGwB1MHtgjG8Ywjqejg90PEbeLvgnadaYuLaVHz0yxm0b+iFDo0q3r2nLdVj16ksrD9wBbv/ug79nf+NcHK0x7v9WzE8ERHRfTFAEY5dVWPCt4cQ7OmMzk280aWpD5r5uz3w5au7/ZWVh8//dx6bD6dDq9MDAOp6OOGFTiEYGBlUozNx9wuviz/O3sD6A1cxaW0qfprYGXXuzKJ9Il2NDQeu4ofUa8gpKJFeE1nfE89EBuE/rQIsOis4ERFZB17CMwFru4SX8MNxrEq+ZLDNx01RFqaa+KBTE2+jl/EQQuBWvhZHr6nx5R8XpQVpgbIZr0d1bojYFn5G1zlVVYG2FE8u/QNns2+ja1MfdA/1wfoDV3EyQyO18XNX4Om2QRgQEYRGlSzPQkREDxcuJmxh1hagnvrkDxy+nIu48ECoC0uw7/wtFJYYTkjZItAdXZr6oHMTb0TW94JOL5CuLkR6biEycotwLbfs97JtRUjPLURxqV56vZ0MiG3hjxc7h5ht9u3TmRo8ufQPaO/qh9zeDo+F+WFAZBA6N/Y2WYAjIiLrwxooqrJSnR6n7ozKvNyzCRr5uKK4VIeDF3Ow+8x1/O+vGziZocGJ9LLHst/OwdFehhJd1XK3n7sCfVoG4IWOIahXx7xrvjXzd8fbcS0x87ujaB7gjmcigtAvvG6Vp0YgIiK6Fwaoh9zZ67dRVKKHq8IBIXVcAAAKB3t0aOyNDo29MasPkJ1XhD/O3sDvf93A/85cx43bZcXfLnJ7BHo4SY+6Hsq/n6uc4KdSQOFg2TmPBkYGo194oMX7QUREtoUB6iF37GrZOm1hge73nB3b102Jpx4JwlOPBEGvF7iWWwh3J0e4Kx1qtNDcVBieiIiopjFAPeTKF7ptVcXb9u3sZAj2Mu+lOCIiotqGFbQPuePpZfVPVQ1QRERExAD1UNPpBU7eCVCcOJKIiKjqGKAeYueu30ZhiQ4ucns09HaxdHeIiIisBgPUQ6y8gLxFoOqeBeRERERUEQPUQ+zYnQJyXr4jIiIyDgPUQ+y4FKBq/2zpREREtQkD1ENKpxc4wTvwiIiIqoUB6iF1/k4BubPcHg25kC4REZFRGKAeUuX1T2EB7rBnATkREZFRGKAeUiwgJyIiqj4GqIeUsUu4EBER0d8YoB5CBgXkQQxQRERExmKAeghduHEbBVodnBzt0YgF5EREREZjgHoIHb9WNvoUFsgCciIioupggHoIHWP9ExER0QNhgHoI8Q48IiKiB8MA9ZDR6wVOcgZyIiKiB8IA9ZC5cDMft4tLoXS0QyMfF0t3h4iIyCoxQD1kyud/ah7gDgd7nn4iIqLq4DfoQ+bYVRaQExERPSgGqIcMC8iJiIgeHAPUQ0R/9wzkDFBERETVxgD1ELl4p4Bc4WCHJr6cgZyIiKi6GKAeIsdYQE5ERFQj+C36EDnOGciJiIhqhNUEqHfeeQcdOnSAs7MzPDw8Km1z+fJl9O3bF87OzvD19cW0adNQWlpq0Oa3335D27ZtoVAo0LhxY6xcubLCcT7++GM0aNAASqUSUVFR+PPPP03wicyvfA08BigiIqIHYzUBSqvV4plnnsHYsWMr3a/T6dC3b19otVrs3bsXX331FVauXImEhASpzYULF9C3b190794dqampmDRpEl588UXs2LFDarNu3TpMmTIFc+bMwaFDh9CmTRvExsYiOzvb5J/RlIQQOJ7OO/CIiIhqgkwIISzdCWOsXLkSkyZNQm5ursH2n376CY8//jjS09Ph5+cHAFi+fDlmzJiB69evQy6XY8aMGfjxxx9x/Phx6XWDBw9Gbm4utm/fDgCIiorCo48+iqVLlwIA9Ho9goOD8fLLL2PmzJlV6qNGo4FKpYJarYa7u3sNfOoHd/FGPrq9/xvkDnY4MS8WjqyBIiIiMmDM97fNfIsmJyejVatWUngCgNjYWGg0Gpw4cUJqExMTY/C62NhYJCcnAygb5Tp48KBBGzs7O8TExEhtKlNcXAyNRmPwqG3uLiBneCIiInowNvNNmpmZaRCeAEjPMzMz79tGo9GgsLAQN27cgE6nq7RN+TEqk5iYCJVKJT2Cg4Nr4iPVqL8LyGvHiBgREZE1s2iAmjlzJmQy2X0fp0+ftmQXq2TWrFlQq9XS48qVK5buUgXHeAceERFRjXGw5JtPnToVw4cPv2+bhg0bVulY/v7+Fe6Wy8rKkvaV/yzfdncbd3d3ODk5wd7eHvb29pW2KT9GZRQKBRQKRZX6aQlCCGkEqkUgAxQREdGDsmiA8vHxgY+PT40cKzo6Gu+88w6ys7Ph6+sLAEhKSoK7uzvCwsKkNtu2bTN4XVJSEqKjowEAcrkcERER2LVrF+Li4gCUFZHv2rULEyZMqJF+WsLlWwXQFJVCbm+Hpn5ulu4OERGR1bOaGqjLly8jNTUVly9fhk6nQ2pqKlJTU3H79m0AQK9evRAWFobnnnsOR44cwY4dOzB79myMHz9eGh0aM2YMzp8/j+nTp+P06dP45JNPsH79ekyePFl6nylTpuCzzz7DV199hVOnTmHs2LHIz8/HiBEjLPK5a0L55btmAW6QO1jNKSciIqq1LDoCZYyEhAR89dVX0vNHHnkEAPDrr7+iW7dusLe3x9atWzF27FhER0fDxcUF8fHxePPNN6XXhISE4Mcff8TkyZOxePFiBAUF4fPPP0dsbKzUZtCgQbh+/ToSEhKQmZmJ8PBwbN++vUJhuTUpD1Cc/4mIiKhmWN08UNagts0DNezzffjj7E0kPt0KQ9rVs3R3iIiIaqWHch4oqlxZATmXcCEiIqpJDFA27mpOIdSFJSwgJyIiqkEMUDauvP4p1J8F5ERERDWF36g2jgXkRERENY8BysYd5wzkRERENY4ByoYJIbiECxERkQkwQNmwqzmFyC0ogaO9DE39XS3dHSIiIpvBAGXDjt9VQK5wsLdwb4iIiGwHA5QNkwrIuYAwERFRjWKAsmG8A4+IiMg0GKBsVNkM5CwgJyIiMgUGKBt1LbcQOQUlcLCTIdSfM5ATERHVJAYoG1U++tTUzw1KRxaQExER1SQGKBt17no+AKBZAEefiIiIahoDlI3KLy4FALgrHS3cEyIiItvDAGWjCrQ6AICznJfviIiIahoDlI0qZIAiIiIyGQYoG1VQUhagnOQOFu4JERGR7WGAslGF2rIaKI5AERER1TwGKBvFGigiIiLTYYCyUeUByolzQBEREdU4Bigb9XcROWugiIiIahoDlI0qKCmrgXLiJTwiIqIaxwBloziNARERkekwQNkoFpETERGZDgOUDRJCoFCaB4oBioiIqKYxQNmgohI9hCj7nUXkRERENY8BygaVjz4BnMaAiIjIFBigbFDBnVnIFQ52sLeTWbg3REREtocBygbxDjwiIiLTYoCyQQWcRJOIiMikGKBskLSMC0egiIiITIIBygYV3pmFnJfwiIiITIMBygZxIWEiIiLTYoCyQbyER0REZFoMUDaId+ERERGZFgOUDfr7Eh7vwiMiIjIFBigbVKhlETkREZEpMUDZoAJewiMiIjIpBigbVFDCInIiIiJTYoCyQSwiJyIiMi0GKBtUvpiwE5dyISIiMgkGKBsk1UBxIk0iIiKTYICyQbyER0REZFoMUDaIM5ETERGZFgOUDSosKR+BYg0UERGRKTBA2aACTqRJRERkUgxQNoiX8IiIiEyLAcoGsYiciIjItBigbIy2VI9SvQAAOHMxYSIiIpNggLIx5aNPAC/hERERmQoDlI0pKCkrIHewk0HuwNNLRERkCvyGtTEsICciIjI9BigbwwJyIiIi02OAsjHSOnicRJOIiMhkGKBsTPkkmk5cSJiIiMhkrCJAXbx4ESNHjkRISAicnJzQqFEjzJkzB1qt1qDd0aNH0blzZyiVSgQHB2P+/PkVjrVhwwY0a9YMSqUSrVq1wrZt2wz2CyGQkJCAgIAAODk5ISYmBmfOnDHp56tJvIRHRERkelYRoE6fPg29Xo9PP/0UJ06cwIcffojly5fjtddek9poNBr06tUL9evXx8GDB7FgwQLMnTsXK1askNrs3bsXQ4YMwciRI3H48GHExcUhLi4Ox48fl9rMnz8fS5YswfLly5GSkgIXFxfExsaiqKjIrJ+5ulhETkREZHoyIYSwdCeqY8GCBVi2bBnOnz8PAFi2bBlef/11ZGZmQi6XAwBmzpyJzZs34/Tp0wCAQYMGIT8/H1u3bpWO0759e4SHh2P58uUQQiAwMBBTp07Fq6++CgBQq9Xw8/PDypUrMXjw4Cr1TaPRQKVSQa1Ww93dvSY/9r/6et8lvLH5OGJb+OHT5yLN+t5ERETWzJjvb6sYgaqMWq2Gl5eX9Dw5ORldunSRwhMAxMbGIi0tDTk5OVKbmJgYg+PExsYiOTkZAHDhwgVkZmYatFGpVIiKipLa1HaF0kLCLCInIiIyFasMUGfPnsVHH32El156SdqWmZkJPz8/g3blzzMzM+/b5u79d7+usjaVKS4uhkajMXhYCi/hERERmZ5FA9TMmTMhk8nu+yi//Fbu2rVr6N27N5555hmMGjXKQj03lJiYCJVKJT2Cg4Mt1hepiJx34REREZmMRa/zTJ06FcOHD79vm4YNG0q/p6eno3v37ujQoYNBcTgA+Pv7Iysry2Bb+XN/f//7trl7f/m2gIAAgzbh4eH37OOsWbMwZcoU6blGo7FYiCrgXXhEREQmZ9EA5ePjAx8fnyq1vXbtGrp3746IiAh8+eWXsLMzHDyLjo7G66+/jpKSEjg6OgIAkpKSEBoaCk9PT6nNrl27MGnSJOl1SUlJiI6OBgCEhITA398fu3btkgKTRqNBSkoKxo4de8++KRQKKBSKqn5sk/r7Eh5roIiIiEzFKmqgrl27hm7duqFevXp4//33cf36dWRmZhrUJQ0dOhRyuRwjR47EiRMnsG7dOixevNhgZGjixInYvn07Fi5ciNOnT2Pu3Lk4cOAAJkyYAACQyWSYNGkS3n77bWzZsgXHjh3D888/j8DAQMTFxZn7Y1dLYUl5ETlHoIiIiEzFKoYpkpKScPbsWZw9exZBQUEG+8pnYVCpVNi5cyfGjx+PiIgIeHt7IyEhAaNHj5badujQAWvWrMHs2bPx2muvoUmTJti8eTNatmwptZk+fTry8/MxevRo5ObmolOnTti+fTuUSqV5PuwDYhE5ERGR6VntPFC1mSXngRr4aTL+vHALS4c+gsdbB5r1vYmIiKzZQzEPFFWuqIRF5ERERKbGAGVjpEt4jlZxdZaIiMgqMUDZGC4mTEREZHoMUDamQMu78IiIiEyNAcrG8C48IiIi02OAsiE6vUBxqR4AFxMmIiIyJQYoG1J45w48AHDiWnhEREQmwwBlQ8rrn2QyQOnIU0tERGQq/Ja1IYXSFAb2kMlkFu4NERGR7WKAsiEFnMKAiIjILBigbAjvwCMiIjIPBigbIk2iyVnIiYiITKpaAaqwsBAFBQXS80uXLmHRokXYuXNnjXWMjFdeRM4RKCIiItOqVoDq168fVq1aBQDIzc1FVFQUFi5ciH79+mHZsmU12kGqukIuJExERGQW1QpQhw4dQufOnQEAGzduhJ+fHy5duoRVq1ZhyZIlNdpBqjoWkRMREZlHtQJUQUEB3NzcAAA7d+7E008/DTs7O7Rv3x6XLl2q0Q5S1f1dRM4aKCIiIlOqVoBq3LgxNm/ejCtXrmDHjh3o1asXACA7Oxvu7u412kGqusLyhYQ5CzkREZFJVStAJSQk4NVXX0WDBg0QFRWF6OhoAGWjUY888kiNdpCqjtMYEBERmUe1rvUMGDAAnTp1QkZGBtq0aSNt79mzJ5566qka6xwZhzVQRERE5lHtYhl/f3/4+/sbbGvXrt0Dd4iqr5ABioiIyCyqFaDy8/Px7rvvYteuXcjOzoZerzfYf/78+RrpHBmnoIRF5EREROZQrW/aF198Ebt378Zzzz2HgIAALlxbS0hF5ByBIiIiMqlqBaiffvoJP/74Izp27FjT/aEHwBooIiIi86jWXXienp7w8vKq6b7QA5LuwuM0BkRERCZVrQD11ltvISEhwWA9PLK8v4vIWQNFRERkStX6pl24cCHOnTsHPz8/NGjQAI6Ojgb7Dx06VCOdI+MUlHAxYSIiInOoVoCKi4ur4W5QTeA0BkREROZhdIAqLS2FTCbDCy+8gKCgIFP0iaqJReRERETmYXQNlIODAxYsWIDS0lJT9IeqSQiBwhIu5UJERGQO1Soi79GjB3bv3l3TfaEHUFSihxBlv7OInIiIyLSq9U3bp08fzJw5E8eOHUNERARcXFwM9j/55JM10jmqugLt3yOCnMaAiIjItKoVoMaNGwcA+OCDDyrsk8lk0Ol0D9YrMlp5/ZPCwQ72dpwZnoiIyJSqFaD+ufYdWV55/RMLyImIiEyvWjVQVPsUcBJNIiIis6nWt+2bb7553/0JCQnV6gxVX3kNFO/AIyIiMr1qBahNmzYZPC8pKcGFCxfg4OCARo0aMUBZACfRJCIiMp9qBajDhw9X2KbRaDB8+HA89dRTD9wpMh4XEiYiIjKfGquBcnd3x7x58/DGG2/U1CHJCByBIiIiMp8aLSJXq9VQq9U1eUiqovIaKBaRExERmV61vm2XLFli8FwIgYyMDHz99dfo06dPjXSMjFPAZVyIiIjMploB6sMPPzR4bmdnBx8fH8THx2PWrFk10jEyThEv4REREZlNtQLUhQsXarof9ICkInIGKCIiIpOrVg3UCy+8gLy8vArb8/Pz8cILLzxwp8h45ZfwnB1ZA0VERGRq1QpQX331FQoLCytsLywsxKpVqx64U2Q83oVHRERkPkYNV2g0GgghIIRAXl4elEqltE+n02Hbtm3w9fWt8U7Sv+NM5EREROZjVIDy8PCATCaDTCZD06ZNK+yXyWSYN29ejXWOqo4TaRIREZmPUQHq119/hRACPXr0wHfffQcvLy9pn1wuR/369REYGFjjnaR/x0t4RERE5mNUgOratSuAsrvw6tWrB5lMZpJOkfF4Fx4REZH5VKuIvH79+tizZw+effZZdOjQAdeuXQMAfP3119izZ0+NdpCqprD8LjzORE5ERGRy1QpQ3333HWJjY+Hk5IRDhw6huLgYQNlSLv/v//2/Gu0gVc3fS7lwBIqIiMjUqhWg3n77bSxfvhyfffYZHB0dpe0dO3bEoUOHaqxzVHW8hEdERGQ+1QpQaWlp6NKlS4XtKpUKubm5D9onqgYWkRMREZlPtQKUv78/zp49W2H7nj170LBhwwfuFBlHW6pHqV4A4EzkRERE5lCtADVq1ChMnDgRKSkpkMlkSE9Px+rVqzF16lSMHTu2pvtI/6J89AngJTwiIiJzqNZwxcyZM6HX69GzZ08UFBSgS5cuUCgUmDZtGl588cWa7iP9i4KSsgJyBzsZ5A7VysRERERkhGp928pkMrz++uu4desWjh8/jn379uH69etQqVQICQmp6T7Sv2ABORERkXkZFaCKi4sxa9YsREZGomPHjti2bRvCwsJw4sQJhIaGYvHixZg8ebKp+kr3wAJyIiIi8zIqQCUkJGDZsmVo0KABLly4gGeeeQajR4/Ghx9+iIULF+LChQuYMWOGSTr65JNPol69elAqlQgICMBzzz2H9PR0gzZHjx5F586doVQqERwcjPnz51c4zoYNG9CsWTMolUq0atUK27ZtM9gvhEBCQgICAgLg5OSEmJgYnDlzxiSfqaYUaDmJJhERkTkZFaA2bNiAVatWYePGjdi5cyd0Oh1KS0tx5MgRDB48GPb2phsB6d69O9avX4+0tDR89913OHfuHAYMGCDt12g06NWrF+rXr4+DBw9iwYIFmDt3LlasWCG12bt3L4YMGYKRI0fi8OHDiIuLQ1xcHI4fPy61mT9/PpYsWYLly5cjJSUFLi4uiI2NRVFRkck+24Mqn0STCwkTERGZiTCCo6OjuHr1qvRcqVSKo0ePGnOIGvPDDz8ImUwmtFqtEEKITz75RHh6eori4mKpzYwZM0RoaKj0fODAgaJv374Gx4mKihIvvfSSEEIIvV4v/P39xYIFC6T9ubm5QqFQiG+//bbKfVOr1QKAUKvV1fpsxtp2NF3Un7FV9P/kD7O8HxERkS0y5vvbqBEonU4HuVwuPXdwcICrq2sNR7p/d+vWLaxevRodOnSQZkJPTk5Gly5dDPoXGxuLtLQ05OTkSG1iYmIMjhUbG4vk5GQAZYskZ2ZmGrRRqVSIioqS2lSmuLgYGo3G4GFOLCInIiIyL6OKZoQQGD58OBQKBQCgqKgIY8aMgYuLi0G777//vuZ6eJcZM2Zg6dKlKCgoQPv27bF161ZpX2ZmZoU7AP38/KR9np6eyMzMlLbd3SYzM1Nqd/frKmtTmcTERMybN6/6H+wBFZSwiJyIiMicjBqBio+Ph6+vL1QqFVQqFZ599lkEBgZKz8sfVTVz5kzIZLL7Pk6fPi21nzZtGg4fPoydO3fC3t4ezz//PIQQxnwEk5g1axbUarX0uHLlilnfv1BaSJhF5EREROZg1Dful19+WaNvPnXqVAwfPvy+be5eGsbb2xve3t5o2rQpmjdvjuDgYOzbtw/R0dHw9/dHVlaWwWvLn/v7+0s/K2tz9/7ybQEBAQZtwsPD79lHhUIhjcpZAi/hERERmZdFhyx8fHzg4+NTrdfq9XoAZfVHABAdHY3XX38dJSUlUl1UUlISQkND4enpKbXZtWsXJk2aJB0nKSkJ0dHRAICQkBD4+/tj165dUmDSaDRISUmp1UvUSPNA8S48IiIis7CKdT9SUlKwdOlSpKam4tKlS/jll18wZMgQNGrUSAo/Q4cOhVwux8iRI3HixAmsW7cOixcvxpQpU6TjTJw4Edu3b8fChQtx+vRpzJ07FwcOHMCECRMAlM2wPmnSJLz99tvYsmULjh07hueffx6BgYGIi4uzxEevkgJOpElERGRWVlE04+zsjO+//x5z5sxBfn4+AgIC0Lt3b8yePVu6dKZSqbBz506MHz8eERER8Pb2RkJCAkaPHi0dp0OHDlizZg1mz56N1157DU2aNMHmzZvRsmVLqc306dORn5+P0aNHIzc3F506dcL27duhVCrN/rmr6u9LeFZxOomIiKyeTNSGKmwbo9FooFKpoFar4e7ubvL3G7f6ILYdy8S8J1sgvkMDk78fERGRLTLm+9sqLuHR/bGInIiIyLwYoGwAa6CIiIjMiwHKBhQyQBEREZkVA5QN+HsxYRaRExERmQMDlA3gCBQREZF5MUDZAK6FR0REZF4MUDaAd+ERERGZFwOUldPpBbSlZcvacDFhIiIi82CAsnLlBeQAL+ERERGZCwOUlSsvIJfJAIUDTycREZE58BvXykmTaDraQyaTWbg3REREDwcGKCvHhYSJiIjMjwHKyhVyCgMiIiKzY4CycpxEk4iIyPwYoKyctIwLAxQREZHZMEBZOV7CIyIiMj8GKCsnFZE7MkARERGZCwOUleNdeERERObHAGXlCu/UQDlzBIqIiMhsGKCsHBcSJiIiMj8GKCtXwGkMiIiIzI4ByspxHigiIiLzY4CycgUlLCInIiIyNwYoKycVkXMEioiIyGwYoKwca6CIiIjMjwHKynEiTSIiIvNjgLJyfxeRswaKiIjIXBigrFxBCRcTJiIiMjcGKCvHaQyIiIjMjwHKyrGInIiIyPwYoKyYEAKFJVzKhYiIyNwYoKxYUYkeQpT9ziJyIiIi82GAsmIFdybRBDiNARERkTkxQFmx8vonhYMd7O1kFu4NERHRw4MByoqV1z+xgJyIiMi8GKCsWAEn0SQiIrIIBigrVl4DxTvwiIiIzIsByopxEk0iIiLLYICyYlxImIiIyDIYoKwYR6CIiIgsgwHKipXXQLGInIiIyLwYoKxYAZdxISIisggGKCvGS3hERESWwQBlxaQicgYoIiIis2KAsmLSRJqOrIEiIiIyJwYoK1YoFZFzBIqIiMicGKCsGC/hERERWQYDlBXjYsJERESWwQBlxQp4Fx4REZFFMEBZsb8v4bGInIiIyJwYoKwYi8iJiIgsgwHKinExYSIiIstggLJiLCInIiKyDAYoK/b3Ui6sgSIiIjInBigrpS3Vo1QvAHAeKCIiInNjgLJS5aNPAGugiIiIzM3qAlRxcTHCw8Mhk8mQmppqsO/o0aPo3LkzlEolgoODMX/+/Aqv37BhA5o1awalUolWrVph27ZtBvuFEEhISEBAQACcnJwQExODM2fOmPIjVUtBSdkdeA52MsgdrO40EhERWTWr++adPn06AgMDK2zXaDTo1asX6tevj4MHD2LBggWYO3cuVqxYIbXZu3cvhgwZgpEjR+Lw4cOIi4tDXFwcjh8/LrWZP38+lixZguXLlyMlJQUuLi6IjY1FUVGRWT5fVXEZFyIiIsuxqgD1008/YefOnXj//fcr7Fu9ejW0Wi2++OILtGjRAoMHD8Yrr7yCDz74QGqzePFi9O7dG9OmTUPz5s3x1ltvoW3btli6dCmAstGnRYsWYfbs2ejXrx9at26NVatWIT09HZs3bzbXx6ySQs5CTkREZDFWE6CysrIwatQofP3113B2dq6wPzk5GV26dIFcLpe2xcbGIi0tDTk5OVKbmJgYg9fFxsYiOTkZAHDhwgVkZmYatFGpVIiKipLaVKa4uBgajcbgYWoFvAOPiIjIYqwiQAkhMHz4cIwZMwaRkZGVtsnMzISfn5/BtvLnmZmZ921z9/67X1dZm8okJiZCpVJJj+DgYCM+XfUU3JmFnAXkRERE5mfRADVz5kzIZLL7Pk6fPo2PPvoIeXl5mDVrliW7e0+zZs2CWq2WHleuXDH5e/ISHhERkeVY9PrP1KlTMXz48Pu2adiwIX755RckJydDoVAY7IuMjMSwYcPw1Vdfwd/fH1lZWQb7y5/7+/tLPytrc/f+8m0BAQEGbcLDw+/ZR4VCUaFvpsYiciIiIsuxaIDy8fGBj4/Pv7ZbsmQJ3n77bel5eno6YmNjsW7dOkRFRQEAoqOj8frrr6OkpASOjo4AgKSkJISGhsLT01Nqs2vXLkyaNEk6VlJSEqKjowEAISEh8Pf3x65du6TApNFokJKSgrFjx9bER64xBVzGhYiIyGKsogK5Xr16Bs9dXV0BAI0aNUJQUBAAYOjQoZg3bx5GjhyJGTNm4Pjx41i8eDE+/PBD6XUTJ05E165dsXDhQvTt2xdr167FgQMHpKkOZDIZJk2ahLfffhtNmjRBSEgI3njjDQQGBiIuLs48H7aKCu/UQLGInIiIyPxs5ttXpVJh586dGD9+PCIiIuDt7Y2EhASMHj1aatOhQwesWbMGs2fPxmuvvYYmTZpg8+bNaNmypdRm+vTpyM/Px+jRo5Gbm4tOnTph+/btUCqVlvhY98RLeERERJYjE0IIS3fC1mg0GqhUKqjVari7u5vkPRK3ncKnv5/Hi51CMPvxMJO8BxER0cPEmO9vq5jGgCoq4F14REREFsMAZaX+voRnM1dhiYiIrAYDlJUqLCkvIucIFBERkbkxQFkpFpETERFZDgOUlWINFBERkeUwQFkpLuVCRERkOQxQVurvxYRZRE5ERGRuDFBWiiNQRERElsMAZaW4Fh4REZHlMEBZKd6FR0REZDkMUFZIpxfQluoBcDFhIiIiS2CAskLlBeQAL+ERERFZAgOUFSovIJfJAIUDTyEREZG58dvXCkmTaDraQyaTWbg3REREDx8GKCvEhYSJiIgsiwHKCnEhYSIiIstigLJCXAePiIjIshigrBDngCIiIrIsBigrxGVciIiILIsBygpJI1BcSJiIiMgiGKCsUPlEmhyBIiIisgwGKCvES3hERESWxQBlhQpKWERORERkSQxQVogjUERERJbFAGWF/q6BYhE5ERGRJTBAWaHCEj0AwMmRI1BERESWwABlhQp5Fx4REZFFMUBZIc5ETkREZFkMUFbo74k0GaCIiIgsgQHKCv19Fx6LyImIiCyBAcoKFZSU1UDxEh4REZFlMEBZIc4DRUREZFkMUFaogAGKiIjIohigrIwQAoVcyoWIiMiiGKCsTFGJHkKU/c4iciIiIstggLIy5cu4AJzGgIiIyFIYoKxMef2TwsEO9nYyC/eGiIjo4cQAZWXK659YQE5ERGQ5DFBWpoCTaBIREVkcA5SVKa+B4h14RERElsMAZWU4iSYREZHlMUBZGS4kTEREZHkMUFaGI1BERESWxwBlZcproFhETkREZDkMUFamgMu4EBERWRwDlJXhJTwiIiLLY4CyMlIROQMUERGRxTBAWRlpIk1H1kARERFZCgOUlSmUisg5AkVERGQpDFBWhpfwiIiILI8ByspwMWEiIiLLY4CyMgW8C4+IiMjiGKCszN+X8FhETkREZCkMUFaGReRERESWxwBlZbiYMBERkeUxQFkZzkRORERkeVYToBo0aACZTGbwePfddw3aHD16FJ07d4ZSqURwcDDmz59f4TgbNmxAs2bNoFQq0apVK2zbts1gvxACCQkJCAgIgJOTE2JiYnDmzBmTfraqEkJIa+FxMWEiIiLLsZoABQBvvvkmMjIypMfLL78s7dNoNOjVqxfq16+PgwcPYsGCBZg7dy5WrFghtdm7dy+GDBmCkSNH4vDhw4iLi0NcXByOHz8utZk/fz6WLFmC5cuXIyUlBS4uLoiNjUVRUZFZP2tltDo9dHoBgPNAERERWZJVBSg3Nzf4+/tLDxcXF2nf6tWrodVq8cUXX6BFixYYPHgwXnnlFXzwwQdSm8WLF6N3796YNm0amjdvjrfeegtt27bF0qVLAZSN8CxatAizZ89Gv3790Lp1a6xatQrp6enYvHmzuT9uBeWX7wBewiMiIrIkqwpQ7777LurUqYNHHnkECxYsQGlpqbQvOTkZXbp0gVwul7bFxsYiLS0NOTk5UpuYmBiDY8bGxiI5ORkAcOHCBWRmZhq0UalUiIqKktpYUnkBuaO9DI72VnXqiIiIbIrVFNK88soraNu2Lby8vLB3717MmjULGRkZ0ghTZmYmQkJCDF7j5+cn7fP09ERmZqa07e42mZmZUru7X1dZm8oUFxejuLhYeq7RaKr5Ke+Pd+ARERHVDhYdxpg5c2aFwvB/Pk6fPg0AmDJlCrp164bWrVtjzJgxWLhwIT766COD4GIpiYmJUKlU0iM4ONgk7/P3HXhWk3uJiIhskkW/iadOnYrhw4fft03Dhg0r3R4VFYXS0lJcvHgRoaGh8Pf3R1ZWlkGb8uf+/v7Sz8ra3L2/fFtAQIBBm/Dw8Hv2cdasWZgyZYr0XKPRmCREFXASTSIiolrBogHKx8cHPj4+1Xptamoq7Ozs4OvrCwCIjo7G66+/jpKSEjg6OgIAkpKSEBoaCk9PT6nNrl27MGnSJOk4SUlJiI6OBgCEhITA398fu3btkgKTRqNBSkoKxo4de8++KBQKKBSKan0OY5RPYcA78IiIiCzLKiqRk5OTsWjRIhw5cgTnz5/H6tWrMXnyZDz77LNSOBo6dCjkcjlGjhyJEydOYN26dVi8eLHByNDEiROxfft2LFy4EKdPn8bcuXNx4MABTJgwAQAgk8kwadIkvP3229iyZQuOHTuG559/HoGBgYiLi7PERzfASTSJiIhqB6soplEoFFi7di3mzp2L4uJihISEYPLkyQbhSKVSYefOnRg/fjwiIiLg7e2NhIQEjB49WmrToUMHrFmzBrNnz8Zrr72GJk2aYPPmzWjZsqXUZvr06cjPz8fo0aORm5uLTp06Yfv27VAqlWb9zJUp5ELCREREtYJMCCEs3Qlbo9FooFKpoFar4e7uXmPH/XrfJbyx+ThiW/jh0+cia+y4REREZNz3t1VcwqMyhVIROUegiIiILIkByooIASgd7VgDRUREZGG8hGcCprqEV04IAZlMVuPHJSIiepjxEp6NY3giIiKyLAYoIiIiIiMxQBEREREZiQGKiIiIyEgMUERERERGYoAiIiIiMhIDFBEREZGRGKCIiIiIjMQARURERGQkBigiIiIiIzFAERERERmJAYqIiIjISAxQREREREZigCIiIiIykoOlO2CLhBAAAI1GY+GeEBERUVWVf2+Xf4/fDwOUCeTl5QEAgoODLdwTIiIiMlZeXh5UKtV928hEVWIWGUWv1yM9PR1ubm6QyWQ1emyNRoPg4GBcuXIF7u7uNXpsMh2eN+vE82adeN6sT205Z0II5OXlITAwEHZ2969y4giUCdjZ2SEoKMik7+Hu7s7/MFghnjfrxPNmnXjerE9tOGf/NvJUjkXkREREREZigCIiIiIyEgOUlVEoFJgzZw4UCoWlu0JG4HmzTjxv1onnzfpY4zljETkRERGRkTgCRURERGQkBigiIiIiIzFAERERERmJAYqIiIjISAxQVuTjjz9GgwYNoFQqERUVhT///NPSXaK7/P7773jiiScQGBgImUyGzZs3G+wXQiAhIQEBAQFwcnJCTEwMzpw5Y5nOkiQxMRGPPvoo3Nzc4Ovri7i4OKSlpRm0KSoqwvjx41GnTh24urqif//+yMrKslCPCQCWLVuG1q1bSxMvRkdH46effpL285zVfu+++y5kMhkmTZokbbOm88YAZSXWrVuHKVOmYM6cOTh06BDatGmD2NhYZGdnW7prdEd+fj7atGmDjz/+uNL98+fPx5IlS7B8+XKkpKTAxcUFsbGxKCoqMnNP6W67d+/G+PHjsW/fPiQlJaGkpAS9evVCfn6+1Gby5Mn473//iw0bNmD37t1IT0/H008/bcFeU1BQEN59910cPHgQBw4cQI8ePdCvXz+cOHECAM9Zbbd//358+umnaN26tcF2qzpvgqxCu3btxPjx46XnOp1OBAYGisTERAv2iu4FgNi0aZP0XK/XC39/f7FgwQJpW25urlAoFOLbb7+1QA/pXrKzswUAsXv3biFE2XlydHQUGzZskNqcOnVKABDJycmW6iZVwtPTU3z++ec8Z7VcXl6eaNKkiUhKShJdu3YVEydOFEJY379rHIGyAlqtFgcPHkRMTIy0zc7ODjExMUhOTrZgz6iqLly4gMzMTINzqFKpEBUVxXNYy6jVagCAl5cXAODgwYMoKSkxOHfNmjVDvXr1eO5qCZ1Oh7Vr1yI/Px/R0dE8Z7Xc+PHj0bdvX4PzA1jfv2tcTNgK3LhxAzqdDn5+fgbb/fz8cPr0aQv1ioyRmZkJAJWew/J9ZHl6vR6TJk1Cx44d0bJlSwBl504ul8PDw8OgLc+d5R07dgzR0dEoKiqCq6srNm3ahLCwMKSmpvKc1VJr167FoUOHsH///gr7rO3fNQYoIqI7xo8fj+PHj2PPnj2W7gpVQWhoKFJTU6FWq7Fx40bEx8dj9+7dlu4W3cOVK1cwceJEJCUlQalUWro7D4yX8KyAt7c37O3tK9yJkJWVBX9/fwv1ioxRfp54DmuvCRMmYOvWrfj1118RFBQkbff394dWq0Vubq5Be547y5PL5WjcuDEiIiKQmJiINm3aYPHixTxntdTBgweRnZ2Ntm3bwsHBAQ4ODti9ezeWLFkCBwcH+Pn5WdV5Y4CyAnK5HBEREdi1a5e0Ta/XY9euXYiOjrZgz6iqQkJC4O/vb3AONRoNUlJSeA4tTAiBCRMmYNOmTfjll18QEhJisD8iIgKOjo4G5y4tLQ2XL1/muatl9Ho9iouLec5qqZ49e+LYsWNITU2VHpGRkRg2bJj0uzWdN17CsxJTpkxBfHw8IiMj0a5dOyxatAj5+fkYMWKEpbtGd9y+fRtnz56Vnl+4cAGpqanw8vJCvXr1MGnSJLz99tto0qQJQkJC8MYbbyAwMBBxcXGW6zRh/PjxWLNmDX744Qe4ublJtRYqlQpOTk5QqVQYOXIkpkyZAi8vL7i7u+Pll19GdHQ02rdvb+HeP7xmzZqFPn36oF69esjLy8OaNWvw22+/YceOHTxntZSbm5tUW1jOxcUFderUkbZb1Xmz9G2AVHUfffSRqFevnpDL5aJdu3Zi3759lu4S3eXXX38VACo84uPjhRBlUxm88cYbws/PTygUCtGzZ0+RlpZm2U5TpecMgPjyyy+lNoWFhWLcuHHC09NTODs7i6eeekpkZGRYrtMkXnjhBVG/fn0hl8uFj4+P6Nmzp9i5c6e0n+fMOtw9jYEQ1nXeZEIIYaHsRkRERGSVWANFREREZCQGKCIiIiIjMUARERERGYkBioiIiMhIDFBERERERmKAIiIiIjISAxQRERGRkRigiOihdvHiRchkMqSmpprsPYYPH84Z54lsDAMUEVm14cOHQyaTVXj07t27Sq8PDg5GRkZGhSUmiIjuh2vhEZHV6927N7788kuDbQqFokqvtbe3r5UrvRNR7cYRKCKyegqFAv7+/gYPT09PAIBMJsOyZcvQp08fODk5oWHDhti4caP02n9ewsvJycGwYcPg4+MDJycnNGnSxCCcHTt2DD169ICTkxPq1KmD0aNH4/bt29J+nU6HKVOmwMPDA3Xq1MH06dPxzxWz9Ho9EhMTERISAicnJ7Rp08agT//WByKyPAYoIrJ5b7zxBvr3748jR45g2LBhGDx4ME6dOnXPtidPnsRPP/2EU6dOYdmyZfD29gYA5OfnIzY2Fp6enti/fz82bNiAn3/+GRMmTJBev3DhQqxcuRJffPEF9uzZg1u3bmHTpk0G75GYmIhVq1Zh+fLlOHHiBCZPnoxnn30Wu3fv/tc+EFEtYeHFjImIHkh8fLywt7cXLi4uBo933nlHCCEEADFmzBiD10RFRYmxY8cKIYS4cOGCACAOHz4shBDiiSeeECNGjKj0vVasWCE8PT3F7du3pW0//vijsLOzE5mZmUIIIQICAsT8+fOl/SUlJSIoKEj069dPCCFEUVGRcHZ2Fnv37jU49siRI8WQIUP+tQ9EVDuwBoqIrF737t2xbNkyg21eXl7S79HR0Qb7oqOj73nX3dixY9G/f38cOnQIvXr1QlxcHDp06AAAOHXqFNq0aQMXFxepfceOHaHX65GWlgalUomMjAxERUVJ+x0cHBAZGSldxjt79iwKCgrw2GOPGbyvVqvFI4888q99IKLagQGKiKyei4sLGjduXCPH6tOnDy5duoRt27YhKSkJPXv2xPjx4/H+++/XyPHL66V+/PFH1K1b12BfeeG7qftARA+ONVBEZPP27dtX4Xnz5s3v2d7Hxwfx8fH45ptvsGjRIqxYsQIA0Lx5cxw5cgT5+flS2z/++AN2dnYIDQ2FSqVCQEAAUlJSpP2lpaU4ePCg9DwsLAwKhQKXL19G48aNDR7BwcH/2gciqh04AkVEVq+4uBiZmZkG2xwcHKTC6w0bNiAyMhKdOnXC6tWr8eeff+L//u//Kj1WQkICIiIi0KJFCxQXF2Pr1q1S2Bo2bBjmzJmD+Ph4zJ07F9evX8fLL7+M5557Dn5+fgCAiRMn4t1330WTJk3QrFkzfPDBB8jNzZWO7+bmhldffRWTJ0+GXq9Hp06doFar8ccff8Dd3R3x8fH37QMR1Q4MUERk9bZv346AgACDbaGhoTh9+jQAYN68eVi7di3GjRuHgIAAfPvttwgLC6v0WHK5HLNmzcLFixfh5OSEzp07Y+3atQAAZ2dn7NixAxMnTsSjjz4KZ2dn9O/fHx988IH0+qlTpyIjIwPx8fGws7PDCy+8gKeeegpqtVpq89Zbb8HHxweJiYk4f/48PDw80LZtW7z22mv/2gciqh1kQvxjghIiIhsik8mwadMmLqVCRDWKNVBERERERmKAIiIiIjISa6CIyKaxSoGITIEjUERERERGYoAiIiIiMhIDFBEREZGRGKCIiIiIjMQARURERGQkBigiIiIiIzFAERERERmJAYqIiIjISAxQREREREb6/8lP2IhBJwZHAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(episodes), returns)\n",
    "plt.title(\"Returns Over Episodes\")\n",
    "plt.xlabel(\"Episodes\")\n",
    "plt.ylabel(\"Returns\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Episode Videos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/sarsa_fa//MountainCar-v0-episode-0.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-0.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episide 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/sarsa_fa//MountainCar-v0-episode-10.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-10.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/sarsa_fa//MountainCar-v0-episode-20.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-20.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/sarsa_fa//MountainCar-v0-episode-30.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-30.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Episode 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"./videos/sarsa_fa//MountainCar-v0-episode-40.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(f\"{VIDEOS_ROOT_DIR}/{ENV_TYPE}-episode-40.mp4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Greedily On An Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Reward: -200.0, Steps Taken: 200\n"
     ]
    }
   ],
   "source": [
    "# Run the algorithm in inferencing mode and \n",
    "# observe its performance\n",
    "env = gym.make(ENV_TYPE, render_mode=\"human\")\n",
    "state, info = env.reset(seed=100)\n",
    "G = 0\n",
    "num_steps = 0\n",
    "terminated = False\n",
    "truncated = False\n",
    "\n",
    "while not terminated and not truncated:\n",
    "    estimated_returns = predict_returns(state, Q)\n",
    "    action = np.argmax(estimated_returns) \n",
    "    state, reward, terminated, truncated, info = env.step(action)\n",
    "    G += reward\n",
    "    num_steps += 1\n",
    "\n",
    "print(f'Total Reward: {G}, Steps Taken: {num_steps}')\n",
    "\n",
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "flax",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0a8d1f2580cdfde5c5829808ec6fccc81a351d243fb8b1925f7928e44ccf575b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}