RL in Large Discrete Action Spaces - Wolpertinger Agent (#394)

* Currently this is specific to the case of discretizing a continuous action space. Can easily be adapted to other case by feeding the kNN otherwise, and removing the usage of a discretizing output action filter

docs_raw/source/algorithms.xml

@@ -1 +1 @@
-<mxfile modified="2019-06-13T11:04:47.252Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="OGV5teY4xcR0Xj7nvQBA" version="10.7.7" type="device"><diagram id="Fja6IZyvrddIfr-74nt7" name="Page-1">7V1bk5s2FP41O9M+JIMQ18e9NduZbCdp0jR5ysgg26SAXMC73v76CnMVEphdkLGzbB5sDhJgne/7dCQdkQt4HezeRWizvicu9i9Uxd1dwJsLVQVAt+hHannKLIZuZ4ZV5Ll5ocrwyfsP50Ylt249F8dMwYQQP/E2rNEhYYidhLGhKCKPbLEl8dm7btAKc4ZPDvJ569+em6wzq6Urlf0Oe6t1cWeg5GcCVBTODfEaueSxZoK3F/A6IiTJvgW7a+ynjVe0S1bvt5az5YNFOEz6VPh+fW+gL8GXH+p7HP5156+j3dc3mp4/XPJU/GLs0gbID0mUrMmKhMi/raxXEdmGLk4vq9Cjqsx7QjbUCKjxB06Sp9ybaJsQalongZ+fxTsv+ZpWf6vnR99qZ252+ZX3B0/FQZhET7VK6eG3+rmq2v6oqBcnEfmn9F16j+wXpz+ztSVzU0y2kYM7mi8ttYckilY46SgItdLjlCqYBJg+I60YYR8l3gP7JCjH7Kosl1e9jCL0VCuwIV6YxLUrf0gNtEBOP6jl2MvJpxsNiDTLm1ZXefole4LiqPZTKtMeds+AYP6jH5C/zZuBg2QFuNR/j2svwZ82aO+ZRyo7LLiWnu9fE59E+7pwqaf/SiDUzhj7v7QGCZOaPftLaxSMTXGEIifHs8qhSi1R9YCjBO+6ccWjoKigs+4ChXQ8Vsqj5i5R1jXVgYXoiJBT894LnKNwzpj14Rn60FceBqrDIB+rHAHJIsbRA701CYdxkWEQT0DHwoulkE0NErsIW0uHYSEYiXRU9N7qDO0MnnW2zpNOlcY5wDX6zLn+nIM9OTe0Rx7kYyjo9AyfPu7Vgn5ZJfsmyQxp78Sgwfh3S4oTb+K9Py9pAaBvdtXJ4iq/h5ttWv02WGDXxVFxVfrU2YXZm1Fz7QHkUd9SF3Df9x6mvo4tV2Oob4zEfMVqMB9AnvpAhzz3LVnc146Fi3vPdX38SBtrRkQNETaDBxoBC/Cg8njQZOFBHzk4PgMfADYILrnG+EAQBJfkHd0JhgRSQhEpb24+vKMnL52EtrOq3GHkDmEndULScD/j8ZCEuOHa3IR8bxXSQ4c6jfYZ8Cp1qecg/zI/EezlowVvbCzCQG4EgABDacg2FNBUEyBEXsSm/ewR24jxmdkzPrOmjM/MVsr3ZbgtInirbiyiUgNwGGMxvZtF/9gG9EZkm9AQL87uGm6D78hJB24xX/6AltQLyutjXNNeKPwgQNDHLJfYcCQN+lRWQVRL0NGropkWXZaEzBOxQ0TF6ikq9pSiYo0fRwhV5jMK754VN7w+/utqM4aA8C2vAYZo3gfKkgB7loABEmD3lAAwaWBhcxrwlfe673ubuC26rvkOxZtszXPp7VIMXLE8HZ2IYzDPZId30OSHd1C4xCGLdXBm3QDWaT1ZZ05JOmBO6WNQ83Dl70M+ZjxcOXwKHwPjHKIr0DlNcziUskSh1GXbWOpPFK56DtM47L2aeRlz4nkZYxjvwSvXdtB3VJXNSh87u6UctOdw06wD2S2ws7yk7BZ+0JfNz1BbgNFrXF/XtIkX2IsLzylHKrDYcFyzBSlHotUWaEjzjtraj/deXVFEfblDQgclrRMjLx2EncGYy26k9QHeyaZglVsaAeHYOX9nsKxpaKwPBLGRcGkZyPOCiGg/uRcaCbGmIMMSiDIsAZCW8WHNMeqQbK++6V5Z8DdZvld7wtcJLSjOC4Rt05S6LVgcMARhkaz1QW0eyg6Sib7zlNCYVCba8/9mmThBmYAaO7SdXCaKCHKWiZfJhN5TJrRJp7qhKC10lomTlYlmNGEIZsCOKxPDtpi8epnouyCmK2JcHEkm2lfEZpk4QZloRhOTy4Q2RxPH2R0+dP/ni9bPnr07XIFd5eWsn2nGDMH5BQWTvqBAH7a7YxoIqqeDQa1vZqY+iQyqjSVQ07QPyGBneUkYPJtVgiNAaShCDkLAMhurPRl081qNoGqMTo7PCp4kUBcG5WXlkN9uBOZwvmux0WSFQrC7/Lij/mF7jKbpyMzT6ciKPJgT7chgMatTdkwHginV7CovqSMT5V7JkLrxZiAGPVeXaM4SCUEzOXB6kTybASevdSewu1vvm49hqFJCOQ2wotY3lHu22AJ2MsTSDoitAbrKSxJbCa8ZEqrcvRceELXXvuHUMliZsy3BdtNipuEo20314YMOAERg+DjOZYqrxNsSOPWes2amnyhIMREu4vQj+0UH8diZ/vvTpeFr7MyZJdh3KdztLG1vjioJfoEX/jIBBGkrKB8FFVRxhV9nfLKRoAobkaAIoeZREdqeZVG68fMNlWfly0X6EM1OrfH6pkWrL/n9gCM0KLAbgXUxyVTfjCdI+i/W/MZvzh6v0RK+le7YDWexEaQhyBHXRB21tA0xZ7P+cJIjkhzPR9rEKNh12Nhh1Xx35EgDkuZuyOJF9G0DEq3xXM8tf+TdlvSwelV+Vrz6Dwfg7f8=</diagram></mxfile>
+<mxfile modified="2019-09-04T08:15:58.568Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="KVJnAPzF4j_VN_ncw2yr" version="11.2.4" type="device" pages="2"><diagram id="Fja6IZyvrddIfr-74nt7" name="DDPG">7V1bk5s2FP41O9M+ZAdJXB/31t3OZDtJk6bJUwaDbJMCcgHvevvrK4ywEYiL18jgLJsHm4MEWOf7Ph1JR+QC3QSb+8heLR+Ji/0LqLibC3R7ASFQoU4/UstLZjGgmhkWkeeyQnvDJ+8/zIwKs649F8dcwYQQP/FWvNEhYYidhLPZUUSe+WJz4vN3XdkLXDF8cmy/av3bc5NlZjU1ZW9/wN5imd8ZKOxMYOeFmSFe2i55LpjQ3QW6iQhJsm/B5gb7aePl7ZLV+63m7O7BIhwmXSp8v3nU7S/Blx/wPQ7/evCX0ebrO1VjD5e85L8Yu7QB2CGJkiVZkND27/bW64isQxenl1Xo0b7Me0JW1Aio8QdOkhfmTXudEGpaJoHPzuKNl3xNq19q7Ohb4czthl15e/CSH4RJ9FKolB5+K57bV9se5fXiJCL/7HyX3qPacKwtY7KOHNzQWmmpLQLtaIGThoKIgTxty8ItmGPuMQkwfUZaIMK+nXhPPNhshtnFrhyrehVF9kuhwIp4YRIXrvwhNdACjH5IZdhj5NP0EkTK5Q2zqTz9kj1BflT4KXvTFnYHQJD96CfbX7NmqEByD7jUf89LL8GfVvbWVc9UdnhwzT3fvyE+ibZ10VxL/+2AUDijb//SGiRMCvbsL62RMzbFkR05DM+wgirYhKonHCV404iC/KzGuwvk0vG8Vx7IXKIsC6qDctERIafgvVc4R6k4Y9KHen3oKg99q8NRPoYVApJZjKMnemsSHsdFjkFVAjomns2FbCqR2LWxOXc4FoKeSEdF71LjaKdXWWdpVdJBaZwDlUafONfGpVbO9d4jH+VjJOj0dJ8+//WMflkk2ybJDGnvxKFB/3dN8hPv4q0/r2gBoK02+5P5VX4PV+u0+l0ww66Lo/yq9KmzC/M3o+bCA8ijvglnaNv3tlNfw6arctTXe2K+YpaYD1CV+kBDVe6bsrivngoXj57r+viZtt+EiAIiLA4PNAIW4AFW8aDKwoPWc3B8Bj4AfBC84xrnA0EQvCNv707QJZASiUh5e/vhnp68chLazlB5wLZ7DDtpiycl93MeD0mIS65lJtv3FiE9dKgfaZ+BrlP/eY7tX7ETwVY+avDGxyIc5HoACNCVkmwjAU1VAULkRWzqzx6xvT4+MzrGZ+ao4jOjlvJdGW6JCF6rG7NopwE4jLGY3uWif6wDeiOyTmiIF2d3DdfBd9tJB25xtXyLlhQLyutjXMOaKdVBgKCPmc+x7kga9EFeQaAp6OihaKZFkyUh00TsAaJidhQVa1SiYvYfRwhV5rMdPhwUN7w9/muwHEMgdFnVAF0074NkSYA1SUB3CbA6SgAYV2BhVTTga9Xrvu+t4rrouuA7O15la55zb5Ni4Jrnae9E7IN5Bj+8Q0Z1eIeESxyyWIcm1nVnndqRdcaoSAeMIX0MCh7e+7vNx5yH9w4/gY+BfpbRFWicpmkPpUxRKHVVN5b60w4XHYdpFey9mXkZY+B5Gf043oO3pe2g66gqn4QeOLtlN2hncFPNluwW1FheUnZLddCXzc9QW4Dtt7i+rqoDL7DnF55SjiAw+XBctQQpR6LVFqRL8w6s7cc7r64oor7cIaFjJ7UTI68dhJ3BmMsqpfWBqpMNwSq3NAKivnP+zmBZU1d5HwhiI+HSMpDnBRHRfnIvlBJiDUGGJRBlWAIgLePDnGLUA7K9uqZ75bHeSAanqD7ha0QLitMCYd00pWYJFgd0QVgka31QnYayh8hE13lKpI9LJurz/yaZGKFMIJUf2g4uE3kEOclEJ5nQOsqEOq6pbiRKC51kYrQyUY4mdMEM2Gll4rgtJm9NJrouiOUTZWORifoVsUkmRigT5WhicJlQp2jiAJnIp8/aBx297/981frZwbvDFdRUXs76mapPEJxeUDDoCwq043Z3DANBOFyKWNfMTG0cMghLS6CGYbXIYGN5SRg8m1UCGVDqGSGtEDCN0mpPhmVWqxRU9dHJVbOCBwnUhUH5rnJY3W4EpnC+abHR4IVCsLv8tKP+4/YYDdORGYN1ZPlo/lw6MpTP6uw6ppZgChpN5SV1ZKLcKxlS198MxFHP1SSak0QiUE4OHF4kz2bAWdW60+/u1rrmY+jwNKGcCnhR6xrKHSy2gJ8MMdUWsdVBU3lJYivhNUNClXv0whZRe+sbTk2dlznLFGw3zWcaTrLdVDt+0AGACAwf+7lMfpV4vQNOsecsmOmnHaSYCGdx+pH9olY8Nqb//nRp+Co/c2YK9l0KdztL25sDJcEv8MJfBoAgbQXlo6ACFFf4dcInHwlCVIoERQg1TorQ+iyLnRs/31J5Vr5cpA9R7tRKr2+a1fqyuh+whwYFVimwzieZipvxBEn/+Zpf/83Z4TVawrfSnbrhTD6C1AU54qqoo5a2IeZs1h/GMCJh8B1qE6Ng12Fph1X53ZE9DUjKuyHzF9HXDUjU0nMdWv7Euy3p4f5V+Vnx/X84gO7+Bw==</diagram><diagram id="CBe-Ox6p2qOorUqqi51D" name="Wolpertinger">7Vxbt5o4FP415/F0kQQCPOq5tJ1e1plpV6edl1kRojIHiYNYtb++QYkQEi+nEPGcqg/KzgXI/vaXnZ0NV+hmsnydkun4AwtpfAWtcHmFbq8gBDbE/CeXrDYSF9obwSiNwqJSKfgU/aCF0Cqk8yikM6lixlicRVNZGLAkoUEmyUiasoVcbchi+axTMqKK4FNAYlX6dxRm443Uc6xS/oZGo7E4M7CKkgkRlQvBbExCtqiI0N0VukkZyzb/JssbGueDJ8Ylip0vDPcH797f/fEwsd8FD9eL601n909psr2FlCbZL3f9o2e/S1b35Os/N29Xt70VfvsQi66/k3hejFdxr9lKDGDK5klI807AFeovxlFGP01JkJcuOGS4bJxN4qJ4GMXxDYtZum6Lhk7+5fJZlrJHWinB60/egiVZRb755C3EaFv8gKRBgSy47UxoM5cUd0HTjC5raj8wZmCrSG4BlE1olq54O9GLgEkBfiCwsCihBHEhG1dghASKSAHf0bbvUkX8T6GlJ2hMVRANOeCLQ5ZmYzZiCYnvSmm/VGE+mGWd94xNC8X9R7NsVYwxmWdMVitdRtnXvPkrpzj6Vim5XRY9rw9W4iDht1tplB9+q5aVzdZHop2sXbDVbn6bv6BbPlRsngZ0z5CigpZIOqLZnnqOHispjUkWfZcvrnW9I8VS2WBG0+/81CxpZrSSqamWGnh0MNSaXc3aQ0K9YSCZKzBpnRC5rxzJPrFqnr6jWic0ZZz2xTjbNk7nSOPEXRqno5lGccwvtz/gf0bZepw2gny+kyCC/58zUXA9Wyu5xysAZ7osC0Uvb5PpPG9+NxnQMKSp6JVf9aZj+WRcXLkAcxzhwQFaz+aHOcKhXmhLHIFNUoTl1SgCIJUjgINUkvBMkQQ+FVg+RGEY0wUfwQtMDsHEl0ACoaMBCVRBYpsCiduyY/5cFQNkB3xrlZJiNA741sxb14xnwHyRznxvbx9e88JekPHBh9YbSsImdsw1k9UwIcEgYQmt6bsQkTgaJfww4JrkUw7q53qO+BK7VxRM1kSzA4SyfyPh0BRqALZqrI80Bm1rYGPMM/Rfumdo2g8URHDQEQSwS09QXKaGHo5lA19HBjs5ZpBu+YImM6qngnrVj/MJPxGbZ9ybnG3Omswn/5IgX0zO1PoHeKda0dwkFbr+wFIXIZpJajikODjlQhTKbAN9zSwFtWEiUzAEiiZeGN+cfiUK4LEMZHfKQLB9B0VLSZ9J8uZJDsmFLNbk4MC6c4LQK9U9wbrAFTIFGnThi9b5wj6WL/xO+cJW+OKrCoY4jqazXS5+RaVkNt3s4g2jZQ6NvmzTrRutMSt15YUnctUpHZ10RncuFtq2heJjDdTSY+VEBoq71Dyo6L1EwSHNS3ovYXA2mgfu8/DldGHArS932G/zdH5bb9cq7y+SjI5cQCqI/L2jS27H0SXcLLy0ZoP5ZCrqF5o5+/mCpFkvTxEqL3ktu4/y0VvXqWWX+L0bt39/Cn7xj+QX22vIL+umfBTIqlJhyqIkm1V6fsgFFe/GkwFse7XcoVp94FuN6mNrf/369dTq8z+bOywtZDtUDbZYmq15ujWaFwF+3K1f5SuT6zRlXDn8XMFvnVUj27rdcVIN1IXUm2jmhSUpQqvuj2AANAElBHQuiXBd2s+JcZ8xvZr0SUzT7zbN+mA+Y6c5U6LjBjtlwNKtbwKWBCTbGZn+1cjWMw1kIVAPN7sW1rCDq0mBMkfpajarycn2fNNbalFGz3I0qgEYqLrBxnj7spbshreRdyRv2ztAdaJE9N0JUO1mODwhn+HANmSj60rUXAlwyYfYT2ueX5tzPKjzSPU5EaaIzW7B4QA6hPzZTjf8l0xyBCSDWf6zuc6DG+x7vZvfYsWKgLxi9Xxft5+OT+jg2DoHp8ZvX66UpM6nZG8aM15QX0564pHPanjbVkfThoZG02229/lyXZKuInz2sa4KbOqqtBLexv7+8DOy/X31zYSfbROJ5JeHhtpgQNGPgAP0Na6LpxKgY2o6cZs9WHghwNYJ8Ngtjm4IcPvWhrMmQHX/5UKAZ0GACIP6jgJ0teu3U5Kg03y/9kSMp+HaswgwHclZdtN81FacNnAgZ8CGe+sbyhm4bGqd10TsHJvDiZom2Line truncated

docs_raw/source/components/agents/index.rst

@@ -21,8 +21,6 @@ A detailed description of those algorithms can be found by navigating to each of
    imitation/cil
    policy_optimization/cppo
    policy_optimization/ddpg
-   policy_optimization/td3
-   policy_optimization/sac
    other/dfp
    value_optimization/double_dqn
    value_optimization/dqn
@@ -36,6 +34,10 @@ A detailed description of those algorithms can be found by navigating to each of
    policy_optimization/ppo
    value_optimization/rainbow
    value_optimization/qr_dqn
+   policy_optimization/sac
+   policy_optimization/td3
+   policy_optimization/wolpertinger
+
 
 
 .. autoclass:: rl_coach.base_parameters.AgentParameters

docs_raw/source/components/agents/policy_optimization/wolpertinger.rst

@@ -0,0 +1,56 @@
+Wolpertinger
+=============
+
+**Actions space:** Discrete
+
+**References:** `Deep Reinforcement Learning in Large Discrete Action Spaces <https://arxiv.org/abs/1512.07679>`_
+
+Network Structure
+-----------------
+
+.. image:: /_static/img/design_imgs/wolpertinger.png
+   :align: center
+
+Algorithm Description
+---------------------
+Choosing an action
+++++++++++++++++++
+
+Pass the current states through the actor network, and get a proto action :math:`\mu`.
+While in training phase, use a continuous exploration policy, such as the a gaussian noise,
+to add exploration noise to the proto action. Then, pass the proto action to a k-NN tree to find actual valid
+action candidates, which are in the surrounding neighborhood of the proto action. Those actions are then passed to the
+critic to evaluate their goodness, and eventually the discrete index of the action with the highest Q value is chosen.
+When testing, the same flow is used, but no exploration noise is added.
+
+Training the network
+++++++++++++++++++++
+
+Training the network is exactly the same as in DDPG. Unlike when choosing the action, the proto action is not passed
+through the k-NN tree. It is being passed directly to the critic.
+
+Start by sampling a batch of transitions from the experience replay.
+
+* To train the **critic network**, use the following targets:
+
+  :math:`y_t=r(s_t,a_t )+\gamma \cdot Q(s_{t+1},\mu(s_{t+1} ))`
+
+  First run the actor target network, using the next states as the inputs, and get :math:`\mu (s_{t+1} )`.
+  Next, run the critic target network using the next states and :math:`\mu (s_{t+1} )`, and use the output to
+  calculate :math:`y_t` according to the equation above. To train the network, use the current states and actions
+  as the inputs, and :math:`y_t` as the targets.
+
+* To train the **actor network**, use the following equation:
+
+  :math:`\nabla_{\theta^\mu } J \approx E_{s_t \tilde{} \rho^\beta } [\nabla_a Q(s,a)|_{s=s_t,a=\mu (s_t ) } \cdot \nabla_{\theta^\mu} \mu(s)|_{s=s_t} ]`
+
+  Use the actor's online network to get the action mean values using the current states as the inputs.
+  Then, use the critic online network in order to get the gradients of the critic output with respect to the
+  action mean values :math:`\nabla _a Q(s,a)|_{s=s_t,a=\mu(s_t ) }`.
+  Using the chain rule, calculate the gradients of the actor's output, with respect to the actor weights,
+  given :math:`\nabla_a Q(s,a)`. Finally, apply those gradients to the actor network.
+
+After every training step, do a soft update of the critic and actor target networks' weights from the online networks.
+
+
+.. autoclass:: rl_coach.agents.wolpertinger_agent.WolpertingerAlgorithmParameters

docs_raw/source/diagrams.xml

@@ -1 +1 @@
-<mxfile modified="2019-06-12T14:38:13.932Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="KqQkpaiTzICBBUdtGW-V" version="10.7.7" type="device"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1bd6M6sv41eUwWQlwfc+l07zPdPdm7Z87MnJeziE0cZmPwYJxL//qRMMKgkgHbEoa0MrN627K5mK+qVPWpqnSBb5dvn7Ng9fwtnYfxhWnM3y7w3YVpIux75D905H074tj+dmCRRfPyS7uBH9HPsBw0ytFNNA/XjS/maRrn0ao5OEuTJJzljbEgy9LX5tee0rh51VWwCMHAj1kQw9F/RPP8eTvq2cZu/EsYLZ7ZlZFRfvIYzP5cZOkmKa93YeKn4m/78TJg5yq/v34O5ulrbQh/usC3WZrm21fLt9swps+WPbbtcfd7Pq3uOwuTvNcBGG8PeQniTcjuubiz/J09jeL3hPQIdIFvXp+jPPyxCmb001eCPxl7zpdx+fEiDtb06Rvk9SxdRrPy9TrP0j/D2zROs+Ks2Jl54eNT9Ql7zpiMPEVxXPvmPAi9pxkdT5O8FBbTKN/XvmcUf2Q8iKNFQsbi8Cmnb7NZeZRD3sFHVD61lzDLw7faUPnIPofpMsyzd/IV9qnllfi9M7llEv+6ExeLCcVzTVSwUz7foBTRRXX2HUzkRYmUGDXHUwjaPFg/V8e1IvjJof/rg2CpA+RCWTCPwh1qSZqEMoB15QBL8GngahqOAXG1BLg6zDacgqtvaFxzSUi6TQ2lNhogiUVIIksGkgggGZB5ahjL6rt3hutKxg+ZLfjJAMy3rmwOMse/8vDuzwYAVhpTB1AKfma3JobJ/Jr6GuTdjEJDAanjVc3t9AnFwWMY31TeQe0h3hT/E6PV8/GTp569/5NeiTzC8u2/9s67hnt/fdMGWh5kizBvCnI4b3hMEMYaKrYAFDaWhXGQRy9NP0uEVHmFhzQid1dJSWV7mYxULiM7xzrdZLOwPKzu+YAz2by8YYM71/ZBgHMR1IP32tdW9AvrQ24aG633ZvvuiQcgu/0AcEvcAeTF9lfutKXCupcCuT0USE9lPacyzinBfacykxfoY0whUulsDg9eqyFtCJb0sMFs6lwVTtZgdBxRzODLmNE0ikpQRH5PFC1+pjpKGZ1fHkZJoR7mZ0CBNmIsMqoy/Es4O27WIRlIwvw1zf4s+C76C7NwHtG4wSDRQ5Qm5MU8IgBEj5vt27HGEwMGDthpAimaHZXFCZA9+3uBY/5M/w3fVuQnByVyqzSOZu8M2nWwXMX0S09ZuqyOKAEnEGqku5F2B0TaAkjPsjDIKWhBUlfQKHmi+KaP/w63mptQMLfqHeUV/Hma7QSlOnj75SApwov5PKLDQbw7a/CYbnJwmJYNIBsWHlA24KRcaXdAY16CaqHjFcybJHpKs2X8zut/9Y11AZh5S8fiNFkUv4T8EPIJuT9qHcKnYBM3poZSRorbWGupqIF7DoMBw6adwaB4E6lYR0we1lEB8VYIyLPPmYmYFy+YnZlX30nCVzrBJC9UvsL1Kk2ogRkr5COgF40m2eNbA4oC80g62GDyU3KOQmw8e/bUag+5HGKkw4w8oJCM39AHE82C+Lr8YBnN5/QyQiHYiYnRl2jpy4YU3yt/oWBx8lSFNgW+npgIkQCipoQbs91IGGFsN0UCORzWvQlhgzsRH69LYoP5G+7idpEHaGpObE9jalGPDIBfQq7NjynXyOfkh2cTzyTXtqFYrmHElm2KOPw5KIKxYoz6YVFSeF8O8dmJrCSP61UlMpOfnVuVSfbsjPlFd8MF0zMScmoy5me7F97PxAffrDTaElxq1ANtUxXaLkB79pym6xZa5HRIyzW9j+pc8xyKKQiXK69IOqBwgSp9XIfZCwxrp6mb54yTPAGSniIg2Tkm4k++RXnNnSTv/lXeg2RPc+uuXdQjqLrzyfJwR+J98kGH5drHeZ+8V+jwIibJ++Rv2Ebt3id/X9z3T/Y+K5XVWrBPC9AE1ICLnSyeDTxWDVysRg34oOrsagBJM2654kPM7cNGWbbPZS46AhqUGWzp07umixgMozJVvCZ7xybGYq62xT85K7YhWgfaD0jidKx5aQPSw4BwomJC8yFaCpNiPiBL00yF2C5ssiXPKKk+mG2y4jfvPnwJsih4jPW65wHrnsgwINjKFj5NyNIE83kFKTEQWZTQ6YEq9GuQFR+l1ef1NXINcV/WdVeNWodYtCwqBeIe6eEf1h+ohzPMsjWWlNxR+wjIOD6esa6c2p9rNX2GgdZOkeF11Ln47QecHODgDxLnSxL+0Qq6e7Sgc2caavm0U7T55ZfDDwDNATp/uylZeWAsSaZZ6nyDfEg+q7aZMZdu8lWRF8sOH6u3cEbPzx4y4w3DuO1XNIsjozNtxM3aTQGRZSOdk8nNExgDDANMzqbwBAJnS6JkGqbk/IEHb2E8b0gLI0gAqDEE4eqiZA/CVbRO50UCCEv6WGwrc1mYpNmjw9gjnlGwHQA7UkU/YxhuNvmjRl59jT5ahsu0vJRW5548gqD5jTIeAfcoUP4VPAZvVB6DxcuEeWwa6mCBFAj5zx7mMPWoifa80O57Pf9ImH9ElduuQCVkTD/W2drljab+XlEbBVHBpriPgpRqXhifclHCzo9YgxBhlTVr9XWocPAylI3P1tvLghFiUQTwQYzxOfNMkWV1waoq7dTq0RllRP6jurwU2xyVBwm8K8sQisipaSrI8lvPOyQHZQlqv9OEatWe6YVcs+jucEE7COczWubPmG49p/SfU8QSMMScwpRO1AIijopy/vQJwN5o/aAJioMBt9zzAQ7px4GgG00gIKkRF5+5jxwB7STsxCWjLZ4Naac4DLJkp52lQZbuG37wYrKKXa9gFawOIFMRTexAymWzmm/TBktnqRbAaZPb2+S65+vJ7Ewq1ebUErHygY3Fi7f8phwcn17On8nnZWNIT53txjANmTpPsWIJz6hjTGKUHMdDlmMb2OMdCg9JiTj5BjG7zSFUp4J5VvsKhu2ceoDV0S8cRvGS+4U7kJSL05Qu4j+llJRNNsuLerOOYomfxlB5tBQ0u5smdTfoOgqQAUfAwKsqA3FE4ZMTU8zm0UsDS+c/G7ob0g2F9LJE55qG1dR/rj4lr8qsju1Z1ivalXM7FqcEhXKc3Fj9o9pwcV02qsXpUHHim0cgT9QYW5E4udDd19uttMJl+i7Y/uJ8rr0LXXvxkoxGsA1ByzwfgioXxn/x/T5swcSsbusqFy6Na1t6qCba3vk0EXrSIX1dkF8fwq8ZlscUgNuZu6BqkduFi9zlPMk6x+u2wseBzIX2WFASr6xzoesCzCixwn5gCUbT4NafWg3ZHUPk1ikiLuXAhRzRwRxWkOWMFStvsBi7j+jPLC8o4M3IYPkVerv/DvP8vQQ82OQpGUqz/Dld0F0PvtL4vy6weyizVkk5kfxiU3Gd/HL7FvX0ZrV6t5SHYQ6UnPGwlepyXJiPMhL+0bI4jw1UQPXfldDjZxtXUcc0eNMdHKDD858HH8AqzPb/et7zbR5wMsvoTat50TnoflekbuNajLJsTkyq7bgPVjf+TI6ixgVAslmVQm/dOfgA3KGd4JawZErfg85qtFxlqe7oenGMi8pvfWF00nWqohDvg5RqqbSiIr/RHJcV5QXKObb+2+U7/4MzKXNZuowi+I2HHmB2NZjhb8mU3GDGF7ksqlbKqub4Z10r++iZcYKd7oekG/wevTkmlLp6CDu/S1712iz5AcmrBl/HZmBYDW+JdkGwWI+Uk7CEZC95YitaQLAMkmAhqEobn681BqXctUJrbWsgmo+l6OTZ9nV2Zl74+NRHJ+dB6D3NZCgfkrTwQjDiQEOCTshYlHWAeIbmKNRgnU8cvoSxUu2TOzuOQ/dA5wnBiqc63esRzqjRPc98xE6v+XBuh97cOl735KDE5VUiQUa/J0joZ609TttCU5T9rcoPJqpLnfyP5AaPQ9H53hyIcQkDJImRKV4reh+UEERpSEWHLq1W9KkpOqgbGlbRz+ZNT0vRMdjy1oUoqVP0IXPAtaKrUXSMgKJ7Ayp6j+bzOmzuDptNUcm1srCZuIEANh03y4ibTVGNtaq4ueJa9DTbBRNnI7GA3lA2zSIdOH8ATedDMowH9KeRqRX9qMB5WEUfcqVYK/pAgfOwiq4Zsl4ogcAZmwMGzkgzZNNXdBA4Yzxg4IzOxpD5jouDXooeIqLq7plndJvPx0eWYIkZsTxi+bo+JEkWJi9RliZLrfEqNB65GMoSnDYQm0rkK71my47BzeYne9OCqInZMhkdChHSbJkU/bP5VD3TEWifMrbM1GxZT5h4x0hgJJX51qZmyz6ApvO5nNgc0Ldmeq0VvR0lvtZiYEXXbNn0Fd3h9wIaVtE1W9YLJb4+jCg6REmdomu2bPqK7jpA0d0BFf1sgfO02DLHAAwHFmYkqGLLTBgqa7ZsmhrvIMC8YkEOqTq2DMN4HdAuxksUvmo8+9kGzgmwBP1hke+qQhPG9aBQUqPZn8vGCHDZw+IJfW9NisogRZHAN1dHimKRc65DqG5SFDkDhlBYtFqsQ6iJaTpPtyF3wBAKn22/tGkpOiBFh1V0UaCrFX1aig5I0WEVXVfT90IJkKLIgSgpU3RLtBitFX1aig5IUcT6gw+h6JZopXwQRa+eaP82Un2bBe7ShxwlYTO/abVrwKAZC6Mtfq+x4zAzAWaf6izmIPANv4GtYlSBHrqsLVeDChHYcjnllBZcyr5eiPDUrb/2pGIDqtkVJPm6glULOZYUkiBRstrQLpLine truncated
+<mxfile modified="2019-09-03T07:00:02.990Z" host="www.draw.io" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36" etag="X_429KB6d-D659ARfbKd" version="11.2.4" type="device" pages="1"><diagram id="33c2a640-8c1e-935c-0e0a-86b5dd5c932c" name="Page-1">7V1be6M40v41uUwehDhe5tDpmd3u3kz37Dez380+xCYOOxi8GCed/vUrYYRBJQO2JQxpZffpsWVzMG9VqepVVekC3y6/f8yC1fPndB7GF6Yx/36B7y5ME1mmQ/5DR962I66LtwOLLJqXX9oNfIt+hOWgUY5uonm4bnwxT9M4j1bNwVmaJOEsb4wFWZa+Nr/2lMbNq66CRQgGvs2CGI7+Ec3z5+2oZxu78V/CaPHMroyM8pPHYPbXIks3SXm9CxM/FX/bj5cBO1f5/fVzME9fa0P4wwW+zdI0375afr8NY/ps2WPbHne/59PqvrMwyXsdgEtgXoJ4E7J7Lu4sf2NPo/g9IT0CXeCb1+coD7+tghn99JXgT8ae82VcfryIgzV9+gZ5PUuX0ax8vc6z9K/wNo3TrDgrdmZe+PhUfcKeMyYjT1Ec1745D0LvaUbH0yQvhcU0yve17xnFHxkP4miRkLE4fMrp22xWHuWQd/ARlU/tJczy8HttqHxkH8N0GebZG/lKJeBeid8bk1vf2w687sTFYkLxXBMV7JTPNyhFdFGdfQcTeVEiJUbN8RSCNg/Wz9VxrQh+cOj/+iBY6gC5UBbMo3CHWpImoQxgXTnAEnwauJqGY0BcLQGuDrMNp+DqGxrXXBKSblNDqY0GSGIRksiSgSQCSAZknhrGsvruneG6kvFDZgt+MgDzrSubg8zxrzy8+7MBgJXG1AGUgp/ZrYlhMr+mvgZ5N6PQUEDqeFVzO31CcfAYxjeVd1B7iDfF/8Ro9Xz85Klnb3/SK5FHWL79195513Dvr2/aQMuDbBHmTUEO5w2PCcJYQ8UWgMLGsjAO8uil6WeJkCqv8JBG5O4qKalsL5ORymVk51inm2wWlofVPR9wJpuXN2xw59o+CHAugnrwVvvain5hfchNY6P13mzfPfEAZLcfAG6JO4C82P7KnbZUWPdSILeHAumprOdUxjkluO9UZvICfYwpRCqdzeHBazWkDcGSHjaYTZ2rwskajI4jihl8GTOaRlEJisjviaLFz1RHKaPz08MoKdTD/Awo0EaMRUZVhn8JZ8fNOiQDSZi/ptlfBd9Ff2EWziMaNxgkeojShLyYRwSA6HGzfTvWeGLAwAE7TSBFs6OyOAGyZ/8scMyf6b/h9xX5yUGJ3CqNo9kbg3YdLFcx/dJTli6rI0rACYQa6W6k3QGRtgDSsywMcgpakNQVNEqeKL7p43/CreYmFMytekd5BX+eZjtBqQ7efjlIivBiPo/ocBDvzho8ppscHKZlA8iGhQeUDTgpV9od0JiXoFroeAXzJome0mwZv/H6X31jXQBm3tKxOE0WxS8hP4R8Qu6PWofwKdjEjamhlJHiNtZaKmrgnsNgwLBpZzAo3kQq1hGTh3VUQLwVAvLsc2Yi5sULZmfm1XeS8JVOMMkLla9wvUoTamDGCvkI6EWjSfb41oCiwDySDjaY/JScoxAbz549tdpDLocY6TAjDygk4zf0wUSzIL4uP1hG8zm9jFAIdmJi9CVa+rIhxffKXyhYnDxVoU2BrycmQiSAqCnhxmw3EkYY202RQA6HdW9C2OBOxMfrkthg/oa7uF3kAZqaE9vTmFrUIwPgp5Br833KNfI5+eHZxDPJtW0olmsYsWWbIg5/DopgrBijfliUFN6XQ3x2IivJ43pViczkZ+dWZZI9O2N+0d1wwfSMhJyajPnZ7oX3M/HBNyuNtgSXGvVA21SFtgvQnj2n6bqFFjkd0nJN77061zyHYgrC5corkg4oXKBKH9dh9gLD2mnq5jnjJE+ApKcISHaOifiT36O85k6Sd/8q70Gyp7l11y7qEVTd+WR5uCPxPvmgw3Lt47xP3it0eBGT5H3yN2yjdu+Tvy/u+yd7n5XKai3YpwVoAmrAxU4WzwYeqwYuVqMGfFB1djWApBm3XPEu5vZhoyzb5zIXHQENygy29Old00UMhlGZKl6TvWMTYzFX2+KfnBXbEK0D7QckcTrWvLQB6WFAOFExofkQLYVJMR+QpWmmQmwXNtmSZ5RUH8w2WfGbdx++BFkUPMZ63fOAdU9kGBBsZQufJmRpgvm8gpQYiCxK6PRAFfo1yIqP0urz+hq5hrgv67qrRq1DLFoWlQJxj/Twd+sP1MMZZtkaS0ruqH0EZBwfz1hXTu3PtZo+w0Brp8jwOupc/PYDTg5w8DuJ8yUJ/2gF3T1a0LkzDbV82ina/PLL4QeA5gCdv92UrDwwliTTLHW+QT4kn1XbzJhLN/mqyItlh4/VWzij52cPmfGGYdz2M5rFkdGZNuJm7aaAyLKRzsnk5gmMAYYBJmdTeAKBsyVRMg1Tcv7Ag7cwnjekhREkANQYgnB1UbIH4Spap/MiAYQlfSy2lbksTNLs0WHsEc8o2A6AHaminzEMN5v8USOvvkYfLcNlWl5Kq3NPHkHQ/EYZj4B7FCj/DB6DNyqPweJlwjw2DXWwQAqE/GcPc5h61ER7Xmj3vZ5/JMw/osptV6ASMqYf62zt8kZTf6+ojYKoYFPcR0FKNS+MT7koYedHrEGIsMqatfo6VDh4GcrGZ+vtZcEIsSgCeCfG+Jx5psiyumBVlXZq9eiMMiL/UV1eim2OyoME3pVlCEXk1DQVZPmt5x2Sg7IEtd9pQrVqz/RCrll0d7igHYTzGS3zZ0y3nlP6zyliCRhiTmFKJ2oBEUdFOX/6BGBvtH7QBMXBgFvu+QCH9ONA0I0mEJDUiIvP3EeOgHYSduKS0RbPhrRTHAZZstPO0iBL9w3feTFZxa5XsApWB5CpiCZ2IOWyWc23aYOls1QL4LTJ7W1y3fP1ZHYmlWpzaolY+cDG4sVbflMOjk8v58/k87IxpKfOdmOYhkydp1ixhGfUMSYxSo7jIcuxDezxDoWHpEScfIOY3eYQqlPBPKt9BcN2Tj3A6ugXDqN4yf3CHUjKxWlKF/GfUkrKJpvlRb1ZR7HET2OoPFoKmt1Nk7obdB0FyIAjYOBVlYE4ovDJiSlm8+ilgaXz3w3dDemGQnpZonNNw2rqP1efkldlVsf2LOsV7cq5HYtTgkI5Tm6s/lFtuLguG9XidKg48c0jkCdqjK1InFzo7uvtVlrhMn0XbH9xPtfeha69eElGI9iGoGWeD0GVC+M/+X4ftmBiVrd1lQuXxrUtPVQTbe98mgg96ZC+Lsivd+HXDMtjCsDtzF1QtcjtwkXucp5kneN1W+HjQOZCeywoiVfWudB1AWaUWGE/sASjaXDrT62G7I4hcusUEZdy4EKO6GAOK8hyxoqVN1iM3Uf0Z5YXFPBmZLD8Cr3d/4R5/lYCHmzylAylWf6cLuiuB59o/F8X2D2UWauknEh+sam4Tn65fYt6erNavVvKwzAHSs542Ep1OS7MRxkJ/2hZnMcGKqD670ro8bONq6hjGrzpDg7Q4fnPgw9gFWb7fz3v+TYPOJll9KbVvOgcdL8rUrdxLUZZNicm1XbcB6sbfyZHUeMCINmsSqG37hx8AO7QTnBLWDKl70FnNVquslR3dL04xkXlt74wOuk6VVGI905KtVRaUZHfaI7LivIC5Rxb/+3ynf/BmZS5LF1GEfzGQw8wuxrM8LdkSm4w44tcFlUrZVVz/LOulb33zDjBTvdD0g1+j94cE0pdPYSd3yWvem2W/IDkVYOvYzMwrIa3RLsgWKxHyklYQrKXPLEVLSBYBkmwEFSljc/XGoNS7lqhtbY1EM3HUnTybPs6OzMvfHzqo5PzIPSeZjKUD0laeCEYcaAhQSdkLMo6QDxDcxRqsM4nDl/CWKn2yZ0dx6F7oPOEYMVTne71CGfU6J5nPmKn13w4t0Nvbh2ve3JQ4vIqkSCj3xMk9LPWHqdtoSnK/lblBxPVpU7+e3KDx6HofG8OxLiEAZLEyBSvFb0PSgiiNKSiQ5dWK/rUFB3UDQ2r6Gfzpqel6BhseetClNQp+pA54FrR1Sg6RkDRvQEVvUfzeR02d4fNpqjkWlnYTNxAAJuOm2XEzaaoxlpV3FxxLXqa7YKJs5FYQG8om2aRDpzfgabzIRnGA/rTyNSKflTgPKyiD7lSrBV9oMB5WEXXDFkvlEDgjM0BA2ekGbLpKzoInDEeMHBGZ2PIfMfFQS9FDxFRdffMM7rN5+MjS7DEjFgesXxdH5IkC5OXKEuTpdZ4FRqPXAxlCU4biE0l8pVes2XH4Gbzk71pQdTEbJmMDoUIabZMiv7ZfKqe6Qi0TxlbZmq2rCdMvGMkMJLKfGtTs2XvQNP5XE5sDuhbM73Wit6OEl9rMbCia7Zs+oru8HsBDavomi3rhRJfH0YUHaKkTtE1WzZ9RXcdoOjugIp+tsB5WmyZYwCGAwszElSxZSYMlTVbNk2NdxBgXrEgh1QdW4ZhvA5oF+MlCl81nv1sA+cEWIL+sMh3VaEJ43pQKKnR7M9lYwS47GHxhL63JkVlkKJI4JurI0WxyDnXIVQ3KYqcAUMoLFot1iHUxDSdp9uQO2AIhc+2X9q0FB2QosMquijQ1Yo+LUUHpOiwiq6r6XuhBEhR5ECUlCm6JVqM1oo+LUUHpChi/cGHUHRLtFI+iKJXT7R/G6m+zQJ36UOOkrCZ37TaNWDQjIXRFr/X2HGYmQCzD3UWcxD4ht/AVjGqQA9d1parQYUIbLmcckoLLmVfL0R46tZfe1KxAdXsCpJ8XcGqhRxLLine truncated