notebooks/Module 2.3.1 - Statistical Significance.ipynb · sinomiko/QuantFinance

加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

文件

克隆/下载

Module 2.3.1 - Statistical Significance.ipynb 40.57 KB

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Quantitative Finance\n",
    "\n",
    "Copyright (c) 2019 Python Charmers Pty Ltd, Australia, <https://pythoncharmers.com>. All rights reserved.\n",
    "\n",
    "<img src=\"img/python_charmers_logo.png\" width=\"300\" alt=\"Python Charmers Logo\">\n",
    "\n",
    "Published under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. See `LICENSE.md` for details.\n",
    "\n",
    "Sponsored by Tibra Global Services, <https://tibra.com>\n",
    "\n",
    "<img src=\"img/tibra_logo.png\" width=\"300\" alt=\"Tibra Logo\">\n",
    "\n",
    "\n",
    "## Module 2.3: Testing and Benchmarking\n",
    "\n",
    "### 2.3.1 Statistical Significance\n",
    "\n",
    "When we want to test the results of an experiment, a normal pattern is to setup a Control and treatment group. The terminology comes from medical science, but applies to any field. The Control group is \"do the task as you have previously done\", or \"keep everything normal\". The Treatment group is \"apply my idea to this group\". We then evaluate the different in the two groups to determine if there is a difference or not - normally referred to as a *significant* difference, but we will come back to that term.\n",
    "\n",
    "Suppose we have a new trading algorithm (we'll call NEW) we want to test. The current one works (named OLD) well enough, but initial testing of NEW indicates some good results. Given the random nature of the stock market, and randomness inherent in both the NEW and OLD algorithms, we can't just run them once and compare the results. In statistics, we always want to ask \"Isn't there some probability that this difference happened by chance?\". Statistics helps us protect against making decisions based on (un)lucky data sampling that happens.\n",
    "\n",
    "We create 30 iterations of both NEW and OLD, and backtest on the last 5 years of trading data. We get the following profits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run setup.ipy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "NEW = np.array([1.05174941, 1.06930758, 1.11729439, 1.12624468, 1.60443569,\n",
    "       1.16272344, 1.47328065, 1.05485681, 0.74028953, 1.21066169,\n",
    "       1.13720454, 1.70111553, 1.22645839, 1.26188635, 1.30603338,\n",
    "       2.10036382, 1.68648174, 1.27467569, 0.37090243, 1.17720112,\n",
    "       1.25108935, 1.21632526, 1.58731637, 1.08608151, 2.08776142,\n",
    "       0.63474195, 0.70729046, 0.6496959 , 1.61753557, 1.0645431 ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "OLD = np.array([0.9847197 , 1.27252081, 0.77785125, 1.53240267, 0.97049964,\n",
    "       1.09014078, 0.92529125, 0.93391001, 1.06337962, 1.04707554,\n",
    "       1.04699074, 1.24765968, 0.97622673, 1.21298906, 1.14389947,\n",
    "       1.08432808, 1.24983952, 0.98100972, 1.34957539, 1.1513302 ,\n",
    "       1.63546461, 0.69778236, 1.46165873, 1.09680951, 1.18708603,\n",
    "       1.04704617, 1.24966216, 0.90329866, 1.41676504, 1.59918173])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.2251849250000002, 1.1445464953333333)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NEW.mean(), OLD.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result is great! NEW outperforms OLD. However, isn't there a chance this occurred by chance? First, check the histograms. The mean is (usually) the best single-value summary of data, but we can learn so much, so quickly by just doing some quick visualisations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAbvUlEQVR4nO3df5DVVf348dcCspDDLqHyKxHRyh+IqCUMaonjKhI58E9Kow5RalNYEaVJk7KM6aofR01jpBwVKwVtEm1MKaMWJsNfgCVmBEa2ZUCa7gLq1rDn+0fD/Xbl58K95+6Vx2Pmznjf99z3+9zDe+99evdeqEkppQAAyKRbpScAAOxfxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGTVo9ITeLeOjo549dVXo0+fPlFTU1Pp6QAAeyClFJs2bYrBgwdHt267fm+jy8XHq6++GkOGDKn0NACAvdDS0hKHHnroLsd0ufjo06dPRPx38nV1dRWeDQCwJ9ra2mLIkCGF1/Fd6XLxse1XLXV1deIDAKrMnnxkwgdOAYCsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZ9aj0BIC911jpCeyFxkpPAKg473wAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGTV6fhYunRpnHvuuTF48OCoqamJhx9+uOj2lFJcffXVMWjQoOjdu3c0NDTEmjVrSjVfAKDKdTo+tmzZEiNHjow5c+bs8PYbb7wxbrvttpg7d248/fTTceCBB8a4cePinXfe2efJAgDVr0dn7zB+/PgYP378Dm9LKcWtt94a3/rWt2LixIkREfGDH/wgBgwYEA8//HBMnjx532YLAFS9kn7mY926dbF+/fpoaGgobKuvr4/Ro0fHsmXLdnif9vb2aGtrK7oAAO9dJY2P9evXR0TEgAEDirYPGDCgcNu7NTU1RX19feEyZMiQUk4JAOhiKv5tl5kzZ0Zra2vh0tLSUukpAQBlVNL4GDhwYEREbNiwoWj7hg0bCre9W21tbdTV1RVdAID3rpLGx7Bhw2LgwIGxePHiwra2trZ4+umnY8yYMaU8FABQpTr9bZfNmzfH2rVrC9fXrVsXzz//fPTr1y8OO+ywmD59enz729+OD33oQzFs2LC46qqrYvDgwTFp0qRSzhsAqFKdjo/nnnsuzjjjjML1GTNmRETElClTYt68eXHFFVfEli1b4tJLL40333wzTjvttFi0aFH06tWrdLMGAKpWTUopVXoS/6utrS3q6+ujtbXV5z9gNxorPYG90FjpCQBl0ZnX74p/2wUA2L+IDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJBVyeNj69atcdVVV8WwYcOid+/eceSRR8Y111wTKaVSHwoAqEI9Sr3DG264Ie6444649957Y/jw4fHcc8/F1KlTo76+Pr785S+X+nAAQJUpeXz89re/jYkTJ8aECRMiIuLwww+P+fPnxzPPPFPqQwEAVajkv3Y55ZRTYvHixfGnP/0pIiJ+97vfxW9+85sYP378Dse3t7dHW1tb0QUAeO8q+TsfV155ZbS1tcXRRx8d3bt3j61bt8a1114bF1xwwQ7HNzU1xezZs0s9DQCgiyr5Ox8PPvhg3HfffXH//ffHihUr4t57742bbrop7r333h2OnzlzZrS2thYuLS0tpZ4SANCFlPydj8svvzyuvPLKmDx5ckREjBgxIl555ZVoamqKKVOmbDe+trY2amtrSz0NAKCLKvk7H2+99VZ061a82+7du0dHR0epDwUAVKGSv/Nx7rnnxrXXXhuHHXZYDB8+PFauXBk333xzfPazny31oQCAKlTy+Lj99tvjqquuii9+8YuxcePGGDx4cHz+85+Pq6++utSHAgCqUE3qYn/1aFtbW9TX10dra2vU1dVVejrQpTVWegJ7obHSEwDKojOv3/5tFwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAsipLfPz973+PCy+8MA466KDo3bt3jBgxIp577rlyHAoAqDI9Sr3DN954I0499dQ444wz4vHHH49DDjkk1qxZE+9///tLfSgAoAqVPD5uuOGGGDJkSNxzzz2FbcOGDSv1YQCAKlXyX7v89Kc/jY9+9KPxqU99Kvr37x8nnnhi3HnnnTsd397eHm1tbUUXAOC9q+TvfPz5z3+OO+64I2bMmBHf/OY349lnn40vf/nL0bNnz5gyZcp245uammL27NmlngZ0LY2N5dnv2LE7/m+ALqzk73x0dHTESSedFNddd12ceOKJcemll8Yll1wSc+fO3eH4mTNnRmtra+HS0tJS6ikBAF1IyeNj0KBBceyxxxZtO+aYY+Kvf/3rDsfX1tZGXV1d0QUAeO8qeXyceuqpsXr16qJtf/rTn2Lo0KGlPhQAUIVKHh9f/epX46mnnorrrrsu1q5dG/fff398//vfj2nTppX6UABAFSp5fJx88smxcOHCmD9/fhx33HFxzTXXxK233hoXXHBBqQ8FAFShkn/bJSLik5/8ZHzyk58sx64BgCrn33YBALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALLqUekJQJfQ2FjpGey75uZKz2DP7Gqe74U/hy6ksdIT2AuNlZ4AWXjnAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgq7LHx/XXXx81NTUxffr0ch8KAKgCZY2PZ599Nr73ve/F8ccfX87DAABVpGzxsXnz5rjgggvizjvvjPe///3lOgwAUGXKFh/Tpk2LCRMmRENDwy7Htbe3R1tbW9EFAHjv6lGOnS5YsCBWrFgRzz777G7HNjU1xezZs8sxDaALahw7ttJT6LTGSk8A3mNK/s5HS0tLfOUrX4n77rsvevXqtdvxM2fOjNbW1sKlpaWl1FMCALqQkr/zsXz58ti4cWOcdNJJhW1bt26NpUuXxne/+91ob2+P7t27F26rra2N2traUk8DAOiiSh4fZ555ZrzwwgtF26ZOnRpHH310fOMb3ygKDwBg/1Py+OjTp08cd9xxRdsOPPDAOOigg7bbDgDsf/wNpwBAVmX5tsu7NTc35zgMAFAFvPMBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGTVo9ITYPcaKz2BvdBY6QkA0GV55wMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AIKuSx0dTU1OcfPLJ0adPn+jfv39MmjQpVq9eXerDAABVquTxsWTJkpg2bVo89dRT8cQTT8R//vOfOPvss2PLli2lPhQAUIV6lHqHixYtKro+b9686N+/fyxfvjw+/vGPl/pwAECVKXl8vFtra2tERPTr12+Ht7e3t0d7e3vheltbW7mnBABUUFnjo6OjI6ZPnx6nnnpqHHfccTsc09TUFLNnzy7nNCi35uY927YvGhtLu78dHWLs2LIfg90o9Xnzbjn+jDOcq3tsbx+vn4VOaaz0BPZCY4WPX9Zvu0ybNi1WrVoVCxYs2OmYmTNnRmtra+HS0tJSzikBABVWtnc+Lrvssnj00Udj6dKlceihh+50XG1tbdTW1pZrGgBAF1Py+EgpxZe+9KVYuHBhNDc3x7Bhw0p9CACgipU8PqZNmxb3339/PPLII9GnT59Yv359RETU19dH7969S304AKDKlPwzH3fccUe0trbG2LFjY9CgQYXLAw88UOpDAQBVqCy/dgEA2Bn/tgsAkJX4AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkFWPSk8gt8ZKT2A/0Th2bKWn0DnVNl/2TnPzXt2tsTODnUuwW975AACyEh8AQFbiAwDISnwAAFmJDwAgK/EBAGQlPgCArMQHAJCV+AAAshIfAEBW4gMAyEp8AABZiQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkJX4AACyEh8AQFbiAwDIqmzxMWfOnDj88MOjV69eMXr06HjmmWfKdSgAoIqUJT4eeOCBmDFjRsyaNStWrFgRI0eOjHHjxsXGjRvLcTgAoIqUJT5uvvnmuOSSS2Lq1Klx7LHHxty5c+N973tf3H333eU4HABQRXqUeof//ve/Y/ny5TFz5szCtm7dukVDQ0MsW7Zsu/Ht7e3R3t5euN7a2hoREW1tbaWe2n+PV5a97ue2bCn/Mcp0PhTkeAxQLcr987arQ1fsyHuvGl9XyrHO2163U0q7HVvy+Hjttddi69atMWDAgKLtAwYMiD/+8Y/bjW9qaorZs2dvt33IkCGlnhoAXdz1lZ7AfqKc67xp06aor6/f5ZiSx0dnzZw5M2bMmFG43tHREf/617/ioIMOipqamgrOrHza2tpiyJAh0dLSEnV1dZWeTkVYA2sQYQ0irEGENdim2tchpRSbNm2KwYMH73ZsyePj4IMPju7du8eGDRuKtm/YsCEGDhy43fja2tqora0t2ta3b99ST6tLqqurq8oTrJSsgTWIsAYR1iDCGmxTzeuwu3c8tin5B0579uwZH/nIR2Lx4sWFbR0dHbF48eIYM2ZMqQ8HAFSZsvzaZcaMGTFlypT46Ec/GqNGjYpbb701tmzZElOnTi3H4QCAKlKW+Dj//PPjn//8Z1x99dWxfv36OOGEE2LRokXbfQh1f1VbWxuzZs3a7tdN+xNrYA0irEGENYiwBtvsT+tQk/bkOzEAACXi33YBALISHwBAVuIDAMhKfAAAWYmPEpgzZ04cfvjh0atXrxg9enQ888wzOx07b968qKmpKbr06tWraExKKa6++uoYNGhQ9O7dOxoaGmLNmjXlfhj7pDNrMHbs2O3WoKamJiZMmFAY85nPfGa7288555wcD2WvLF26NM4999wYPHhw1NTUxMMPP7zb+zQ3N8dJJ50UtbW18cEPfjDmzZu33ZjOrGuldXYNHnrooTjrrLPikEMOibq6uhgzZkz8/Oc/LxrT2Ni43Xlw9NFHl/FR7JvOrkFzc/MOfxbWr19fNK6azoOIzq/Djn7ea2pqYvjw4YUx1XQuNDU1xcknnxx9+vSJ/v37x6RJk2L16tW7vd+Pf/zjOProo6NXr14xYsSIeOyxx4pur8bXhp0RH/vogQceiBkzZsSsWbNixYoVMXLkyBg3blxs3Lhxp/epq6uLf/zjH4XLK6+8UnT7jTfeGLfddlvMnTs3nn766TjwwANj3Lhx8c4775T74eyVzq7BQw89VPT4V61aFd27d49PfepTRePOOeeconHz58/P8XD2ypYtW2LkyJExZ86cPRq/bt26mDBhQpxxxhnx/PPPx/Tp0+Piiy8uevHdm3Orkjq7BkuXLo2zzjorHnvssVi+fHmcccYZce6558bKlSuLxg0fPrzoPPjNb35TjumXRGfXYJvVq1cXPcb+/fsXbqu28yCi8+vwne98p+jxt7S0RL9+/bZ7TqiWc2HJkiUxbdq0eOqpp+KJJ56I//znP3H22WfHll38A5a//e1v49Of/nR87nOfi5UrV8akSZNi0qRJsWrVqsKYantt2KXEPhk1alSaNm1a4frWrVvT4MGDU1NT0w7H33PPPam+vn6n++vo6EgDBw5M//d//1fY9uabb6ba2to0f/78ks27lDq7Bu92yy23pD59+qTNmzcXtk2ZMiVNnDix1FPNIiLSwoULdznmiiuuSMOHDy/adv7556dx48YVru/rulbSnqzBjhx77LFp9uzZheuzZs1KI0eOLN3EMtqTNfj1r3+dIiK98cYbOx1TzedBSnt3LixcuDDV1NSkv/zlL4Vt1XwubNy4MUVEWrJkyU7HnHfeeWnChAlF20aPHp0+//nPp5Sq87VhV7zzsQ/+/e9/x/Lly6OhoaGwrVu3btHQ0BDLli3b6f02b94cQ4cOjSFDhsTEiRPjxRdfLNy2bt26WL9+fdE+6+vrY/To0bvcZ6Xs7Rr8r7vuuismT54cBx54YNH25ubm6N+/fxx11FHxhS98IV5//fWSzr2Sli1bVrRmERHjxo0rrFkp1rXadHR0xKZNm6Jfv35F29esWRODBw+OI444Ii644IL461//WqEZls8JJ5wQgwYNirPOOiuefPLJwvb98TyI+O9zQkNDQwwdOrRoe7WeC62trRER253b/2t3zwnV9tqwO+JjH7z22muxdevW7f7m1gEDBmz3O9ttjjrqqLj77rvjkUceiR/96EfR0dERp5xySvztb3+LiCjcrzP7rKS9WYP/9cwzz8SqVavi4osvLtp+zjnnxA9+8INYvHhx3HDDDbFkyZIYP358bN26taTzr5T169fvcM3a2tri7bff3ud1rUY33XRTbN68Oc4777zCttGjR8e8efNi0aJFcccdd8S6deviYx/7WGzatKmCMy2dQYMGxdy5c+MnP/lJ/OQnP4khQ4bE2LFjY8WKFRGx7z9f1ejVV1+Nxx9/fLvnhGo9Fzo6OmL69Olx6qmnxnHHHbfTcTt7Ttj251xtrw27U5a/Xp2dGzNmTNE/sHfKKafEMcccE9/73vfimmuuqeDMKuOuu+6KESNGxKhRo4q2T548ufDfI0aMiOOPPz6OPPLIaG5ujjPPPDP3NCmz+++/P2bPnh2PPPJI0ecdxo8fX/jv448/PkaPHh1Dhw6NBx98MD73uc9VYqolddRRR8VRRx1VuH7KKafEyy+/HLfcckv88Ic/rODMKufee++Nvn37xqRJk4q2V+u5MG3atFi1alWX/XxKpXjnYx8cfPDB0b1799iwYUPR9g0bNsTAgQP3aB8HHHBAnHjiibF27dqIiML99mWfOe3LGmzZsiUWLFiwR08cRxxxRBx88MGFdap2AwcO3OGa1dXVRe/evUtyblWLBQsWxMUXXxwPPvjgdm87v1vfvn3jwx/+8HvmPNiRUaNGFR7f/nQeRPz32xx33313XHTRRdGzZ89djq2Gc+Gyyy6LRx99NH7961/HoYceusuxO3tO2PbnXG2vDbsjPvZBz5494yMf+UgsXry4sK2joyMWL15c9O7GrmzdujVeeOGFGDRoUEREDBs2LAYOHFi0z7a2tnj66af3eJ857csa/PjHP4729va48MILd3ucv/3tb/H6668X1qnajRkzpmjNIiKeeOKJwpqV4tyqBvPnz4+pU6fG/Pnzi75qvTObN2+Ol19++T1zHuzI888/X3h8+8t5sM2SJUti7dq1e/Q/JF35XEgpxWWXXRYLFy6MX/3qVzFs2LDd3md3zwnV9tqwW5X+xGu1W7BgQaqtrU3z5s1Lf/jDH9Kll16a+vbtm9avX59SSumiiy5KV155ZWH87Nmz089//vP08ssvp+XLl6fJkyenXr16pRdffLEw5vrrr099+/ZNjzzySPr973+fJk6cmIYNG5befvvt7I9vT3R2DbY57bTT0vnnn7/d9k2bNqWvf/3radmyZWndunXpl7/8ZTrppJPShz70ofTOO++U/fHsjU2bNqWVK1emlStXpohIN998c1q5cmV65ZVXUkopXXnllemiiy4qjP/zn/+c3ve+96XLL788vfTSS2nOnDmpe/fuadGiRYUxu1vXrqaza3DfffelHj16pDlz5qR//OMfhcubb75ZGPO1r30tNTc3p3Xr1qUnn3wyNTQ0pIMPPjht3Lgx++PbE51dg1tuuSU9/PDDac2aNemFF15IX/nKV1K3bt3SL3/5y8KYajsPUur8Omxz4YUXptGjR+9wn9V0LnzhC19I9fX1qbm5uejcfuuttwpj3v28+OSTT6YePXqkm266Kb300ktp1qxZ6YADDkgvvPBCYUy1vTbsivgogdtvvz0ddthhqWfPnmnUqFHpqaeeKtx2+umnpylTphSuT58+vTB2wIAB6ROf+ERasWJF0f46OjrSVVddlQYMGJBqa2vTmWeemVavXp3r4eyVzqxBSin98Y9/TBGRfvGLX2y3r7feeiudffbZ6ZBDDkkHHHBAGjp0aLrkkku69JPttq9Mvvuy7XFPmTIlnX766dvd54QTTkg9e/ZMRxxxRLrnnnu22++u1rWr6ewanH766bscn9J/v348aNCg1LNnz/SBD3wgnX/++Wnt2rV5H1gndHYNbrjhhnTkkUemXr16pX79+qWxY8emX/3qV9vtt5rOg5T27ufhzTffTL17907f//73d7jPajoXdvTYI6LoZ3xHz4sPPvhg+vCHP5x69uyZhg8fnn72s58V3V6Nrw07U5NSSuV9bwUA4P/zmQ8AICvxAQBkJT4AgKzEBwCQlfgAALISHwBAVuIDAMhKfAAAWYkPACAr8QEAZCU+AICsxAcAkNX/A7+HmcKZy4+BAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(OLD, alpha=0.5, color='red')\n",
    "plt.hist(NEW, alpha=0.5, color='aqua');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NEW has a greater spread, but looks to be generally a bit higher. More profit, but more risk. (Sidenote: normally we would analyse the variance more here, but we are focusing on testing this improvement in this module.)\n",
    "\n",
    "We *could* fit distribution models to the data, determine if the results are normal, fit normal distributions, compare the results using a two-side t-test. This will work for this data, but may not work for other data in the real world.\n",
    "\n",
    "A method that is as rigourous\\*, but more broadly applicable is simulation. You did a Monte Carlo simulation in a previous Extended Exercise - here we will cover a simple pattern that you can use, even if you don't have a simulation environment setup.\n",
    "\n",
    "\\* It is just as rigourous, only if you have sufficient data. The data requirement for simulations is higher than a t-test.\n",
    "\n",
    "### Re-randomised subsets\n",
    "\n",
    "Our data is currently split into two groups - NEW and OLD. Comparing the difference of means, we get this value: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0806384296666669"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_difference_in_means = NEW.mean() - OLD.mean()\n",
    "experiment_difference_in_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to know - \"is this result impressive?\". More specifically, \"what is the probability that this happened by chance?\".\n",
    "\n",
    "In trading, results can be volatile. This means that we can observe differences this large, by just choosing a different subset of stocks, or a different random starting point in our algorithm.\n",
    "\n",
    "To evaluate this, we take *all* the data we have collected so far, and create a new, randomly selected split. We then compute the difference of means. We do this many times, and then see how many resulting in a difference of means at least as great:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.shuffle?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_random_subsets(all_data):\n",
    "    \"\"\"Randomly splits all_data into two groups\"\"\"\n",
    "    # Note this is an in-place operation - all_data is changed by this call!\n",
    "    # If you didn't want this to happen, create a copy of all_data first. We dont' care here though\n",
    "    np.random.shuffle(all_data)\n",
    "    midpoint = int(len(all_data) / 2)\n",
    "    return all_data[:midpoint], all_data[midpoint:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60,)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_data = np.hstack([NEW, OLD])\n",
    "all_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample1, sample2 = create_random_subsets(all_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.70729046, 1.68648174, 2.08776142, 1.04704617, 1.70111553,\n",
       "       0.77785125, 1.09680951, 1.24765968, 0.93391001, 1.0645431 ,\n",
       "       1.14389947, 1.06930758, 1.21632526, 1.11729439, 0.98100972,\n",
       "       1.04707554, 1.09014078, 1.08608151, 1.1513302 , 1.13720454,\n",
       "       1.58731637, 0.37090243, 1.59918173, 1.25108935, 1.05485681,\n",
       "       0.97622673, 1.30603338, 1.61753557, 0.69778236, 1.06337962])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04210193966666664"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample1.mean() - sample2.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the time, when you run this, you'll get a value near 0. This is because:\n",
    "\n",
    "* We assume that NEW was generated by one mechanism, just randomly within that mechanism\n",
    "* We assume that OLD was generated by one mechanism (different to NEW)\n",
    "* Our random sample will have about the same number of NEW and OLD values\n",
    "\n",
    "This all averages out, and we expect each sample's mean to be the mean of the whole dataset. Given two samples like this, the difference will be about zero. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_and_compare(all_data):\n",
    "    \"\"\"Run a single iteration of (1) split randomly, and (2) compute difference of means\"\"\"\n",
    "    # Same code as above, just in a function\n",
    "    sample1, sample2 = create_random_subsets(all_data)\n",
    "    return sample1.mean() - sample2.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do that 1000 times\n",
    "differences = np.array([sample_and_compare(all_data) for _ in range(1000)])\n",
    "\n",
    "# Sidenote: the _ after the word for is a valid variable name.\n",
    "# We use _ as a variable name to indicate to future readers of the program that:\n",
    "# \"This is a variable, but I don't really care about it, its value doesn't matter\".\n",
    "# Often you'll see i here instead - that's perfectly fine too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.00028032517400000987"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "differences.mean()  # Expected to be about zero. Your result may vary slightly\n",
    "# The reason behind this zero estimate is that each data point, ether being from NEW or OLD,\n",
    "# is always counted once either positively or negatively (with the same probability) in \n",
    "# the sample_and_compare(all_data) iteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([  2.,  27.,  66., 148., 201., 236., 171., 102.,  33.,  14.]),\n",
       " array([-0.26254333, -0.21219628, -0.16184923, -0.11150218, -0.06115513,\n",
       "        -0.01080808,  0.03953897,  0.08988602,  0.14023307,  0.19058012,\n",
       "         0.24092717]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAa4ElEQVR4nO3df2yV5d348U9LbcHJaa1CayOIbiri0BmctZlOn9lYkLkRWTYJMciILAZctHMKxonuRyCGTKJRiWYTl8j8sUQ3ZWIYKMxZUBlmij+iBiMKrQqjBYwF7P394xvP81RBLLQ9V8vrldyJve+rp59zhY137p5zKMqyLAsAgIQUF3oAAIDPEygAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkp6TQAxyIjo6O2LRpUwwePDiKiooKPQ4A8BVkWRbbt2+PmpqaKC7+8nskfTJQNm3aFMOGDSv0GADAAdi4cWMce+yxX7qmTwbK4MGDI+L/P8FcLlfgaQCAr6KtrS2GDRuW/3v8y/TJQPns1zq5XE6gAEAf81VenuFFsgBAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJCckkIPAPB5I2YtKfQIXfbOvPGFHgH6FXdQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkdClQ5s6dG9/+9rdj8ODBMXTo0JgwYUK88cYbndZ88sknMWPGjDjqqKPiiCOOiIkTJ0ZLS0unNe+++26MHz8+Dj/88Bg6dGj88pe/jD179hz8swEA+oUuBcrKlStjxowZsXr16li2bFns3r07Lrzwwti5c2d+zTXXXBOPP/54PPLII7Fy5crYtGlTXHLJJfnrn376aYwfPz527doVzz33XNx///2xaNGiuOmmm7rvWQEAfVpRlmXZgX7zhx9+GEOHDo2VK1fGd7/73WhtbY0hQ4bE4sWL40c/+lFERLz++utxyimnRFNTU5x99tnx5JNPxve///3YtGlTVFVVRUTEwoUL4/rrr48PP/wwSktL9/tz29raory8PFpbWyOXyx3o+HBIGDFrSaFHOCS8M298oUeA5HXl7++Deg1Ka2trRERUVlZGRMTatWtj9+7dUV9fn18zcuTIGD58eDQ1NUVERFNTU4wePTofJxERDQ0N0dbWFuvXr9/rz2lvb4+2trZOBwDQfx1woHR0dMTVV18d3/nOd+Kb3/xmREQ0NzdHaWlpVFRUdFpbVVUVzc3N+TX/N04+u/7Ztb2ZO3dulJeX549hw4Yd6NgAQB9wwIEyY8aMeOWVV+LBBx/sznn2avbs2dHa2po/Nm7c2OM/EwAonJID+aaZM2fGE088EatWrYpjjz02f766ujp27doV27Zt63QXpaWlJaqrq/Nrnn/++U6P99m7fD5b83llZWVRVlZ2IKMCAH1Ql+6gZFkWM2fOjEcffTRWrFgRxx9/fKfrY8aMicMOOyyWL1+eP/fGG2/Eu+++G3V1dRERUVdXFy+//HJ88MEH+TXLli2LXC4Xo0aNOpjnAgD0E126gzJjxoxYvHhx/PWvf43BgwfnXzNSXl4egwYNivLy8pg2bVo0NjZGZWVl5HK5uOqqq6Kuri7OPvvsiIi48MILY9SoUXHZZZfFrbfeGs3NzXHjjTfGjBkz3CUBACKii4Fy9913R0TE+eef3+n8fffdF5dffnlERNx2221RXFwcEydOjPb29mhoaIi77rorv3bAgAHxxBNPxJVXXhl1dXXxta99LaZMmRK//vWvD+6ZAAD9xkF9Dkqh+BwU+Op8Dkrv8DkosH+99jkoAAA9QaAAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkJySQg8A0B+MmLWk0CN02Tvzxhd6BNgnd1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOV0OlFWrVsXFF18cNTU1UVRUFI899lin65dffnkUFRV1OsaOHdtpzdatW2Py5MmRy+WioqIipk2bFjt27DioJwIA9B9dDpSdO3fG6aefHnfeeec+14wdOzY2b96cP/785z93uj558uRYv359LFu2LJ544olYtWpVTJ8+vevTAwD9UklXv2HcuHExbty4L11TVlYW1dXVe7322muvxdKlS+OFF16IM888MyIi7rjjjrjoooti/vz5UVNT09WRAIB+pkdeg/LMM8/E0KFD4+STT44rr7wytmzZkr/W1NQUFRUV+TiJiKivr4/i4uJYs2ZNT4wDAPQxXb6Dsj9jx46NSy65JI4//vh4++2344Ybbohx48ZFU1NTDBgwIJqbm2Po0KGdhygpicrKymhubt7rY7a3t0d7e3v+67a2tu4eGwBISLcHyqWXXpr/79GjR8dpp50WX//61+OZZ56JCy644IAec+7cuXHLLbd014gAQOJ6/G3GJ5xwQhx99NHx1ltvRUREdXV1fPDBB53W7NmzJ7Zu3brP163Mnj07Wltb88fGjRt7emwAoIB6PFDee++92LJlSxxzzDEREVFXVxfbtm2LtWvX5tesWLEiOjo6ora2dq+PUVZWFrlcrtMBAPRfXf4Vz44dO/J3QyIiNmzYEC+99FJUVlZGZWVl3HLLLTFx4sSorq6Ot99+O6677rr4xje+EQ0NDRERccopp8TYsWPjiiuuiIULF8bu3btj5syZcemll3oHD8kbMWtJoUcAOCR0+Q7Kiy++GGeccUacccYZERHR2NgYZ5xxRtx0000xYMCA+M9//hM/+MEP4qSTTopp06bFmDFj4p///GeUlZXlH+OBBx6IkSNHxgUXXBAXXXRRnHPOOXHPPfd037MCAPq0Lt9BOf/88yPLsn1ef+qpp/b7GJWVlbF48eKu/mgA4BDh3+IBAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASE6XA2XVqlVx8cUXR01NTRQVFcVjjz3W6XqWZXHTTTfFMcccE4MGDYr6+vp48803O63ZunVrTJ48OXK5XFRUVMS0adNix44dB/VEAID+o6Sr37Bz5844/fTT46c//WlccsklX7h+6623xu233x73339/HH/88fGrX/0qGhoa4tVXX42BAwdGRMTkyZNj8+bNsWzZsti9e3dMnTo1pk+fHosXLz74ZwTAVzJi1pJCj9Bl78wbX+gR6CVdDpRx48bFuHHj9noty7JYsGBB3HjjjfHDH/4wIiL+9Kc/RVVVVTz22GNx6aWXxmuvvRZLly6NF154Ic4888yIiLjjjjvioosuivnz50dNTc1BPB0AoD/o1tegbNiwIZqbm6O+vj5/rry8PGpra6OpqSkiIpqamqKioiIfJxER9fX1UVxcHGvWrNnr47a3t0dbW1unAwDov7o1UJqbmyMioqqqqtP5qqqq/LXm5uYYOnRop+slJSVRWVmZX/N5c+fOjfLy8vwxbNiw7hwbAEhMn3gXz+zZs6O1tTV/bNy4sdAjAQA9qFsDpbq6OiIiWlpaOp1vaWnJX6uuro4PPvig0/U9e/bE1q1b82s+r6ysLHK5XKcDAOi/ujVQjj/++Kiuro7ly5fnz7W1tcWaNWuirq4uIiLq6upi27ZtsXbt2vyaFStWREdHR9TW1nbnOABAH9Xld/Hs2LEj3nrrrfzXGzZsiJdeeikqKytj+PDhcfXVV8dvf/vbOPHEE/NvM66pqYkJEyZERMQpp5wSY8eOjSuuuCIWLlwYu3fvjpkzZ8all17qHTwAQEQcQKC8+OKL8T//8z/5rxsbGyMiYsqUKbFo0aK47rrrYufOnTF9+vTYtm1bnHPOObF06dL8Z6BERDzwwAMxc+bMuOCCC6K4uDgmTpwYt99+ezc8HQCgPyjKsiwr9BBd1dbWFuXl5dHa2ur1KPSqvvjBVtCf+KC2vq0rf3/3iXfxAACHFoECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAckoKPQCHrhGzlhR6BAAS5Q4KAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyen2QLn55pujqKio0zFy5Mj89U8++SRmzJgRRx11VBxxxBExceLEaGlp6e4xAIA+rEfuoJx66qmxefPm/PHss8/mr11zzTXx+OOPxyOPPBIrV66MTZs2xSWXXNITYwAAfVRJjzxoSUlUV1d/4Xxra2v84Q9/iMWLF8f3vve9iIi477774pRTTonVq1fH2Wef3RPjAAB9TI/cQXnzzTejpqYmTjjhhJg8eXK8++67ERGxdu3a2L17d9TX1+fXjhw5MoYPHx5NTU37fLz29vZoa2vrdAAA/Ve3B0ptbW0sWrQoli5dGnfffXds2LAhzj333Ni+fXs0NzdHaWlpVFRUdPqeqqqqaG5u3udjzp07N8rLy/PHsGHDuntsACAh3f4rnnHjxuX/+7TTTova2to47rjj4uGHH45BgwYd0GPOnj07Ghsb81+3tbWJFADox3r8bcYVFRVx0kknxVtvvRXV1dWxa9eu2LZtW6c1LS0te33NymfKysoil8t1OgCA/qvHA2XHjh3x9ttvxzHHHBNjxoyJww47LJYvX56//sYbb8S7774bdXV1PT0KANBHdPuveK699tq4+OKL47jjjotNmzbFnDlzYsCAATFp0qQoLy+PadOmRWNjY1RWVkYul4urrroq6urqvIMHAMjr9kB57733YtKkSbFly5YYMmRInHPOObF69eoYMmRIRETcdtttUVxcHBMnToz29vZoaGiIu+66q7vHAAD6sKIsy7JCD9FVbW1tUV5eHq2trV6P0oeNmLWk0CMAfcw788YXegQOQlf+/vZv8QAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAckoKPQAAfFUjZi0p9Ahd9s688YUeoU9yBwUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDk+qK2f6IsfXgQA++IOCgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkR6AAAMkRKABAcgQKAJAcgQIAJEegAADJESgAQHIECgCQHIECACRHoAAAyREoAEByBAoAkByBAgAkp6TQAwBAfzZi1pJCj3BA3pk3vqA/3x0UACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJIjUACA5AgUACA5AgUASI5AAQCSI1AAgOQUNFDuvPPOGDFiRAwcODBqa2vj+eefL+Q4AEAiSgr1gx966KFobGyMhQsXRm1tbSxYsCAaGhrijTfeiKFDhxZqrIiIGDFrSUF/PgAc6gp2B+X3v/99XHHFFTF16tQYNWpULFy4MA4//PD44x//WKiRAIBEFOQOyq5du2Lt2rUxe/bs/Lni4uKor6+PpqamL6xvb2+P9vb2/Netra0REdHW1tYj83W0f9wjjwsAfUVP/B372WNmWbbftQUJlI8++ig+/fTTqKqq6nS+qqoqXn/99S+snzt3btxyyy1fOD9s2LAemxEADmXlC3rusbdv3x7l5eVfuqZgr0HpitmzZ0djY2P+646Ojti6dWscddRRUVRUVMDJ+p62trYYNmxYbNy4MXK5XKHH6Zfsce+wzz3PHveOQ2mfsyyL7du3R01NzX7XFiRQjj766BgwYEC0tLR0Ot/S0hLV1dVfWF9WVhZlZWWdzlVUVPTkiP1eLpfr9/9DKDR73Dvsc8+zx73jUNnn/d05+UxBXiRbWloaY8aMieXLl+fPdXR0xPLly6Ourq4QIwEACSnYr3gaGxtjypQpceaZZ8ZZZ50VCxYsiJ07d8bUqVMLNRIAkIiCBcpPfvKT+PDDD+Omm26K5ubm+Na3vhVLly79wgtn6V5lZWUxZ86cL/zKjO5jj3uHfe559rh32Oe9K8q+ynt9AAB6kX+LBwBIjkABAJIjUACA5AgUACA5AqWf27p1a0yePDlyuVxUVFTEtGnTYseOHV+6/qqrroqTTz45Bg0aFMOHD4+f//zn+X//iL3r6j5HRNxzzz1x/vnnRy6Xi6Kioti2bVvvDNuH3HnnnTFixIgYOHBg1NbWxvPPP/+l6x955JEYOXJkDBw4MEaPHh1///vfe2nSvqsre7x+/fqYOHFijBgxIoqKimLBggW9N2gf15V9vvfee+Pcc8+NI488Mo488sior6/f75/9/kig9HOTJ0+O9evXx7Jly+KJJ56IVatWxfTp0/e5ftOmTbFp06aYP39+vPLKK7Fo0aJYunRpTJs2rRen7nu6us8RER9//HGMHTs2brjhhl6asm956KGHorGxMebMmRP//ve/4/TTT4+Ghob44IMP9rr+ueeei0mTJsW0adNi3bp1MWHChJgwYUK88sorvTx539HVPf7444/jhBNOiHnz5u31U7/Zu67u8zPPPBOTJk2Kp59+OpqammLYsGFx4YUXxvvvv9/LkxdYRr/16quvZhGRvfDCC/lzTz75ZFZUVJS9//77X/lxHn744ay0tDTbvXt3T4zZ5x3sPj/99NNZRGT//e9/e3DKvuess87KZsyYkf/6008/zWpqarK5c+fudf2Pf/zjbPz48Z3O1dbWZj/72c96dM6+rKt7/H8dd9xx2W233daD0/UfB7PPWZZle/bsyQYPHpzdf//9PTViktxB6ceampqioqIizjzzzPy5+vr6KC4ujjVr1nzlx2ltbY1cLhclJX3i35bsdd21z/yvXbt2xdq1a6O+vj5/rri4OOrr66OpqWmv39PU1NRpfUREQ0PDPtcf6g5kj+m67tjnjz/+OHbv3h2VlZU9NWaSBEo/1tzcHEOHDu10rqSkJCorK6O5ufkrPcZHH30Uv/nNb/b764pDWXfsM5199NFH8emnn37hk6Wrqqr2uafNzc1dWn+oO5A9puu6Y5+vv/76qKmp+UKA93cCpQ+aNWtWFBUVfenx+uuvH/TPaWtri/Hjx8eoUaPi5ptvPvjB+5je2meAfZk3b148+OCD8eijj8bAgQMLPU6vcs++D/rFL34Rl19++ZeuOeGEE6K6uvoLL8Las2dPbN26db8vcNu+fXuMHTs2Bg8eHI8++mgcdthhBzt2n9Mb+8zeHX300TFgwIBoaWnpdL6lpWWfe1pdXd2l9Ye6A9ljuu5g9nn+/Pkxb968+Mc//hGnnXZaT46ZJIHSBw0ZMiSGDBmy33V1dXWxbdu2WLt2bYwZMyYiIlasWBEdHR1RW1u7z+9ra2uLhoaGKCsri7/97W+HXLV/pqf3mX0rLS2NMWPGxPLly2PChAkREdHR0RHLly+PmTNn7vV76urqYvny5XH11Vfnzy1btizq6up6YeK+50D2mK470H2+9dZb43e/+1089dRTnV7fdkgp9Kt06Vljx47NzjjjjGzNmjXZs88+m5144onZpEmT8tffe++97OSTT87WrFmTZVmWtba2ZrW1tdno0aOzt956K9u8eXP+2LNnT6GeRvK6us9ZlmWbN2/O1q1bl917771ZRGSrVq3K1q1bl23ZsqUQTyE5Dz74YFZWVpYtWrQoe/XVV7Pp06dnFRUVWXNzc5ZlWXbZZZdls2bNyq//17/+lZWUlGTz58/PXnvttWzOnDnZYYcdlr388suFegrJ6+oet7e3Z+vWrcvWrVuXHXPMMdm1116brVu3LnvzzTcL9RT6hK7u87x587LS0tLsL3/5S6f/D96+fXuhnkJBCJR+bsuWLdmkSZOyI444IsvlctnUqVM7/SHfsGFDFhHZ008/nWXZ/77ldW/Hhg0bCvMk+oCu7nOWZdmcOXP2us/33Xdf7z+BRN1xxx3Z8OHDs9LS0uyss87KVq9enb923nnnZVOmTOm0/uGHH85OOumkrLS0NDv11FOzJUuW9PLEfU9X9vizP8efP84777zeH7yP6co+H3fccXvd5zlz5vT+4AVUlGVZ1nv3awAA9s+7eACA5AgUACA5AgUASI5AAQCSI1AAgOQIFAAgOQIFAEiOQAEAkiNQAIDkCBQAIDkCBQBIjkABAJLz/wBFObW98Ca7cgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(differences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just eyeballing it, you can see *about* 1/3 of the values are above 0.08. We normally test the absolute value - i.e. how likely is it we get an absolute value above the observed one. This gives a clearer picture if we are unsure of which group (control - OLD, or treatment - NEW) is better.\n",
    "\n",
    "Let's find out exactly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.356"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(np.abs(differences) > experiment_difference_in_means).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that 36% of the values in our simulation are above the observed difference in the data. Therefore we can say that:\n",
    "\n",
    "    There is a 36% chance that the observed difference between NEW and OLD occurred by chance.\n",
    "    \n",
    "A drawback of using this method to determine significance is the necessity of having the same number of data points in OLD and NEW, one can use some random sampling method to adress this problem such as bootstrapping (https://en.wikipedia.org/wiki/Bootstrapping_(statistics)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is that difference \"significant\"?\n",
    "\n",
    "A key term you'll hear, and use, when presenting results of statistical tests is whether the result is significant or not. This is a loaded term in statistics, so it is important to distinguish between these two variants:\n",
    "\n",
    "* **Statistical significance** occurs when the Null Hypothesis is unlikely to have occurred by chance, given a significance threshold (usually 0.05). That is, if there is a less than 5% chance of the Null hypothesis having occurred by chance, we claim this result is **statistically significant**.\n",
    "* A **significant** result is one that causes an improvement that is worth the investment. That is, if a treatment causes an improvement, and we want to use the treatment now, that result is significant.\n",
    "\n",
    "\n",
    "In the above result, we can say that the result of our experiment is **not statistically significant**. That is because there is a 36% chance that the difference in the profit between NEW and OLD occurred purely by chance.\n",
    "\n",
    "However, when we compare the average profit, NEW has a profit 8 percentage points higher than OLD. If we are happy with the increased risk, then we would say that this increase in profits is **significant** and that we should switch to NEW for our trading strategy.\n",
    "\n",
    "More often than not, a significant result that is not statistically significant is just a case of not enough data. If I generate the data again, using the same mechanism as I did for the data at the start of this module, but with many, many more data points, we'll get a result that is both significant and statistically significant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NEW: mu=0.17, std = 0.35\n",
    "# OLD: mu = 0.12, std=0.25\n",
    "N_SAMPLES = 100000\n",
    "NEW_large = 1 + np.random.randn(N_SAMPLES) * 0.35 + 0.17\n",
    "OLD_large = 1 + np.random.randn(N_SAMPLES) * 0.25 + 0.12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment_difference_in_means_large = NEW_large.mean() - OLD_large.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.04871454033710165"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_difference_in_means_large  # Not as large as before, but 5 percentage points is still good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rerun our simulations. This takes quite a bit longer than before.\n",
    "all_data_large = np.hstack([NEW_large, OLD_large])\n",
    "differences_large = np.array([sample_and_compare(all_data_large) for _ in range(10000)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(np.abs(differences_large) > experiment_difference_in_means_large).mean()  # Percentage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(np.abs(differences_large) > experiment_difference_in_means_large).sum()  # Absolute value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In other words, *none* of the observed difference in means in our randomised subsets were as high as our experimental difference. (Your results may vary, but you'll get a very low number here, and a percentage nearly 0).\n",
    "\n",
    "This result is both **significant**, in that we expect more money now from the new strategy, and **statistically significant** because it is unlikely this result occurred by chance. The data didn't change from above - we just used it in a more structured format.\n",
    "\n",
    "Note that there are some cases where it isn't just a data issue. In these cases the variance is so large, or the data has a pattern that causes the above to be inconclusive from a statistical perspective. However be mindful of the difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A common misconception with p\n",
    "\n",
    "Most people misinterpret what a p value means. Let's say we have our original result of NEW and OLD (the one with 30 samples each). A common misinterpretation of the \"high\" p value is to say there is \"no difference\" between the two samples. If we are measuring a correlation and get a high p value, one might say there is \"no correlation\", even if we do get a linear correlation value, just one that has a high p value. In these cases, one would be wrong to say this.\n",
    "\n",
    "Our observed difference was 0.08 - not \"no difference\". There is just a 33.6% chance that randomly doing the experiment would yield the same, or higher, result.\n",
    "\n",
    "[This article in Nature](https://www.nature.com/articles/d41586-019-00857-9) gives an example of two experiments on the same treatment - an anti-inflammatory drug. Both experiments tested if there was a correlation between using the drug and new-onset atrial fibrillation. One paper concludes the drug is associated with the condition. One paper concludes that it is not associated with the condition. Which is correct? \n",
    "\n",
    "In the \"no association\" experiment, the 95% confidence interval spanned between a decrease of 3% of the risk of the condition to a 48% increase in risk. However, due to the statistical test, the confidence interval included 0, indicating that there is greater than 5% chance that the increased risk observed was due to chance. The Nature article calculated the p value as 0.091.\n",
    "\n",
    "The \"association\" experiment found the 95% confidence interval between a 9% increased risk to a 33% increased risk. As zero is not in the confidence interval, they concluded that there was an association.\n",
    "\n",
    "Here is the key finding - both experiments had a mean increased risk of 20%. The second experiment just had more data. An average increased risk of 20% for atrial fibrillation is **significant**. Data sample size indicated that it was not **statistically significant** in the first \"no association\" experiment.\n",
    "\n",
    "There are calls from lots of scientists to stop using the term \"statistically significant\". Further, studies have shown about *half* of papers wrongly interpret their p values. The issue isn't so much with the tests themselves, it is the interpretation of the tests that is the problem, and the common \"intuition\" that \"not statistically significant\" means \"no difference\" or \"not important\".\n",
    "\n",
    "To make matters worse, most people will use 0.05 as their threshold without any consideration for what this means, and whether such a value makes sense. \n",
    "\n",
    "The solution is to be clear with what your test results present, and how they could be used in decision making. Do not use shortcuts like \"statistically significant\", when key decisions (like your p value threshold of 0.05 or your \"95%\" confidence interval) are arbitrarily defined. \n",
    "\n",
    "\n",
    "#### Exercise\n",
    "\n",
    "1. Read the linked Nature article on problems with P values: https://www.nature.com/articles/d41586-019-00857-9\n",
    "2. Write a summary (about two or three sentences) explaining the results of the 30-sample NEW versus OLD result. Be clear on terminology. You can compute confidence intervals as well to present these findings, but again, be clear about what that means.\n",
    "\n",
    "If you are doing this course with a group or partner, get them to check your explanation, and discuss whether you would move to NEW based on the summary you presented.\n",
    "\n",
    "Note: There is no solution to this exercise"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

一键复制编辑原始数据按行查看历史

提交于 2024-02-12 21:11 . dharshan

马建仓 AI 助手

尝试更多

代码解读

代码找茬

代码优化

https://gitee.com/sinomiko/QuantFinance.git

git@gitee.com:sinomiko/QuantFinance.git

sinomiko

QuantFinance

master

sinomiko/QuantFinance

简介

发行版

贡献者

近期动态

sinomiko/QuantFinance .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

sinomiko/QuantFinance