Forked from zhaleh-rahimi/2.1Prediction1Dregression_v3.ipynb
Created
August 17, 2022 16:02
-
-
Save mohayl/3e94dc9f0dcf0109eeedeecdf2c8d9fc to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"cells":[{"cell_type":"markdown","metadata":{},"source":["<center>\n"," <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\" />\n","</center>\n"]},{"cell_type":"markdown","metadata":{},"source":["<h1>Linear regression: Training and Validation Data</h1> \n"]},{"cell_type":"markdown","metadata":{},"source":["<h2>Objective</h2><ul><li> How to use learning rate hyperparameter to improve your model result. .</li></ul> \n"]},{"cell_type":"markdown","metadata":{},"source":["<h2>Table of Contents</h2>\n","<p>In this lab, you will learn to select the best learning rate by using validation data.</p>\n","\n","<ul>\n"," <li><a href=\"#Makeup_Data\">Make Some Data</a></li>\n"," <li><a href=\"#LR_Loader_Cost\">Create a Linear Regression Object, Data Loader and Criterion Function</a></li>\n"," <li><a href=\"#LR_Hyper\">Different learning rates and Data Structures to Store results for Different Hyperparameters</a></li>\n"," <li><a href=\"#Model\">Train different modules for different Hyperparameters</a></li>\n"," <li><a href=\"#Result\">View Results</a></li>\n","</ul>\n","\n","<p>Estimated Time Needed: <strong>30 min</strong></p>\n","\n","<hr>\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2>Preparation</h2>\n"]},{"cell_type":"markdown","metadata":{},"source":["We'll need the following libraries and set the random seed.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Import libraries we need for this lab, and set the random seed\n","\n","from torch import nn\n","import torch\n","import numpy as np\n","import matplotlib.pyplot as plt\n","from torch import nn,optim"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2 id=\"Makeup_Data\">Make Some Data</h2>\n"]},{"cell_type":"markdown","metadata":{},"source":["First, we'll create some artificial data in a dataset class. The class will include the option to produce training data or validation data. The training data will include outliers.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create Data class\n","\n","from torch.utils.data import Dataset, DataLoader\n","\n","class Data(Dataset):\n"," \n"," # Constructor\n"," def __init__(self, train = True):\n"," self.x = torch.arange(-3, 3, 0.1).view(-1, 1)\n"," self.f = -3 * self.x + 1\n"," self.y = self.f + 0.1 * torch.randn(self.x.size())\n"," self.len = self.x.shape[0]\n"," \n"," #outliers \n"," if train == True:\n"," self.y[0] = 0\n"," self.y[50:55] = 20\n"," else:\n"," pass\n"," \n"," # Getter\n"," def __getitem__(self, index): \n"," return self.x[index], self.y[index]\n"," \n"," # Get Length\n"," def __len__(self):\n"," return self.len"]},{"cell_type":"markdown","metadata":{},"source":["Create two objects: one that contains training data and a second that contains validation data. Assume that the training data has the outliers. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create training dataset and validation dataset\n","\n","train_data = Data()\n","val_data = Data(train = False)"]},{"cell_type":"markdown","metadata":{},"source":["Overlay the training points in red over the function that generated the data. Notice the outliers at x=-3 and around x=2:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Plot out training points\n","\n","plt.plot(train_data.x.numpy(), train_data.y.numpy(), 'xr',label=\"training data \")\n","plt.plot(train_data.x.numpy(), train_data.f.numpy(),label=\"true function \")\n","plt.xlabel('x')\n","plt.ylabel('y')\n","plt.legend()\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2 id=\"LR_Loader_Cost\">Create a Linear Regression Object, Data Loader, and Criterion Function</h2>\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create Linear Regression Class\n","\n","from torch import nn\n","\n","class linear_regression(nn.Module):\n"," \n"," # Constructor\n"," def __init__(self, input_size, output_size):\n"," super(linear_regression, self).__init__()\n"," self.linear = nn.Linear(input_size, output_size)\n"," \n"," # Prediction function\n"," def forward(self, x):\n"," yhat = self.linear(x)\n"," return yhat"]},{"cell_type":"markdown","metadata":{},"source":["Create the criterion function and a <code>DataLoader</code> object: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create MSELoss function and DataLoader\n","\n","criterion = nn.MSELoss()\n","trainloader = DataLoader(dataset = train_data, batch_size = 1)"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2 id=\"LR_Hyper\">Different learning rates and Data Structures to Store results for different Hyperparameters</h2>\n"]},{"cell_type":"markdown","metadata":{},"source":["Create a list with different learning rates and a tensor (can be a list) for the training and validating cost/total loss. Include the list MODELS, which stores the training model for every value of the learning rate. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create Learning Rate list, the error lists and the MODELS list\n","\n","learning_rates=[0.0001, 0.001, 0.01, 0.1]\n","\n","train_error=torch.zeros(len(learning_rates))\n","validation_error=torch.zeros(len(learning_rates))\n","\n","MODELS=[]"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2 id=\"Model\">Train different models for different Hyperparameters</h2>\n"]},{"cell_type":"markdown","metadata":{},"source":["Try different values of learning rates, perform stochastic gradient descent, and save the results on the training data and validation data. Finally, save each model in a list.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define the train model function and train the model\n","\n","def train_model_with_lr (iter, lr_list):\n"," \n"," # iterate through different learning rates \n"," for i, lr in enumerate(lr_list):\n"," model = linear_regression(1, 1)\n"," optimizer = optim.SGD(model.parameters(), lr = lr)\n"," for epoch in range(iter):\n"," for x, y in trainloader:\n"," yhat = model(x)\n"," loss = criterion(yhat, y)\n"," optimizer.zero_grad()\n"," loss.backward()\n"," optimizer.step()\n"," \n"," # train data\n"," Yhat = model(train_data.x)\n"," train_loss = criterion(Yhat, train_data.y)\n"," train_error[i] = train_loss.item()\n"," \n"," # validation data\n"," Yhat = model(val_data.x)\n"," val_loss = criterion(Yhat, val_data.y)\n"," validation_error[i] = val_loss.item()\n"," MODELS.append(model)\n","\n","train_model_with_lr(10, learning_rates)"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2 id=\"Result\">View the Results</h2>\n"]},{"cell_type":"markdown","metadata":{},"source":["Plot the training loss and validation loss for each learning rate: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Plot the training loss and validation loss\n","\n","plt.semilogx(np.array(learning_rates), train_error.numpy(), label = 'training loss/total Loss')\n","plt.semilogx(np.array(learning_rates), validation_error.numpy(), label = 'validation cost/total Loss')\n","plt.ylabel('Cost\\ Total Loss')\n","plt.xlabel('learning rate')\n","plt.legend()\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["Produce a prediction by using the validation data for each model: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Plot the predictions\n","\n","i = 0\n","for model, learning_rate in zip(MODELS, learning_rates):\n"," yhat = model(val_data.x)\n"," plt.plot(val_data.x.numpy(), yhat.detach().numpy(), label = 'lr:' + str(learning_rate))\n"," print('i', yhat.detach().numpy()[0:3])\n","plt.plot(val_data.x.numpy(), val_data.f.numpy(), 'or', label = 'validation data')\n","plt.xlabel('x')\n","plt.ylabel('y')\n","plt.legend()\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h3>Practice</h3>\n"]},{"cell_type":"markdown","metadata":{},"source":["The object <code>good_model</code> is the best performing model. Use the train loader to get the data samples x and y. Produce an estimate for <code>yhat</code> and print it out for every sample in a for a loop. Compare it to the actual prediction <code>y</code>.\n"]},{"cell_type":"markdown","metadata":{},"source":["Double-click <b>here</b> for the solution.\n","\n","<!-- \n","for x, y in trainloader:\n"," print(\"yhat= \", good_model(x),\"y\", y)\n","-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<a href=\"https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&apps=data_science_experience,watson_machine_learning\"><img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-DL0110EN-SkillsNetwork/Template/module%201/images/Watson_Studio.png\"/></a>\n"]},{"cell_type":"markdown","metadata":{},"source":["<!--Empty Space for separating topics-->\n"]},{"cell_type":"markdown","metadata":{},"source":["<h2>About the Authors:</h2> \n","\n","<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"]},{"cell_type":"markdown","metadata":{},"source":["Other contributors: <a href=\"https://www.linkedin.com/in/michelleccarey/\">Michelle Carey</a>, <a href=\"www.linkedin.com/in/jiahui-mavis-zhou-a4537814a\">Mavis Zhou</a>\n"]},{"cell_type":"markdown","metadata":{},"source":["## Change Log\n","\n","| Date (YYYY-MM-DD) | Version | Changed By | Change Description |\n","| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n","| 2020-09-23 | 2.0 | Shubham | Migrated Lab to Markdown and added to course repo in GitLab |\n"]},{"cell_type":"markdown","metadata":{},"source":["<hr>\n"]},{"cell_type":"markdown","metadata":{},"source":["## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":2} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment