{"cells":[{"cell_type":"markdown","metadata":{},"source":["
\n"," \"cognitiveclass.ai\n","
\n"]},{"cell_type":"markdown","metadata":{},"source":["

Linear Regression 1D: Training Two Parameter Mini-Batch Gradient Decent

\n"]},{"cell_type":"markdown","metadata":{},"source":["

Objective

\n"]},{"cell_type":"markdown","metadata":{},"source":["

Table of Contents

\n","

In this Lab, you will practice training a model by using Mini-Batch Gradient Descent.

\n","\n","\n","

Estimated Time Needed: 30 min

\n","\n","\n","
\n"]},{"cell_type":"markdown","metadata":{},"source":["

Preparation

\n"]},{"cell_type":"markdown","metadata":{},"source":["We'll need the following libraries:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Import the libraries we need for this lab\n","\n","import numpy as np\n","import matplotlib.pyplot as plt\n","from mpl_toolkits import mplot3d"]},{"cell_type":"markdown","metadata":{},"source":["The class plot_error_surfaces is just to help you visualize the data space and the parameter space during training and has nothing to do with PyTorch. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# The class for plotting the diagrams\n","\n","class plot_error_surfaces(object):\n"," \n"," # Constructor\n"," def __init__(self, w_range, b_range, X, Y, n_samples = 30, go = True):\n"," W = np.linspace(-w_range, w_range, n_samples)\n"," B = np.linspace(-b_range, b_range, n_samples)\n"," w, b = np.meshgrid(W, B) \n"," Z = np.zeros((30, 30))\n"," count1 = 0\n"," self.y = Y.numpy()\n"," self.x = X.numpy()\n"," for w1, b1 in zip(w, b):\n"," count2 = 0\n"," for w2, b2 in zip(w1, b1):\n"," Z[count1, count2] = np.mean((self.y - w2 * self.x + b2) ** 2)\n"," count2 += 1\n"," count1 += 1\n"," self.Z = Z\n"," self.w = w\n"," self.b = b\n"," self.W = []\n"," self.B = []\n"," self.LOSS = []\n"," self.n = 0\n"," if go == True:\n"," plt.figure()\n"," plt.figure(figsize = (7.5, 5))\n"," plt.axes(projection = '3d').plot_surface(self.w, self.b, self.Z, rstride = 1, cstride = 1, cmap = 'viridis', edgecolor = 'none')\n"," plt.title('Loss Surface')\n"," plt.xlabel('w')\n"," plt.ylabel('b')\n"," plt.show()\n"," plt.figure()\n"," plt.title('Loss Surface Contour')\n"," plt.xlabel('w')\n"," plt.ylabel('b')\n"," plt.contour(self.w, self.b, self.Z)\n"," plt.show()\n"," \n"," # Setter\n"," def set_para_loss(self, W, B, loss):\n"," self.n = self.n + 1\n"," self.W.append(W)\n"," self.B.append(B)\n"," self.LOSS.append(loss)\n"," \n"," # Plot diagram\n"," def final_plot(self): \n"," ax = plt.axes(projection = '3d')\n"," ax.plot_wireframe(self.w, self.b, self.Z)\n"," ax.scatter(self.W, self.B, self.LOSS, c = 'r', marker = 'x', s = 200, alpha = 1)\n"," plt.figure()\n"," plt.contour(self.w, self.b, self.Z)\n"," plt.scatter(self.W, self.B, c = 'r', marker = 'x')\n"," plt.xlabel('w')\n"," plt.ylabel('b')\n"," plt.show()\n"," \n"," # Plot diagram\n"," def plot_ps(self):\n"," plt.subplot(121)\n"," plt.ylim()\n"," plt.plot(self.x, self.y, 'ro', label = \"training points\")\n"," plt.plot(self.x, self.W[-1] * self.x + self.B[-1], label = \"estimated line\")\n"," plt.xlabel('x')\n"," plt.ylabel('y')\n"," plt.title('Data Space Iteration: '+ str(self.n))\n"," plt.subplot(122)\n"," plt.contour(self.w, self.b, self.Z)\n"," plt.scatter(self.W, self.B, c = 'r', marker = 'x')\n"," plt.title('Loss Surface Contour')\n"," plt.xlabel('w')\n"," plt.ylabel('b')\n"," plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Make Some Data

\n"]},{"cell_type":"markdown","metadata":{},"source":["Import PyTorch and set random seed:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Import PyTorch library\n","\n","import torch\n","torch.manual_seed(1)"]},{"cell_type":"markdown","metadata":{},"source":["Generate values from -3 to 3 that create a line with a slope of 1 and a bias of -1. This is the line that you need to estimate. Add some noise to the data:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Generate the data with noise and the line\n","\n","X = torch.arange(-3, 3, 0.1).view(-1, 1)\n","f = 1 * X - 1\n","Y = f + 0.1 * torch.randn(X.size())"]},{"cell_type":"markdown","metadata":{},"source":["Plot the results:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Plot the line and the data\n","\n","plt.plot(X.numpy(), Y.numpy(), 'rx', label = 'y')\n","plt.plot(X.numpy(), f.numpy(), label = 'f')\n","plt.xlabel('x')\n","plt.ylabel('y')\n","plt.legend()\n","plt.show()"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Create the Model and Cost Function (Total Loss)

\n"]},{"cell_type":"markdown","metadata":{},"source":["Define the forward function: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define the prediction function\n","\n","def forward(x):\n"," return w * x + b"]},{"cell_type":"markdown","metadata":{},"source":["Define the cost or criterion function: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define the cost function\n","\n","def criterion(yhat, y):\n"," return torch.mean((yhat - y) ** 2)"]},{"cell_type":"markdown","metadata":{},"source":["Create a plot_error_surfaces object to visualize the data space and the parameter space during training:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create a plot_error_surfaces object.\n","\n","get_surface = plot_error_surfaces(15, 13, X, Y, 30)"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Train the Model: Batch Gradient Descent (BGD)

\n"]},{"cell_type":"markdown","metadata":{},"source":["Define train_model_BGD function.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define the function for training model\n","\n","w = torch.tensor(-15.0, requires_grad = True)\n","b = torch.tensor(-10.0, requires_grad = True)\n","lr = 0.1\n","LOSS_BGD = []\n","\n","def train_model_BGD(epochs):\n"," for epoch in range(epochs):\n"," Yhat = forward(X)\n"," loss = criterion(Yhat, Y)\n"," LOSS_BGD.append(loss)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist())\n"," get_surface.plot_ps()\n"," loss.backward()\n"," w.data = w.data - lr * w.grad.data\n"," b.data = b.data - lr * b.grad.data\n"," w.grad.data.zero_()\n"," b.grad.data.zero_()"]},{"cell_type":"markdown","metadata":{},"source":["Run 10 epochs of batch gradient descent: bug data space is 1 iteration ahead of parameter space. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Run train_model_BGD with 10 iterations\n","\n","train_model_BGD(10)"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Stochastic Gradient Descent (SGD) with Dataset DataLoader

\n"]},{"cell_type":"markdown","metadata":{},"source":["Create a plot_error_surfaces object to visualize the data space and the parameter space during training:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create a plot_error_surfaces object.\n","\n","get_surface = plot_error_surfaces(15, 13, X, Y, 30, go = False)"]},{"cell_type":"markdown","metadata":{},"source":["Import Dataset and DataLoader libraries\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Import libraries\n","\n","from torch.utils.data import Dataset, DataLoader"]},{"cell_type":"markdown","metadata":{},"source":["Create Data class\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create class Data\n","\n","class Data(Dataset):\n"," \n"," # Constructor\n"," def __init__(self):\n"," self.x = torch.arange(-3, 3, 0.1).view(-1, 1)\n"," self.y = 1 * X - 1\n"," self.len = self.x.shape[0]\n"," \n"," # Getter\n"," def __getitem__(self, index): \n"," return self.x[index], self.y[index]\n"," \n"," # Get length\n"," def __len__(self):\n"," return self.len"]},{"cell_type":"markdown","metadata":{},"source":["Create a dataset object and a dataloader object: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create Data object and DataLoader object\n","\n","dataset = Data()\n","trainloader = DataLoader(dataset = dataset, batch_size = 1)"]},{"cell_type":"markdown","metadata":{},"source":["Define train_model_SGD function for training the model.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define train_model_SGD function\n","\n","w = torch.tensor(-15.0, requires_grad = True)\n","b = torch.tensor(-10.0, requires_grad = True)\n","LOSS_SGD = []\n","lr = 0.1\n","def train_model_SGD(epochs):\n"," for epoch in range(epochs):\n"," Yhat = forward(X)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), criterion(Yhat, Y).tolist())\n"," get_surface.plot_ps()\n"," LOSS_SGD.append(criterion(forward(X), Y).tolist())\n"," for x, y in trainloader:\n"," yhat = forward(x)\n"," loss = criterion(yhat, y)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist())\n"," loss.backward()\n"," w.data = w.data - lr * w.grad.data\n"," b.data = b.data - lr * b.grad.data\n"," w.grad.data.zero_()\n"," b.grad.data.zero_()\n"," get_surface.plot_ps()"]},{"cell_type":"markdown","metadata":{},"source":["Run 10 epochs of stochastic gradient descent: bug data space is 1 iteration ahead of parameter space. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Run train_model_SGD(iter) with 10 iterations\n","\n","train_model_SGD(10)"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Mini Batch Gradient Descent: Batch Size Equals 5

\n"]},{"cell_type":"markdown","metadata":{},"source":["Create a plot_error_surfaces object to visualize the data space and the parameter space during training:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create a plot_error_surfaces object.\n","\n","get_surface = plot_error_surfaces(15, 13, X, Y, 30, go = False)"]},{"cell_type":"markdown","metadata":{},"source":["Create Data object and create a Dataloader object where the batch size equals 5:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create DataLoader object and Data object\n","\n","dataset = Data()\n","trainloader = DataLoader(dataset = dataset, batch_size = 5)"]},{"cell_type":"markdown","metadata":{},"source":["Define train_model_Mini5 function to train the model.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define train_model_Mini5 function\n","\n","w = torch.tensor(-15.0, requires_grad = True)\n","b = torch.tensor(-10.0, requires_grad = True)\n","LOSS_MINI5 = []\n","lr = 0.1\n","\n","def train_model_Mini5(epochs):\n"," for epoch in range(epochs):\n"," Yhat = forward(X)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), criterion(Yhat, Y).tolist())\n"," get_surface.plot_ps()\n"," LOSS_MINI5.append(criterion(forward(X), Y).tolist())\n"," for x, y in trainloader:\n"," yhat = forward(x)\n"," loss = criterion(yhat, y)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist())\n"," loss.backward()\n"," w.data = w.data - lr * w.grad.data\n"," b.data = b.data - lr * b.grad.data\n"," w.grad.data.zero_()\n"," b.grad.data.zero_()"]},{"cell_type":"markdown","metadata":{},"source":["Run 10 epochs of mini-batch gradient descent: bug data space is 1 iteration ahead of parameter space. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Run train_model_Mini5 with 10 iterations.\n","\n","train_model_Mini5(10)"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Mini Batch Gradient Descent: Batch Size Equals 10

\n"]},{"cell_type":"markdown","metadata":{},"source":["Create a plot_error_surfaces object to visualize the data space and the parameter space during training:\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create a plot_error_surfaces object.\n","\n","get_surface = plot_error_surfaces(15, 13, X, Y, 30, go = False)"]},{"cell_type":"markdown","metadata":{},"source":["Create Data object and create a Dataloader object batch size equals 10\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Create DataLoader object\n","\n","dataset = Data()\n","trainloader = DataLoader(dataset = dataset, batch_size = 10)"]},{"cell_type":"markdown","metadata":{},"source":["Define train_model_Mini10 function for training the model.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Define train_model_Mini5 function\n","\n","w = torch.tensor(-15.0, requires_grad = True)\n","b = torch.tensor(-10.0, requires_grad = True)\n","LOSS_MINI10 = []\n","lr = 0.1\n","\n","def train_model_Mini10(epochs):\n"," for epoch in range(epochs):\n"," Yhat = forward(X)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), criterion(Yhat, Y).tolist())\n"," get_surface.plot_ps()\n"," LOSS_MINI10.append(criterion(forward(X),Y).tolist())\n"," for x, y in trainloader:\n"," yhat = forward(x)\n"," loss = criterion(yhat, y)\n"," get_surface.set_para_loss(w.data.tolist(), b.data.tolist(), loss.tolist())\n"," loss.backward()\n"," w.data = w.data - lr * w.grad.data\n"," b.data = b.data - lr * b.grad.data\n"," w.grad.data.zero_()\n"," b.grad.data.zero_()"]},{"cell_type":"markdown","metadata":{},"source":["Run 10 epochs of mini-batch gradient descent: bug data space is 1 iteration ahead of parameter space. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Run train_model_Mini5 with 10 iterations.\n","\n","train_model_Mini10(10)"]},{"cell_type":"markdown","metadata":{},"source":["Plot the loss for each epoch: \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Plot out the LOSS for each method\n","\n","plt.plot(LOSS_BGD,label = \"Batch Gradient Descent\")\n","plt.plot(LOSS_SGD,label = \"Stochastic Gradient Descent\")\n","plt.plot(LOSS_MINI5,label = \"Mini-Batch Gradient Descent, Batch size: 5\")\n","plt.plot(LOSS_MINI10,label = \"Mini-Batch Gradient Descent, Batch size: 10\")\n","plt.legend()"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

Practice

\n"]},{"cell_type":"markdown","metadata":{},"source":["Perform mini batch gradient descent with a batch size of 20. Store the total loss for each epoch in the list LOSS20. \n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Practice: Perform mini batch gradient descent with a batch size of 20.\n","\n","dataset = Data()"]},{"cell_type":"markdown","metadata":{},"source":["Double-click here for the solution.\n","\n","\n"]},{"cell_type":"markdown","metadata":{},"source":["Plot a graph that shows the LOSS results for all the methods.\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# Practice: Plot a graph to show all the LOSS functions\n","\n","# Type your code here"]},{"cell_type":"markdown","metadata":{},"source":["Double-click here for the solution.\n","\n","\n"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["\n"]},{"cell_type":"markdown","metadata":{},"source":["

About the Authors:

\n","\n","Joseph Santarcangelo has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD. \n"]},{"cell_type":"markdown","metadata":{},"source":["Other contributors: Michelle Carey, Mavis Zhou\n"]},{"cell_type":"markdown","metadata":{},"source":["## Change Log\n","\n","| Date (YYYY-MM-DD) | Version | Changed By | Change Description |\n","| ----------------- | ------- | ---------- | ----------------------------------------------------------- |\n","| 2020-09-23 | 2.0 | Shubham | Migrated Lab to Markdown and added to course repo in GitLab |\n"]},{"cell_type":"markdown","metadata":{},"source":["
\n"]},{"cell_type":"markdown","metadata":{},"source":["##

© IBM Corporation 2020. All rights reserved.

\n"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":2}