Today's Progress: Understood the basics of Neural Network and how to build ANN. Also practiced Python on Hackerrank.
Thoughts: Cleared up my concepts on ANN in which I had earlier found confusing like Activation and Cost functions, Batch and Stochastic Gradient Descent and Backpropagation.
Links to work:
Today's Progress: Learnt about Keras library. Installed it and learnt the theory behind how to implement ANN using Keras. Practiced Statistics on Hackerrank.
Thoughts: Seeing how to implement the concept learnt as a theory yesterday cleared the concepts much more. I will implement the code for the same tomorrow.
Links to work:
Today's Progress: Implemented and practiced ANN network using Keras Library for Churn Prediction
Thoughts: After implementing the ANN, I tried several combinations of parameters by varing the activation functions, number of nodes in hidden layers and the number of hidden layers. Changing these parameters had an effect on the Accuracy and the best found combinations were relu function with 5 hidden layers having 6 nodes each and also elu activation function with 2 hidden layers having 6 nodes each. Both the combinations gave the accuracy of around 86%.
Link to work:
Today's Progress: Studied about how to improve and tune the model.
Thoughts: Studied about K-folds cross validation and Bias-variance tradeoff. Also learnt how to improve the accuracy of the models with the help of hyper-parameter tuning using Grid search method. Making use of Keras and sklearn both together for k-folds cross validation using wrapper was something interesting to learn.
Today's Progress: Studied about basics of Convolutional Neural Networks
Thoughts: Learnt about how an image is translated into a language which machine can understand. Starting from raw 2D image to making it into a 1D array of values, still retaining the spatial co-relations of the original image was great to understand. It involved processes like Convolution (Applying filters), pooling (Retaining the most relevant features), flattening (Converting in 1D array of values) and full connection (fitting the convolved layer inside ANN). Watched the 2D visualization of all the above process from the following link.
Today's Progress: Studied about implementation of CNN using Keras Library
Thoughts: Working with images was something new. I had to learn right from how to import the dataset, since it was no more a CSV file. Then learnt how to implement and complile CNN using convolution, pooling, flattening and full connection. One more interesting thing learnt today is Image Augmentation and how it can be used to prevent overfitting. The code is taking too much time to train unlike all the models run till now.
Today's Progress: Watched videos to learn about basics of RNN and LSTM
Thoughts: Re-current Neural Network basically mimics the short term memory of the human brain. Using RNN is important when the network needs to remember something from the recent past to perform operations in the future. Common applications of RNN lies in Natural Language Processing because it requires remembering the context and the subjects in order to process the information. I learnt about the problem of vanishing gradient and how if the weights are small, the network fails to train the network as the gradient goes on diminishing and eventually backpropagation fails. This problem can be solved with LSTM (Long Short Term Memory). It was interesting to watch a movie which was written by an algorithm which was trained using RNN, making use of LSTM. The link to the video is given below.
Today's Progress: Started off with building RNN
Thoughts: Started with implementation of RNN for predicting Google Stock price. The model is not complete yet. Shall complete it tomorrow.
Today's Progress: Implemented RNN network
Thoughts: Implemented RNN for predicting Google Stock price.
Link to work:
Today's Progress: Started with Unsupervised Deep learing and Learnt about the Self Organizing Maps.
Thoughts: SOM are basically used for simplified visualization of results involving many output variables. It is closely related to dimensionality reduction. It was interesting to see how the BMU (Best Matching Unit) pulls the nodes around it towards itself to form clusters of inputs sharing similar features. Also revised the topic of K-Mean clustering.
Today's Progress: Started implementation of SOM using minisom
Thoughts: Started implementation of SOM using minisom library for probable bank frauds.
Today's Progress: Completed implementation of SOM
Thoughts: Implemented and visualized the SOM for probable bank frauds (Unsupervised learning) also used this information to lable the input dataset. Then also used supervised learning (ANN) to classify the customer based on the outcome from SOM.
Today's Progress: Learnt about Boltzmann Machines
Thoughts: Boltzmann machines gets their name because they follow Boltzmann Distribution. There are Input and Hidden layers connected to each other and also the neuron within the same layer connect to each other. The connections are bi-directional. I learnt the concept of Energy based models and how the model is most stabilized when it is at its lowest energy state. I saw how Boltzmann machines can be used in Recommender Systems. Learnt about Restricted Boltzmann machine in which the neurons are not connected within the same layers.
Today's Progress: Learnt about Auto Encoders
Thoughts: Auto encoders are basically unsupervised deep networks which encodes inputs and reproduce it at the output. I learnt about overcomplete hidden layers in auto encoder. In this architechture there are more hidden nodes than that in input layer. Due to this there arises a possibility that the model will just pass on the signal through the layers and will not encode at all. The method to overcome this are sparse and denoising auto encoder.
Today's Progress: Discussed and learnt about the ways to avoid overfitting in CNN
Thoughts: My CNN model was overfitting, today I found out various ways to avaoid that.
Today's Progress: Added dropout regularization in CNN code
Thoughts: Tried to reduce overfitting by adding 0.2 dropout after full connnection layer. It improved the test set accuracy from 78% to 84%
Today's Progress: Implemented recommender system using Boltzmann machine
Thoughts: Implemented Recommender system to predict whether a particular user will like the movie. Implementation was done using PyTorch library.
Today's Progress: Implemented recommender system using Auto Encoder
Thoughts: Implemented Recommender system to predict a movie rating given by a particular user. Implementation was done using PyTorch library.
Today's Progress: Completed Google's tutorial on Image Classification
Thoughts: Implemented the basic CNN using Keras module of Tensorflow library. The model contained 3 Conv layer. This model was overfitting hence dropout regularization and image augmentation was used. Next the concept of Transfer learning was used to further increase the accuracy of the model using the Inception model.
Today's Progress: Built a CNN by implementing transfer learning using Keras
Thoughts: In order to reduce the Overfitting and to improve the accuracy, I implemented trasfer learning using the model VGG19 pre-trained on ImageNet dataset. This was done using the keras library.
Today's Progress: Studied the filters used in CNN in detail with the focus on Edge Detection
Thoughts: Filters are used in CNN to detect the features from the image. One of the features could be detecting edges in the image and I learnt about the Vertical and Horizontal edge detection filters. 3x3 Prewitt operator is a 3x3 matrix with the first column of 1s, second column of 0s and third column of -1s. To enhance this and to put more emphasis on the middle pixels of the filter, sobel filter or scharr filter is used. For visually seeing how filters detect the features, GIMP image editing platform can be used.
Today's Progress: Studied about Padding and Stride used in CNN
Thoughts: When filters are used to convolve the image, the image size reduces and if many of such operations are performed then it reduces the size of the image significantly and can degrade the image. Hence to avoid this, padding is used. Padding is basically adding rows and columns of zeros around the matrix. If the image size is nxn and the filter of size fxf is used then the output size is n-f+1. If padding of p is used on both the sides then this output size becomes n+2p-f+1. So if we want the output image to be of the same size as that of input then "Same Padding" is used. Here we need n+2p-f+1 = n. Hence in the same padding p= (f-1)/2. If f is odd then p is an integer and integer padding is generally preffered as we cannot add fraction pixels. Hence mostly the filter size is odd. e.g. 3x3 or 5x5.
Today's Progress: Learnt about 3D convolutions and pooling layer
Thoughts: 3D convilutions is also termed as convolution over volumn. This is used when the image is coloured. i.e when it has R, G, B channels. Pooling layer is used to retain the important features from the images even after reduction in size.
Today's Progress: Learnt about LeNet 5 Architecture
Thoughts: LeNet 5 was one of the early models and its purpose was to detect handwritten digits. It was trained on greyscale images and applied no padding. Also instead of Max pooling, it used Average pooling. This model followed the usual process of using the conv layers, pooling layers and then 2 fully connected layers but instead of using softmax function at the end, it used tanh or sigmoid.
Today's Progress: Learnt AlexNet and VGG16
Thoughts:
Today's Progress: Learnt about ResNet
Thoughts:
Today's Progress: Learnt about Transfer Learning
Thoughts: After learning about the Benchmark Models, I learnt how can we make use of these models for our own usecase. The models like VGG16, AlexNet, Inception are very deep models, trained on huge dataset. In these models, the earlier layers identify the less complex features like colour or texture, the middle layers identify contours and the later layers identify complex features like objects. So for our usecase we can use the earlier layers and then add some fully connected layer and softmax function after it. This way we utilize the learning from the Pre trained networks to make our models perform better. If we have very small dataset then we can freeze all the layers and train just the softmax since there is inadequate data to train the later layer. If we have substantial amount of data then we can freeze the first few layers and train the later layers and the softmax unit. And if we have a huge dataset then maybe we can think of training all the layers.
Today's Progress: Read about Neural Style Transfer
Thoughts: I was always curious about how Computer can generate Art. I was amused by the Prisma app and wanted to know the concept behind it. Today I spent time in getting the idea about Neural Style Transfer. Neural Style Transfer is an idea of imposing the style from any artwork on to any other image. I learnt that this Style Transfer actually uses transfer learning and it works on minimizing different losses.
Today's Progress: Learnt in detail about Neural Style Transfer
Thoughts: Neural style transfer is a great concept where you have two images, one style image and one content image. And what we do is we extract the style i.e. colour, texture from the style image and impose it on to the content image. For doing so, we make use of the transfer learning. We are using say VGG16, ideally this model is trained as a image classifier, but here we tweak it and use it to extract features. We feed the Style image and extract the style from the output of earlier layers of the model. Similarly we extract the shapes and objects from the content image using the output of the later layer. We then construct a stylized image by mixing the extracted features. We then compare this mixed image with the two original images and then calculate the content loss and the style loss. The style transfer then becomes the optimization task where we try to minimize these losses.
Neural Style Transfer (Useful Link)
Today's Progress: Understood and implemented the Open Source implementation of Neural Style Transfer
Thoughts: Studied the awesome Jupyter notebook by Siraj Raval which implemented Neural Style Transfer using Tensorflow. It made use of the VGG16 model.
Today's Progress: Learnt about Object Localization
Thoughts: Object Localization is figuring out where in the image the object lies. This can be done by adding more parameters to the softmax function so that the output y comprises of 1. The probability that the object exists in the given grid 2. co-ordinates of the mid-point of the object wrt to the grid in which the object is found 3. Height of the object 4. Width of the object 5. Variables equal to the number of possible objects to be detected. These are binary variables representing 1 if the respective object is predicted in the given grid. The box thus made from parameters 2,3 and 4 is called Bounding Box and this box localize the object on the image.
Today's Progress: Learnt about Landmark Detection and Object Detection
Thoughts: Landmark detection is detecting the position of specific parts of the given object. Say for example the facial features, position of eyes, nose or jaw line. This can be done by defining the landmarks first, then train a supervised learning model to learn the landmarks and then in the softmax function, the relative co-ordinates of the landmark points can be returned. For detecting object the basic method is making use of the sliding window. In this method, the model is first trained to identify the object using the closely cropped image of the object without any background. Then the test image is taken and a window i.e. a small grid is used to detect the object. This window is slid with some stride and the process is repeated. This process is computationally very expensive.
Today's Progress: Learnt about Sliding Window and Bounding Box Prediction
Thoughts:
Today's Progress: Learnt about Non-max suppression and Anchor Boxes
Thoughts: Sometimes one can have multiple bounding boxes for the same object and Non-max suppression is used to eliminate all the secondary boxes and keep just one box for one object. For doing so, it uses Intersection over Union (IoU). The Intersection between the two areas, the bounding box and the object is found. Similarly their union is found out for all the bounding boxes and only the box with highest IoU ratio is retained and the rest are suppressed. Anchor Boxes are used when one is expecting more than one objects are the same location.
Today's Progress: Learnt about YOLO Algorithm
Thoughts: YOLO is an object detection algorithm and stands for You Only Look Once. It basically combines all the concepts covered in last 4 days to make the prediction. The training set comprises of image divided into grids and the classifier output with the dimensions n x n x a x (5+c) where n x n is the grid size, a is the number of Anchor Boxes and c is the number of classes. The next step is making the prediction, where we get output vector y for each grid of the image predicting if the object is detected in that grid and it gives the bounding boxes. Next we eliminate the boxes on grids returning lower probability of object being detected. We also use the Non-max suppression to eliminate the extra bounding boxes.
Today's Progress: Learnt about One Shot Learning and Siamese Network
Thoughts: CNN does a good job when it has a lot of training images to learn from, but in real usecase the network might have to learn from just one image. Say for example, attendace system using facial recognition. This system will have just one image of every student feed into it and have to identify students based on that. This is called one shot learning and it cannot be achieved through traditional CNN, because of the training set constraint and also because if any new student is added then the entired system needs to be retrained. To solve one shot learning, Siamese networks are used. In siamese network, a network similar to CNN but without the sigmoid function is used to make encoding for every image. Then the difference between these encodings is found out, if the difference is very less then the images have a high degree of similarity and hence treated of the same class and vice versa.
Today's Progress: Learnt about Triplet Loss and Face Verification
Thoughts:
Today's Progress: Implemented basic Face Detection
Thoughts: Today I started learning OpenCV. There are some good tutorials and I started with Face Detection using Haar Cascade.
Face Detection using Haar Cascades
Today's Progress: Implemented Color Detection
Thoughts: Implemented colour detection by changing the colour space to HSV and using masking. The code was implemented to detect blue colour and extended to detect blue and green colour in one frame.
Today's Progress: Implemented OCR of Digit using KNN and OpenCV
Thoughts: Optical Character Recognition was done using K-Nearest Neighbours algorithm and OpenCV.
Today's Progress: Implemented Object Detection in Video
Thoughts: Implemented Object detection in Video using Meanshift algorithm.
Today's Progress: Studied about the Capsule Theory
Thoughts:
Nice work @abhinishetye !!