Khoa Duong dnanhkhoa

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

Brew Bundle Brewfile Tips

Copyright & License

Unless otherwise noted (either in this file or in a file's copyright section) the contents of this gist are Copyright ©️2020 by Christopher Allen, and are shared under spdx:Creative Commons Attribution Share Alike 4.0 International (CC-BY-SA-4.) open-source license.

Sponsor

If you more tips and advice like these, you can become a monthly patron on my GitHub Sponsor Page for as little as $5 a month; and your contributions will be multipled, as GitHub is matching the first $5,000! This gist is all about Homebrew, so if you like it you can support it by donating to them or becoming one of their Github Sponsors.

Awesome Knowledge Graph Embedding Approaches

This list contains repositories of libraries and approaches for knowledge graph embeddings, which are vector representations of entities and relations in a multi-relational directed labelled graph. Licensed under CC0.

Libraries

🆕 AmpliGraph (4 algorithms) @ https://github.com/Accenture/AmpliGraph
Embedding framework (5 algorithms) @ https://github.com/BookmanHan/Embedding

	# Copyright (c) 2019-present, Thomas Wolf.
	# All rights reserved. This source code is licensed under the MIT-style license.
	""" A very small and self-contained gist to train a GPT-2 transformer model on wikitext-103 """
	import os
	from collections import namedtuple
	from tqdm import tqdm
	import torch
	import torch.nn as nn
	from torch.utils.data import DataLoader
	from ignite.engine import Engine, Events

	def top_k_top_p_filtering(logits, top_k=0, top_p=0.0, filter_value=-float('Inf')):
	""" Filter a distribution of logits using top-k and/or nucleus (top-p) filtering
	Args:
	logits: logits distribution shape (vocabulary size)
	top_k >0: keep only top k tokens with highest probability (top-k filtering).
	top_p >0.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering).
	Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751)
	"""
	assert logits.dim() == 1 # batch size 1 for now - could be updated for more but the code would be less clear
	top_k = min(top_k, logits.size(-1)) # Safety check

	{
	"python": {
	"library": "import json\ndef black_reformat(cell_text):\n import black\n import re\n cell_text = re.sub('^%', '#%#', cell_text, flags=re.M)\n try:\n reformated_text = black.format_str(cell_text, 88)\n except TypeError:\n reformated_text = black.format_str(cell_text, mode=black.FileMode(line_length=88))\n return re.sub('^#%#', '%', reformated_text, flags=re.M)",
	"prefix": "print(json.dumps(black_reformat(u",
	"postfix": ")))"
	},
	"r": {
	"library": "library(formatR)\nlibrary(jsonlite)",
	"prefix": "cat(toJSON(paste(tidy_source(text=",
	"postfix": ", output=FALSE)[['text.tidy']], collapse='\n')))"

	import numpy as np
	import torch
	import keras

	def pyt_to_keras(pytorch_model, keras_model):
	"""
	Given a PyTorch model, this method transfers the weight to
	a Keras Model (with backend TensorFlow) with the same architecture.
	Assumptions:
	1. The corresponding layer names in both the models will be the same

	#include <opencv\cv.h>
	#include <opencv\highgui.h>
	#include <iostream>
	#include <string>
	#include <cmath>

	using namespace std;

	// display video from array of frames, using left and right button.
	void displayVideo(cv::Mat* frames, cv::Point2i* centralPoints, int numFrame, int FPS, string windowName) {