RLHF and its limits

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below. ​

1．What is the main goal of using RLHF with large language models? 

2．Describe the job of the two "coaches" involved in the RLHF process. 

3．In practice, it's hard for humans to give consistent scalar feedback (e.g. rate this text from 1 to 10). Instead, how do we collect feedback from humans? How do we then turn this into a scalar number? 

4．In the GPT-2 case study, what went wrong that caused the model to produce "maximally bad output"?​Continuing the general RLHF questions, but reviewing **[this diagram from AWS](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2023/08/31/ML-14874_image001.jpg)**. 

5．What is happening in step 1? 

6．Other than human demonstrations purpose-written for fine-tuning AI models, what other data might models use for fine-tuning in step 1? 

7．What is happening in step 2? Be sure to explain this in detail. 

8．Finally, what is happening in step 3? 

9．Why don't we just do step 1, without bothering with steps 2 and 3? 

10．Why don't we just do step 2 and step 3, without bothering with step 1?​

11．Summarise an open problem of RLHF in your own words. 

12．Summarise a fundamental problem of RLHF in your own words.

You can check your answers against **[the answer key](https://docs.google.com/document/d/1yz3yfUowbYoOcEuMAc2T0xebkNFlBj8xBFLjgX7TsBA/edit?tab=t.0)**.


Comprehension questions

Starting from a base LLM, explain how to train it using RLHF to be a helpful assistant that answers following all the rules in **[Wikipedia’s Manual of Style](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style)**.​

Write about 400-800 words. Your answer should cover:
- Using supervised fine-tuning with augmented data or human demonstrations
- Collecting feedback from humans (assume you have access to 100 expert Wikipedia editors who know all the rules)
- Using this feedback to influence the model outputs
- If the same mistake was made in RLHF while training an assistant to follow all the rules in **[Wikipedia’s Manual of Style](https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style)**, how do you think the model would behave?



Explaining RLHF in your own words

Some models, such as Llama 2 or Gemma, are available both as the base model (the weights and biases before RLHF) and the instruction-following model (the weights and biases after RLHF - sometimes called the 'instruct' model or the 'chat' model). 

You can download and run these models on your computer, or in a Google Colab notebook to see the effect that RLHF has on the model. 

You'll likely notice that base models are very tricky to get to do what you want, because their only focus is completing text.​

**Running on your computer**
You can run both models on your computer. To do this:
- Install **[Jan](https://jan.ai/)**
- Download the Gemma 2B **[base model](https://huggingface.co/MaziyarPanahi/gemma-2b-GGUF/blob/main/gemma-2b.Q4_K_M.gguf)** and **[instruction-following model](https://huggingface.co/MaziyarPanahi/gemma-2b-it-GGUF/blob/main/gemma-2b-it.Q4_K_M.gguf)** in GGUF format. This contains the weights and biases in the network, plus some metadata that explains how to run the model.
- In Jan, open the 'Hub'. Click 'Import Model' and select the GGUF files you downloaded.
- Switch back to the chat interface, then select the model in the dropdown on the right.
- Now try talking to the model, and try to get it to answer questions like 'What are good things to do in London?' and 'What's your job?'. You might find it interesting to change the prompt template in 'model parameters'.​**Running in a Google Colab**
- You can see **[this notebook](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/pytorch_gemma.ipynb)** for an example of running the instruction-following ("it") model. Use this as a starting point to get the base model running too. (if you've got a good Colab notebook that shows running both, **[let us know](https://bluedot.org/contact)** and we'll link to it from here)



Technical AI Safety

RLHF and its limits

Resources (15 mins)

Exercises