User talk:Kjerish

Image

Hi Kjerish. I noticed the image you added. I think an image explaining the training steps of GPT models can indeed be very useful. The diagram looks accurate. One suggestion however, I propose to include the image in the articles ChatGPT and Generative pre-trained transformer instead of Fine-tuning (deep learning) and Reasoning language model, because fine-tuning is broad and includes many techniques, and the image does not illustrate the core innovation of reasoning language models. Also, I'm not sure whether it would be an improvement or not, and it's ok if you don't want to change, but have considered making the image vertical? (the horizontal format may not be ideal for readability on phone screens) Alenoach (talk) 18:36, 27 July 2025 (UTC)[reply]

@Alenoach: Hi Alenoach, thanks for reaching out. I took inspiration from the diagram made by OpenAI for InstructGPT, released ~10 months before ChatGPT. I could see the argument you're making in terms of scope. I tried to qualify it in a way that was clear that it's referring to a specific thing but I think in the case of the Reasoning language model article I should just make a new diagram. I'll give the vertical layout a shot too. Once I do that I'll add it to the ChatGPT and Generative pre-trained transformer articles. I am not overly attached to the way that it is, just thought that it should exist in some form. I uploaded the TikZ code in case anyone wanted to change it but I have some time right now to work on it – Kjerish (talk) 21:10, 27 July 2025 (UTC)[reply]

Thanks! For the vertical layout, might be worth getting some more feedback from other people, I'm not really confident that vertical is better. Perhaps adjusting the font or font size could be another way to make it easier to read. I made an edit to the article on LLMs btw. Alenoach (talk) 21:43, 27 July 2025 (UTC)[reply]

@Alenoach: What do you think of it now? I added 4 changes: vertical layout, examples, order of magnitude, legend. Might be too much at once but the other 3 aspects are probably things that would have been added over time anyway – Kjerish (talk) 23:17, 27 July 2025 (UTC)[reply]

I suggest to undo and upload the new (vertical) version in a new Wikimedia Commons page, in case someone wants to use the old version (also because the current image parameters on Wikipedia would need to be adjusted for the new image to not be oversized). Alenoach (talk) 23:42, 27 July 2025 (UTC)[reply]

I undid the upload as a temporary solution to avoid oversized images on Wikipedia articles, I hope that's ok to you. Alenoach (talk) 00:20, 28 July 2025 (UTC)[reply]

For the blue boxes, it seems well-researched but I worry that these orders of magnitude would be outdated or may not be accurate for some models. Maybe also a few aspects could be adjusted to be a little easier to understand for people who discover the topic, for example "KL" may be confusing to many readers. Otherwise, the design is clean and the yellow explanations are insightful. Alenoach (talk) 23:56, 27 July 2025 (UTC)[reply]

@Alenoach: Updated – Kjerish (talk) 03:03, 28 July 2025 (UTC)[reply]

It looks good!

A few minor suggestions: "The model acquires grammar, facts, and coding patterns from raw text." may be simplified to "The model learns grammar, facts and coding from raw text." Also, the reinforcement learning step of stage 3, people may not understand that the goal is to train a reward model to predict human preferences, and then train the GPT model to satisfy this reward model, thus assimilating human preferences.

Also, for your information, training the reward model doesn't have to involve the model that is being trained, in principle you could reuse a reward model that was trained using the outputs of a completely different LLM (a benefit of RLHF is that once the reward model is trained, you don't need humans to label data anymore). Also, the SFT and RL steps can be mixed (afaik companies make multiple iterations of SFT and RL). The graph still looks fine though, that's just technical details.

On phone, it looks better, but it's still a little difficult to read for me, something like slightly increasing font size could make it more comfortable to read without the need for people to zoom. I let you decide. Alenoach (talk) 04:49, 28 July 2025 (UTC)[reply]

@Alenoach: Updated caption text and made it larger. I kept the arrows in place but mentioned that the base model for the reward model might be a different origin.

I saw a cool video about reward modeling recently. We've gone from doing P(x,y) -> ELO to P(x,y,y') -> Probability y'>y (which is obvious).

But in the future, there will be an input that describes the human grading it so that we can have a single reward model that can guess how different types of people will grade a given prompt. Then we could reduce bias by having benchmarks for how different demographics of people will grade particular answers. That and maybe each user will have their own fine-tuned LLM. Pretty creepy – Kjerish (talk) 06:15, 28 July 2025 (UTC)[reply]

That's an interesting development; hard to tell whether it will turn out to be socially good or detrimental for chatbots to be individually customized.

I propose to now move the image from the articles Fine-tuning (deep learning) and Reasoning language model to ChatGPT and Generative pre-trained transformer.

These articles make a lot of views by the way, so that should be really useful. Thanks! Alenoach (talk) 04:00, 29 July 2025 (UTC)[reply]

@Alenoach: Done. The ChatGPT one is a great idea. Hopefully more people will see it and improve up on it. Thank you for your suggestions! – Kjerish (talk) 04:20, 29 July 2025 (UTC)[reply]