LLM Context & Memory 101

In order to understand why we manage game state and context the way we do, we should have a general sense of how language models work.

This is massively oversimplifying it, but you can think of an LLM as autocomplete on steroids. The way it works is it takes a big block of text (in the form of prompts / context) and then spits out some more text.

One misconception is that language models have memory. This is not true! Language models are stateless and forget everything as soon as they are done with their response. It’s the apps around the language models that handle the memory. So GPT-4 (LLM) does not have memory, but ChatGPT (the app) does have memory. ChatGPT gives GPT-4 the memories anytime you interact with it.

The diagram above is a simplified explanation of what it looks like when you talk to an AI game master in Friends & Fables. Meanwhile, the diagram below zooms in on the F&F Builds Context step.

A lot of our players are building MASSIVE worlds, with hundreds of locations, NPCs, and items. In addition to things from the SRD, sometimes we’re looking at over a thousand entities, each of which have paragraphs of descriptions, appearances, and other data. If we sent all of that data to a language model every time you sent a message, it would be massively expensive, and we’d be bankrupt!

Not only is it expensive, but it’s also confusing to both humans and to LLMs if there’s a lot of unrelated information to parse through for your task! It’s much easier and faster for the LLM to complete the task when it only has the relevant information it needs and doesn’t need to parse through and throw out irrelevant information.

This is why we do our best to pick the most relevant context whenever you send a message. We do this by creating relationships between entities, using character locations, `@mentions` and more.

So, what game state can ACE (Agentic Campaign Engine) actually see?

Depending on the situation, ACE will get a different set of context. For example, ACE gets different information in a narration context “I do the thing… what happens next?” vs state update context “Can you update my HP to 20?”. In the former, we’ll want to include more world information, memories, etc. In the latter, we’d want to include your character’s stats.

For most of your interactions, ACE will see:

• Basic world/theme information

• The last 5-8 messages in the chat history

• The campaign summary (memories)

• Basic character information of in-party characters, nearby NPCs, and any mentioned NPCs.

• Basic information about the current location and area

Can I see what ACE sees?

Yes! If you click on a message from the GM, click the 3 dots and then click “View Context”.

We built this feature so you can examine the memory being used for each message. If you think something should be in there that we’re not currently using, please let us know!

Why can’t you just add more context?

There are two main reasons we don’t throw your entire chat history into context:

1) More context is not always better! It can confuse the LLM

Your intuition might be that shoving more context into each request will make the LLM smarter, but that isn’t always the case. Sometimes having too much information can actually be a bad thing. Imagine you have a description on one thing that has conflicting information with another. By pulling them both into context, we’d actually be making the lore of the world more confusing to the LLM.

Our goal is to give the LLM everything that it needs to create an immersive and cohesive experience. While we’re not going to throw the kitchen sink at it, we will be adding in more context as we find reasons to give it more information. By keeping the context limited only to what is necessary, we’re creating a more immersive and intelligent experience.

2) Cost

Across all of our users, we’re handling A LOT of messages. Every word that we send to an LLM costs money. The more we send, the more it costs us. To keep Friends & Fables online, we need it to be a sustainable business. In the future, we’ll open up more subscription options for users who do want to inject more context into each message.