Conversational RAG: Follow-Ups and Memory

The Follow-Up Problem

"What about pricing?" means little alone. The system must fold the conversation history into the retrieval query.

Rewrite Before Retrieving

history = ["What is SQL?", "Show me example join queries."]
follow_up = "And in Groovy?"
rewritten = f"{follow_up} (context: {history[-1]})"
hits = collection.query(query_texts=[rewritten], n_results=3)

A better approach: ask the model to rewrite the question with history as context first.

Keep Chat Memory Separate

Store the short conversation separately from the knowledge base. Only the rewritten question should query the vectors.

Handling Ambiguity

• Detect genuinely new topics and start fresh.
• Show which history the rewrite used.
• Let users reset context when it goes off track.

Key Points

• Rewriting with history fixes follow-up questions.
• Chat memory and knowledge base stay separate.
• Surface the rewritten query to the user.
• Support context resets for long sessions.