I received an interesting newsletter by email recently and I assumed it was from the Risk & Assurance Group (RAG). The subject read:
Everybody Likes RAG
I was not sure about that. Think of the dodgy conference organizers and shady characters who lead leeching “assurance” corporations like Kelni/GVG. They have all been exposed by Commsrisk editor Eric Priezkalns (also CEO of RAG). I am sure whenever they read about themselves at Commsrisk, they inwardly curse Eric, Commsrisk and RAG — I just don’t know in which order. That is the smaller issue. The bigger issue is: why, pray, would anybody want to be liked by everybody? That is a very futile existence! I also wondered why Eric would allow such cheesy marketing. It seemed very bizarre. Then the newsletter made another bold claim.
Unless you’ve been living under a rock, you are sure to have heard of RAG.
That I agree with. However, which RAG are we talking about? The Risk & Assurance Group? The very much (mis)used Red, Amber, Green lights system prevalent in a lot of dashboards? Just a few weeks ago, I had crawled from under my rock to discover there is even RAG and BRAG in project management.
- B is for Blue: Project has closed
- R is for Red: Project is likely to be delivered late/over budget
- A is for Amber: Project has missed some targets but overall end date/budget is not at risk
- G is for Green: Project is on track
Who would’ve thunk?
The email also had a very disturbing gif of a naked potbellied man lying on a bed. The man was saying, “I wash myself with a rag on a stick”. The relevance of the graphic was unclear to me. Truth be told, if I see a potbellied man, I immediately think of Kenyan politicians. I think whenever those cretins tell us this or that money cannot be accounted for, the said funds somehow can be found in their own bellies. The proof is in the girth. As such, gifs of naked potbellied men are very triggering for me.
To cut the long story short, the newsletter (from Analytics India magazine) was talking about Retrieval Augmented Generation. According to this IBM research site, Retrieval Augmented Generation is…
…an AI framework for retrieving facts from an external knowledge base to ground large language models (LLMs) on the most accurate, up-to-date information and to give users insight into LLMs’ generative process.
If you have suffered an involuntary eye roll, I fully understand. Never have so many words been used to give so little meaning and confound so many people. However, the same writeup does a good job of explaining, if you keep on reading.
Large language models can be inconsistent. Sometimes they nail the answer to questions, other times they regurgitate random facts from their training data. If they occasionally sound like they have no idea what they’re saying, it’s because they don’t. LLMs know how words relate statistically, but not what they mean.
Now we are getting somewhere.
Retrieval-augmented generation (RAG) is an AI framework for improving the quality of LLM-generated responses by grounding the model on external sources of knowledge to supplement the LLM’s internal representation of information. Implementing RAG in an LLM-based question answering system has two main benefits: It ensures that the model has access to the most current, reliable facts, and that users have access to the model’s sources, ensuring that its claims can be checked for accuracy and ultimately trusted.
That sounds like what we spend our time doing in revenue assurance, minus the machines. Sometimes we sit in meetings where everybody has their own alternative reality. We are trusted to be a bit removed from the stage. We are expected to use data and process reviews to give a version of truth that the bosses can (often begrudgingly) accept. To do this, we must fact-check and in so doing, build trust. We don’t always do a good job.
In the interests of brevity, this piece will not delve into AI usage in revenue assurance and fraud management. Neither do I wish to go into the question of whether the external sources of knowledge need to be subjected to more iterations of RAG to be able to serve their purpose of grounding the LLMS — I detest loops. From my informal conversations with folks in the industry, suffice it to say that bold claims are being made by people in a lot of companies. Sometimes the value can be seen. Sometimes it cannot. Let us ground ourselves in the mission, adopt technology where we can but resist the temptation of being in “it” because “it” is the talk of the town.
Jokes aside, the newsletter was a very concise communication about the dangers of RAG. It explained vulnerabilities such as those found here and here, then concluded:
[It is] important for you to know the limitations and have an understanding of some of these tools and APIs to build your LLM applications.
Back to the original definition, I really do like the term “grounding”. For some reason, my teenage daughters don’t, and I wonder why.
Maybe I should employ LLM with RAG to get to the answer. I have a large dataset of experiences I have gone through, with my offspring. AI might yet prove its utility in my household.
