This is the first in a series of blog posts about big language models and CableLabs’ efforts to apply them to transform knowledge-based work across the cable industry.
What happens if I ask questions related to the ChatGPT cable? To cite an example, we asked an AI chatbot to describe neighborhood misalignment and its symptoms. Here’s ChatGPT’s response:
Neighbor misalignment, also known as neighbor mismatch, is a common problem in network routing. This occurs when two routers have different views about the adjacency of their network. In other words, one router thinks it has a direct connection to another router, but the second router doesn’t have the same understanding. This can happen due to various reasons such as incorrect configuration, network congestion, or hardware failure.
(ChatGPT then describes the symptoms of this problem.)
If you’re not familiar with neighborhood misalignment, the ChatGPT definition sounds pretty convincing! If you’re familiar with neighborhood misalignment, you’ll probably notice right away that the answer is completely wrong. Misalignment is actually a weakness in the radio frequency (RF). ChatGPT’s answer wasn’t even at the correct level of the network stack.
ChatGPT and other tools of its kind are amazing for what they’ve set out to achieve across industries and use cases, but answers like these are not helpful at all. CableLabs is actively working to solve this problem to help bring unprecedented advances in AI to the cable industry. Let’s see how.
Why is ChatGPT so definitely wrong?
ChatGPT and other generative AI products are built on revolutionary machine learning (ML) technology known as large language models, or LLMs. The ability to talk naturally to a computer was science fiction just a few years ago, but MBAs have made a human-AI conversation easier than getting into a Google search.
Unfortunately, LLM programs often fail when it comes to technical, domain-specific questions like the example above. Unless you already know the answer to your question, it becomes difficult to validate the LLM’s answer.
At their core, LLM programs are like highly sophisticated “next word prediction” machines. When you feed LLM a text sequence, the text is broken up into small chunks called “tokens,” which LLM can understand. The LLM then processes to guess the most likely token that will come after the input sequence. The method may seem simple, but when text is repeatedly added to the end of the sequence, you get a chatbot!
The main reason LLMs act so confidently when they make a mistake is that their improved “next word prediction” makes them experts at appearing convincing – but not at recognizing truths and lies. If they don’t “know” the answer to a question, they are still trying to seem legitimate, and this often results in making something up. The AI community has called this behavior “hallucination,” because LLM acts as if his made-up writings are real. In fact, hallucinations are an extension of the generative nature of LLM. They are simply not rooted in reality.
Great opportunity at hand
If we can solve the hallucination problem—that is, build an AI that can provide accurate answers to cable-related questions—we have the potential to revolutionize all types of cognitive work across the industry, including field operations, customer experience, and innovation.
The industry has decades of technical, operational and business knowledge waiting to be leveraged to help enable more seamless connectivity and network platform evolution, as well as pervasive intelligence, security and privacy. These key themes form the foundation of CableLabs’ technology vision.
That’s why CableLabs has put a lot of effort into building our solution to the knowledge accuracy problem that LLM students face.
Solution: CableLabs Expert LLM
CableLabs’ Expert LLM is the first chat app of its kind that answers questions based on a layer of domain expertise consisting of hand-curated authoritative sources on cable technologies. If we asked CableLabs Expert LLM the same question from the beginning of this article, you would get an answer like this:
Misalignment is an impairment of the radio frequency that can be identified as a difference in channel power between adjacent channels or groups of channels. They can be observed multiple times at different points in the spectrum and can be attributed to RF mixing, source issues at the header, the hub, or at the node using broadcast overlay/narrowcast or split band configuration when the two spectrums are not properly combined.
When adjacent misalignment is observed, low power channels may indicate poor performance, which is seen as a poor modulation error ratio (MER) when the delta between channels is large. This condition can manifest as lost packets, video aliasing, freezing, or in extreme cases, black screens in the customer’s home. Because misalignment is introduced so early in the downstream signal pathway, it has the potential to affect a large number of clients.
sources:
SCTE 280 CM-SP-PHYv4.0-I06-221019
Much better!
As you can see, CableLabs Expert cites his sources. Currently, the system has access to DOCSIS 3.0, 3.1, and 4.0 specifications, as well as select SCTE documents including 280 and 40. We will soon expand support to other major sources of information about cable broadband technologies.
The application supports all the latest LLM software, including GPT, Claude, Mistral, Llama series and many more. When a new useful model emerges, the application can be extended in minutes to support the model in minutes without the need for expensive fine-tuning or training from scratch.
The capabilities of CableLabs Expert LLM are mainly due to the powerful technology known as Retrieval Augmented Generation (RAG). In short, RAG is like giving an LLM an open book exam. When a user asks a question, the words are converted into a digital representation known as vector embeddings, and these representations then help us automatically select excerpts from CableLabs specifications and SCTE standards that are most likely to contain the user’s answer. LLM is given these excerpts as context to provide an accurate, fact-based answer. Additionally, RAG can run on cheap, low-end hardware instead of alternative methods such as fine-tuning, which requires GPUs to complete in a timely manner.
In addition to the chat interface, CableLabs Expert provides a comprehensive validation dataset and a frame of reference to automatically evaluate models against a wide range of known questions and answers from a variety of sources. Model evaluation is an important part of this process: we must be able to accurately understand how well our system performs, especially when comparing specific methods, datasets, or models.
Building for the future
Generative AI is here to stay. ChatGPT has captured the imagination of people around the world, in all business sectors and walks of life. Everyone agrees that it is a destructive force, but the real question is who will disrupt and who will be disrupted. At CableLabs, we’re building a better future for the broadband industry using cutting-edge AI technologies.
Foundational discussions are now underway between CableLabs and our members to bring the industry together to innovate AI and exchange standards.
Stay tuned for future blog posts on Generative AI for Network Operations, where we will take a closer look under the hood of the CableLabs Expert LLM! Next time, we will explore evaluating and analyzing the expert’s writings.
If you want to learn all about CableLabs’ work with LLMs and RAGs, check out our white paper, “The Conversational Web: AI-Powered Language Models for Smarter Cable Operations,” presented at TechExpo 2024.
