Enterprise knowledge graphs as a substrate to help minimize LLM hallucinations in Data Analytics can’t work. If you are into software development and use loop engineering/graph engineering, this post isn’t for you.
The LLM naysaying never stops.
'They will become the tech bro mob ruling class', ‘LLM outputs have always been unreliable by default.', ‘The age of harness, Jev and GenAI bros sudden discovery and pivot to ML.','We are back to square Zero, after wasting trillions of $$ and a decade of time...'
The same set of characters, every day, in my professional-social network feed (LinkedIn, if you didn’t catch my drift), always naysaying LLMs. At first I chalked them up to sour grapes, folks who couldn’t get funding for their own cool little ML project. But their perseverance got me thinking: are they pure sour grapes, or do they have a viable alternative, or is it just altruism? So I dug deeper, and most, if not all, of them are selling some form of knowledge-engineering product or an ontology-driven solution to the unreliable LLM output.
Context engineering via knowledge graphs and ontologies has been one of the hot topics of 2026. Enough hype that I spent a good chunk of the year working with two different graph companies: one classic graph provider and another neo-modern one meant for the AI world, and I have seen job titles like ‘Cognitive AI Architect”, “Enterprise Knowledge Engineer,” etc. pop up.
My conclusion is that enterprise knowledge graphs can’t help solve AI hallucinations in enterprise data. Not because the technology doesn’t work; it does, brilliantly, much better than using naive RAG or some other variation that turbopuffer or LlamaIndex is selling you. Even knowledge retrieval speed can be solved with a fusion technique combining semantic retrieval + Graph retrieval with full-text search, plus our good old and trusty caching. Here, enterprise means a single source of truth: one knowledge graph or ontology for the entire enterprise. However, if we look at our history, circa 2015, the Big Data era, when Knowledge Graphs and Ontologies were hot, and products like Databricks’ GraphX and GraphFrames and Apache Giraph, etc were emerging. But they didn’t take off, and I predict Knowledge Graphs and Ontologies won’t gain traction this time around either, for the following reasons.
Ontology building is a hard problem. It takes a centralized committee and a few hard-to-find experts to build the first version. An even harder task is keeping the ontology updated: the approval and vetting process required to keep it current. A committee needs to approve every small change before it can be merged into production, and it has to be a committee because no single entity, person, or department can be responsible for the entire living, breathing, ever-changing enterprise. The human element makes it far too slow to match the speed of AI decision-making. Which means your AI agents still hallucinate or make decisions on stale data, making the knowledge graph an oxymoron.
Yes, AI agents could solve ontology-updating speed. But two questions arise: how many autonomous enterprise-wide AI agents have we seen in production or otherwise, and do we really trust the LLM-based AI agents to retrieve and update our knowledge graph, especially when their hallucinations are what we are trying to solve in the first place? Establishing enough guardrails to make sure agents don’t turn the knowledge graph into a data swamp isn’t easy either. I have seen a few players like Databricks’ Genie Ontology, which tries to vote in the most popular metric definition: “Using an approach similar to PageRank, Genie Ontology weighs where a definition came from, the relative authority of that source’s author, how often people rely on it.” However, a wrong KPI doesn’t become right just because 8 out of 10 users like it.
Cost centers for knowledge graphs don’t exist within the enterprise, and they couldn’t be established during the Big Data Ontology era either. This is one reason the ontology and knowledge graph wave faded, despite having its moment a decade ago. Business pays for Business Intelligence and Data Warehousing because they get a tangible report at their desk every day, and it’s explainable enough for business to question the report, the underlying data, and the process.
In my time building and selling Knowledge Graph based products in the AI era, I didn’t find selling the concept difficult at all, and a Naive RAG vs. a Graph RAG demo or POC fetches an easy enough technical win in a carefully controlled environment with a pristine Ontology built on clean data. What I found far more difficult was finding the Economic Buyer, the person or department within the enterprise who would own this and pay for this. I had an even harder time prospecting for (not discounting the fact that I am possibly the worst SDR there is) the right team to reach because everyone could benefit from an enterprise knowledge graph, but no one was willing to own, build, maintain, and pay for one.Zero Trust in LLM output. LLMs and agentic systems are not easily explainable, and there’s enough doom around them for hallucinating the wrong metric that there is zero trust to begin with. A Knowledge graph does build trust, and graph traversal can make LLM output more explainable. However, my concern is about using LLMs to hydrate our knowledge graph and keep it updated. Central IT and business communication is super disconnected and asynchronous. Now, taking on the task of building an ontology by central IT would not only be difficult but perilous because of the zero-trust beginnings, and central IT and business would need to be joined at the hips to build and maintain a knowledge graph, and BUs don’t have that kind of time and luxury while they are busy trying to gauge and react to market conditions.
One might say there are too many AI naysayers on LinkedIn (substitute with your favorite professional social network) these days. Let’s assume ontologies and knowledge graphs can’t work; what alternative am I offering? Well, I found that a set of simple markdown files documenting my project and business requirements goes a long way compared to building a full-blown ontology. One might call this a lightweight ontology, one that's decentralized, puts the ownership into the team building it, solves slow governance, and is a part of the owning team's budget, while still solving the core problem of unreliable LLM output. This circumvents or avoids the single-source-of-truth question altogether; however, in the real world, metrics and KPIs don't have a single version of truth. Sales and Revenue have slightly different definitions for marketing vs. sales departments.
There's a whole class of semantic layers being pushed by companies like Fivetran and Snowflake and the Open Semantic Interchange, but I don’t know much about that work, as I am not an AI researcher but a lowly solutions engineer trying to find a practical solution for a practical problem. If you are interested in what I am proposing here, you can browse my series on “AI-Assisted Modern Data Analytics” (apologies for the shameless self-promotion, but isn’t that what professional social networks are for). In the next installment, I show you how I build a full end-to-end data pipeline from a single prompt and a long-horizon agentic job. Keep watching this space.