The problem is that often once it starts failing, it rarely recovers from an error, even given the exact error code.
In my dashboard, I added OpenRouter access, so you can quickly swap between models and see which one works best (for your desired accuracy/response time/pricing). Grok 4 Fast is one of the best overall.
Cofounder of one of those analytics agents here (https://getdot.ai).
The promise of the technology is not that it can deal with any arbitrarily complex Enterprise setup, but rather that you expose it with enough guidance on a controlled and sufficiently good data model.
Depending on your use case this can be super valuable as it enables a lot more people to use data and get relevant recommendations.
But yeah it's work to make nice roads and put up signs everywhere.
I am working with Databricks' Genies. I have a _very_ complex Enterprise data schema(s). Genies, and from what I can tell, your product work on a set of tables ~20 and expect a well thought out and documented data model.
I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn't need an AI assistant. If I had a perfect data model my team could write simple SQL queries or give chatgpt a schema dump + a natural language query and it would get the answer most of the time.
IMHO, the big value in this space will be when these tools can wrangle realistic databases.
Well, I can't comment much on Genie, but the core question is always how you scale the complexity.
In Dot, it's divide and conquer. If you have several different teams each of them has to maintain their knowledge base.
A bunch of our customer have less than 10 tables hooked up to Dot, but this data is core to their business and so the analytics agent is really useful. Our most complex setup is on more than 5000 tables, but that was a lot more work to lay out the structure and guidelines.
Also, I don't think all organization are ready for AI. If the data model is a huge mess, data quality is poor and analytics use cases are not mature, it's better to focus on the fundamentals first.
Sure, you have to phrase your question in a way that's a bit like trying to ask a very specific question of an annoying "Self-Diagnosed Internet Autistic" co-worker who can't tell the difference between being "precise" and being "a pedantic pain in the arse", but it is just text.
Oh you're upset because SQL isn't in German? Well there's no reason why you can't stick German into the lexer, set your columns up with German names, and get a query like
WAHLEN_SIE zeielen_id, benutzer_namen, eingetragen
AUS benutzern WO aktiviert = WAHR
SORTIEREN NACH registrierungs_datum;
people who can really speak German, ja ich weiss meine Deutsch is so schlect, geh schon ;-)
Well, of course you're correct that SQL is text, but that's not what the article is arguing about. The point isn't whether SQL is text... it's about the kind of text it is.
SQL is a formal language, not a natural one. It's precise, rigid, and requires a specialized understanding of schema, joins, and logic. text-to-sql systems don't exist because people are too lazy to type; they exist because most people can't fluently express analytical intent in sql syntax. They can describe what they want in natural language ("show me all active users who registerd this year"), but translating that into correct, optimized sql requires at least familiarity, and sometimes expertise
So the governance challenges discussed in the article aren't about "oh SQL is too hard to type"...they're about trust, validation, and control when you introduce an AI intermediary that converts natural lang into a query that might affect sensitive data
I think it's important to reinforce that SQL isn't an COBOL like attempt at building a querying language out of natural expressions (which you could see if you squint really hard). Instead SQL is a refinement of various querying languages (hence being the standard one) that co-evolved with relational algebra. If you have a chance to learn more within an academic environment courses on relational algebra and the abstract theory of set operations can be invaluable to building a basis for more naturally understanding the intent and tools available in SQL.
What is the human need to bash SQL at all costs? At its core, such a simple syntax, yet its so powerful at aggregating/manipulating tabular data and the like. Instead we’d rather declaratively say what we want in a more verbose/disjointed way… fascinating
I'm with you. The more SQL alternatives I've seen, the more I'm convinced that SQL doesn't have that much fat or disfunction in it's syntax. There is inherent complexity. Follow an SQL-killer's guide far enough and you will see them quietly reintroduce the very same abstractions SQL solved decades ago.
For me SQL feels like PHP. Sure it's usable, but it doesn't spark joy.
Syntactically PRQL is much simpler, cleaner and more flexible. You simply chain pipeline stages instead of having a single monolithic query statement.
Data model wise EdgeQL is close to what I want (links instead of joins, optionality instead of null, nesting support), but it's syntax is almost as bad as SQL.
I just wish they had mutation in there too. I don't like the idea of swapping between PRQL and SQL, let alone some complex update statements where i'd rather write the query in PRQL. .. Yea you could argue they shouldn't be that complex for updates though heh.
When you're having to work with someone else's godawful convoluted database schema you're sometimes really happy not to have to write the query yourself. Giving the LLM context with previously written queries can be quite effective.
The writing SQL experience is a product of both SQL's syntax, the structure of the database you're querying over and the complexity of your query.
When things get hairy, and you have a good number of representative queries already written that you can use as context, LLMs can be really nice tool.
Agreed, but maybe the step change there is refactoring the data model, not continuing to author “hairy” sql via LLM that’s all fine until it breaks… and you end having to mend the nastiness back into compliance the ol’ fashion way
You could definitely load your billions of records with millions of relationships into memory, denormalize, restructure and rewrite the data (flawlessly) a lot cheaper (computationally) than running a large LLM on all that hardware.
Text-to-SQL is effectively pissing in the wind when we start looking at business value. The trouble is that it works just enough to be dangerous. It is a very interesting idea and can easily consume the resources of a technical team ~indefinitely. One obvious sign we are chasing shiny here is the German-to-SQL example. This is fun, but no one would actually pay for it.
I would classify Text-to-SQL as a Schedule I rabbit hole. It has some serious effects, but it otherwise has no acceptable level of value-add in most reasonable enterprises.
>The trouble is that it works just enough to be dangerous.
This is a major theme with LLMs. When they first came out you'd see it randomly returning garbage in the middle of an otherwise good output maybe 30% of the time. You knew you had to go through it with a fine tooth comb.
Now it's more like 3%. And you just gloss over it.
I think you are right about Text-to-SQL being a trap. In this case the deficiencies are unaccaptable.
But elsewhere? I think we are going to see the "customer service effect" applied all over the place. I am referring to the downward trend of customer service. Where quality was eschewed for scale. We went from highly competent agents providing individual feedback to hard to reach agents with no agency. It scales, but the bar has been lowered significantly. You get less customer service.
I think AI begs for the same tradeoff. Settle for less because it makes some things easier. Which of course makes other things much more complex or even undermines the entire premise.
Reliability is the important dimension to focus on, but what is your baseline? I've worked in data and me and my colleagues regularly had to confront bugs (on many levels) that were communicated to end users
Germany is the 5th biggest economy in the world, then there is Austria and Switzerland. The claim that nobody would pay for German to SQL seems a bit pessimistic ;)
I'd also love to understand better why you think that there is no "acceptable level of value-add in most reasonable enterprises".
The original idea idea behind SQL was to create a language that looked like English and allowed regular users to express their queries in something that resembled natur
Al language. Naturally it has evolved into something far more complex, but maybe today with LLMs it can get back to its origin,
LLMs are getting pretty good at writing SQL. There is so much training material out there, and it is not that hard to validate the results. The real interesting question will be if they will be better at leveraging all the database specific dialects than tool like PowerBI. High-performance databases like Exasol often has a lot of specific features in their SQL dialects that generic tools and ORMs are not able to use, it will be interesting to see if LLMs can make that more accessible.
aren't programming languages like SQL designed to strike a quasi-optimal balance between precision and economy? that is to say, if I want to use plain English to specify some kind of data operations, the prompt will necessarily be more verbose and/or less specific than SQL statements. So why not just learn SQL properly? And then learn the domain-specific considerations of the source data as well as the domain-specific business requirements? Or am I missing something essential?
It's hard because relational algebra is hard. It's hard because it's easy to write a query that returns arbitrary data, but it's hard to write a query that returns only the correct data that you really want.
The problem is that, IMX, AI is just as shit at that as most humans are. AI can find and connect primary keys and columns with the same name. It doesn't understand cardinality of entities or how to look for data that breaks the data model of the application (which is a critical element of data validation).
None of the actual hard parts of writing SQL go away with AI.
This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.
> This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.
For most businesses, it really really is. In general, people (businesses) are incredibly sensitive about any possibility of data leakage, even just the metadata. There are lots of companies who would pay for this, and they tend to have a lot of money.
This would be a (small, granted) red flag for me about a company. As a data architect I work really hard to make sure our model is legible and obvious - the way which the model is adopted to our problem area is certainly a difficult problem that is valuable to have solved. But minor details of that schema should be extremely boring.
Instead of trying to get LLMs from zero to 100, which is impossible to do, you should concentrate on getting them from 75% to 95%.
The idea that someone who has no knowledge of the domain and no understanding of relational data modeling can chat “like to know really good um but don’t make any mistakes” with a GPU an uncover the mysteries of the universe is imbecilic.
But… someone what knows approximately what to do and sort of how to do it could work wonders — if we had LLMs trained on a corpus with specific rules.
I don’t know how to left join and what table I need to get the aggregate of sales in each region by date and price range, but I can describe it halfway and know how to check if each step is valid.
LLMs can do this. They’re trained on English, and they are able to weight definitive rules. But instead we throw a random text at a general purpose transformer.
Parsing a response of tokens into grammatical English is the most expensive computation (after the initial scraping and catalog). Instead of wasting all the cycles doing that against the sun total of GitHub, StackOverflow, Reddit, and Wikipedia, create a fuzzy match on a simplification a rigorous specification and train it on your data (just a few million tokens) to teach it that users have primary addresses and are associated to accounts that have regions and region X has roughly 10 times the sales volume of region y.
So someone intelligent in the matter with an understanding of logical rigor and a general idea of the data shape can actually become 10x more efficient, instead of trying to lift vibe coders to the level of Spakespearean monkeys, you could be turning mid-level devs into super analysts.
Long term, SQL will be phased out for new LLM-friendly query languages. The cost of AI generating tens of billions of queries per day creates a new incentive for rationality and efficiency that overcomes the human institutional weight of legacy languages. I'm working on such a language now: https://memelang.net/
I've only been through a few cycles, but there's three flavors I've seen:
1. Better query language: QUEL, LINQ, etc.
2. Better data model or performance: CouchDB, Mongo, Redis, etc.
3. Better abstraction: Zope, Active Record, et. al.
SQL vendors keep absorbing the differentiating features from other approaches with enough success that they capture most business use cases. That's not to say there aren't successes outside of SQL! But I've seen it claimed SQL will be dead several times over thanks to some new tech and most of the time SQL instead kills that tech or makes it niche.
I do agree. SQL is simply an access API for so many systems, and nice as it's a declarative language rather than a normal programming language. LLMs are super powerful to express questions to data that can then be translated into SQL.
I've always found it odd that most databases don't expose a kind of query bytecode API that SQL can be converted to. It would allow more kinds of languages, different SQL dialects, compile-time optimizations, and perhaps ORM could be compiled directly into your target database spec. Would reduce the amount of string operations by client code, at least
For translation between dialects, you could use projects such as SQLGlot. The advantage of SQL is the standardization over many decades (yes, I know that it's still a mess with the different dialects).
SQLGlot is amazing. In many ways it helps erase the differences and bridge between dialects. It is so useful for moving complex queries between platforms.
It's hit or miss, but for my product type it worked quite well: https://docs.uxwizz.com/guides/ask-ai
The problem is that often once it starts failing, it rarely recovers from an error, even given the exact error code.
In my dashboard, I added OpenRouter access, so you can quickly swap between models and see which one works best (for your desired accuracy/response time/pricing). Grok 4 Fast is one of the best overall.
The current benchmark for real world text to sql is https://spider2-sql.github.io/ and the SOTA is at 64%
Anyone selling you 99% accuracy can prove it there first.
I don't see Grok there
Cofounder of one of those analytics agents here (https://getdot.ai).
The promise of the technology is not that it can deal with any arbitrarily complex Enterprise setup, but rather that you expose it with enough guidance on a controlled and sufficiently good data model.
Depending on your use case this can be super valuable as it enables a lot more people to use data and get relevant recommendations.
But yeah it's work to make nice roads and put up signs everywhere.
I am working with Databricks' Genies. I have a _very_ complex Enterprise data schema(s). Genies, and from what I can tell, your product work on a set of tables ~20 and expect a well thought out and documented data model.
I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn't need an AI assistant. If I had a perfect data model my team could write simple SQL queries or give chatgpt a schema dump + a natural language query and it would get the answer most of the time.
IMHO, the big value in this space will be when these tools can wrangle realistic databases.
Well, I can't comment much on Genie, but the core question is always how you scale the complexity.
In Dot, it's divide and conquer. If you have several different teams each of them has to maintain their knowledge base.
A bunch of our customer have less than 10 tables hooked up to Dot, but this data is core to their business and so the analytics agent is really useful. Our most complex setup is on more than 5000 tables, but that was a lot more work to lay out the structure and guidelines.
Also, I don't think all organization are ready for AI. If the data model is a huge mess, data quality is poor and analytics use cases are not mature, it's better to focus on the fundamentals first.
But SQL is text, that's the whole point of it.
Sure, you have to phrase your question in a way that's a bit like trying to ask a very specific question of an annoying "Self-Diagnosed Internet Autistic" co-worker who can't tell the difference between being "precise" and being "a pedantic pain in the arse", but it is just text.
Oh you're upset because SQL isn't in German? Well there's no reason why you can't stick German into the lexer, set your columns up with German names, and get a query like
people who can really speak German, ja ich weiss meine Deutsch is so schlect, geh schon ;-)But really why would you bother?
Well, of course you're correct that SQL is text, but that's not what the article is arguing about. The point isn't whether SQL is text... it's about the kind of text it is.
SQL is a formal language, not a natural one. It's precise, rigid, and requires a specialized understanding of schema, joins, and logic. text-to-sql systems don't exist because people are too lazy to type; they exist because most people can't fluently express analytical intent in sql syntax. They can describe what they want in natural language ("show me all active users who registerd this year"), but translating that into correct, optimized sql requires at least familiarity, and sometimes expertise
So the governance challenges discussed in the article aren't about "oh SQL is too hard to type"...they're about trust, validation, and control when you introduce an AI intermediary that converts natural lang into a query that might affect sensitive data
I think it's important to reinforce that SQL isn't an COBOL like attempt at building a querying language out of natural expressions (which you could see if you squint really hard). Instead SQL is a refinement of various querying languages (hence being the standard one) that co-evolved with relational algebra. If you have a chance to learn more within an academic environment courses on relational algebra and the abstract theory of set operations can be invaluable to building a basis for more naturally understanding the intent and tools available in SQL.
What is the human need to bash SQL at all costs? At its core, such a simple syntax, yet its so powerful at aggregating/manipulating tabular data and the like. Instead we’d rather declaratively say what we want in a more verbose/disjointed way… fascinating
I'm with you. The more SQL alternatives I've seen, the more I'm convinced that SQL doesn't have that much fat or disfunction in it's syntax. There is inherent complexity. Follow an SQL-killer's guide far enough and you will see them quietly reintroduce the very same abstractions SQL solved decades ago.
For me SQL feels like PHP. Sure it's usable, but it doesn't spark joy.
Syntactically PRQL is much simpler, cleaner and more flexible. You simply chain pipeline stages instead of having a single monolithic query statement.
Data model wise EdgeQL is close to what I want (links instead of joins, optionality instead of null, nesting support), but it's syntax is almost as bad as SQL.
Oh man, PRQL looks so good.
I just wish they had mutation in there too. I don't like the idea of swapping between PRQL and SQL, let alone some complex update statements where i'd rather write the query in PRQL. .. Yea you could argue they shouldn't be that complex for updates though heh.
I'll give one reason of many: Its fixed pipeline model doesn't match how humans think about data transformation.
When you're having to work with someone else's godawful convoluted database schema you're sometimes really happy not to have to write the query yourself. Giving the LLM context with previously written queries can be quite effective.
The writing SQL experience is a product of both SQL's syntax, the structure of the database you're querying over and the complexity of your query.
When things get hairy, and you have a good number of representative queries already written that you can use as context, LLMs can be really nice tool.
Agreed, but maybe the step change there is refactoring the data model, not continuing to author “hairy” sql via LLM that’s all fine until it breaks… and you end having to mend the nastiness back into compliance the ol’ fashion way
You could definitely load your billions of records with millions of relationships into memory, denormalize, restructure and rewrite the data (flawlessly) a lot cheaper (computationally) than running a large LLM on all that hardware.
Text-to-SQL is effectively pissing in the wind when we start looking at business value. The trouble is that it works just enough to be dangerous. It is a very interesting idea and can easily consume the resources of a technical team ~indefinitely. One obvious sign we are chasing shiny here is the German-to-SQL example. This is fun, but no one would actually pay for it.
I would classify Text-to-SQL as a Schedule I rabbit hole. It has some serious effects, but it otherwise has no acceptable level of value-add in most reasonable enterprises.
>The trouble is that it works just enough to be dangerous.
This is a major theme with LLMs. When they first came out you'd see it randomly returning garbage in the middle of an otherwise good output maybe 30% of the time. You knew you had to go through it with a fine tooth comb.
Now it's more like 3%. And you just gloss over it.
I think you are right about Text-to-SQL being a trap. In this case the deficiencies are unaccaptable.
But elsewhere? I think we are going to see the "customer service effect" applied all over the place. I am referring to the downward trend of customer service. Where quality was eschewed for scale. We went from highly competent agents providing individual feedback to hard to reach agents with no agency. It scales, but the bar has been lowered significantly. You get less customer service.
I think AI begs for the same tradeoff. Settle for less because it makes some things easier. Which of course makes other things much more complex or even undermines the entire premise.
Reliability is the important dimension to focus on, but what is your baseline? I've worked in data and me and my colleagues regularly had to confront bugs (on many levels) that were communicated to end users
Germany is the 5th biggest economy in the world, then there is Austria and Switzerland. The claim that nobody would pay for German to SQL seems a bit pessimistic ;)
I'd also love to understand better why you think that there is no "acceptable level of value-add in most reasonable enterprises".
[dead]
The original idea idea behind SQL was to create a language that looked like English and allowed regular users to express their queries in something that resembled natur Al language. Naturally it has evolved into something far more complex, but maybe today with LLMs it can get back to its origin,
LLMs are getting pretty good at writing SQL. There is so much training material out there, and it is not that hard to validate the results. The real interesting question will be if they will be better at leveraging all the database specific dialects than tool like PowerBI. High-performance databases like Exasol often has a lot of specific features in their SQL dialects that generic tools and ORMs are not able to use, it will be interesting to see if LLMs can make that more accessible.
aren't programming languages like SQL designed to strike a quasi-optimal balance between precision and economy? that is to say, if I want to use plain English to specify some kind of data operations, the prompt will necessarily be more verbose and/or less specific than SQL statements. So why not just learn SQL properly? And then learn the domain-specific considerations of the source data as well as the domain-specific business requirements? Or am I missing something essential?
For all it's flaws, I certainly prefer SQL to something like XQuery.
Yes you are. I can show you 15 line SQL queries you would be hard pressed to understand, let alone built with precision as a novice
And you want everyone to learn this? People who don’t even have time or ability to master excel?
Really?
It's hard because relational algebra is hard. It's hard because it's easy to write a query that returns arbitrary data, but it's hard to write a query that returns only the correct data that you really want.
The problem is that, IMX, AI is just as shit at that as most humans are. AI can find and connect primary keys and columns with the same name. It doesn't understand cardinality of entities or how to look for data that breaks the data model of the application (which is a critical element of data validation).
None of the actual hard parts of writing SQL go away with AI.
This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.
> This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.
For most businesses, it really really is. In general, people (businesses) are incredibly sensitive about any possibility of data leakage, even just the metadata. There are lots of companies who would pay for this, and they tend to have a lot of money.
This would be a (small, granted) red flag for me about a company. As a data architect I work really hard to make sure our model is legible and obvious - the way which the model is adopted to our problem area is certainly a difficult problem that is valuable to have solved. But minor details of that schema should be extremely boring.
Cool! I need this at work XD
Horrible title and premise
Instead of trying to get LLMs from zero to 100, which is impossible to do, you should concentrate on getting them from 75% to 95%. The idea that someone who has no knowledge of the domain and no understanding of relational data modeling can chat “like to know really good um but don’t make any mistakes” with a GPU an uncover the mysteries of the universe is imbecilic.
But… someone what knows approximately what to do and sort of how to do it could work wonders — if we had LLMs trained on a corpus with specific rules.
I don’t know how to left join and what table I need to get the aggregate of sales in each region by date and price range, but I can describe it halfway and know how to check if each step is valid.
LLMs can do this. They’re trained on English, and they are able to weight definitive rules. But instead we throw a random text at a general purpose transformer.
Parsing a response of tokens into grammatical English is the most expensive computation (after the initial scraping and catalog). Instead of wasting all the cycles doing that against the sun total of GitHub, StackOverflow, Reddit, and Wikipedia, create a fuzzy match on a simplification a rigorous specification and train it on your data (just a few million tokens) to teach it that users have primary addresses and are associated to accounts that have regions and region X has roughly 10 times the sales volume of region y.
So someone intelligent in the matter with an understanding of logical rigor and a general idea of the data shape can actually become 10x more efficient, instead of trying to lift vibe coders to the level of Spakespearean monkeys, you could be turning mid-level devs into super analysts.
[dead]
Long term, SQL will be phased out for new LLM-friendly query languages. The cost of AI generating tens of billions of queries per day creates a new incentive for rationality and efficiency that overcomes the human institutional weight of legacy languages. I'm working on such a language now: https://memelang.net/
If history is any indication, you've got it backwards: it's more likely that LLMs will be phased out for SQL.
What history are you considering?
I've only been through a few cycles, but there's three flavors I've seen:
1. Better query language: QUEL, LINQ, etc.
2. Better data model or performance: CouchDB, Mongo, Redis, etc.
3. Better abstraction: Zope, Active Record, et. al.
SQL vendors keep absorbing the differentiating features from other approaches with enough success that they capture most business use cases. That's not to say there aren't successes outside of SQL! But I've seen it claimed SQL will be dead several times over thanks to some new tech and most of the time SQL instead kills that tech or makes it niche.
Why would it be phased out? This doesn’t track. If anything SQL is stickier than ever since LLMs are actually pretty good at generating it.
Ah ournfuture where we vibe code memelang
That doesn't make sense. LLMs will be used to write SQL, I'm sure (I do that) but replacing SQL?
I do agree. SQL is simply an access API for so many systems, and nice as it's a declarative language rather than a normal programming language. LLMs are super powerful to express questions to data that can then be translated into SQL.
I've always found it odd that most databases don't expose a kind of query bytecode API that SQL can be converted to. It would allow more kinds of languages, different SQL dialects, compile-time optimizations, and perhaps ORM could be compiled directly into your target database spec. Would reduce the amount of string operations by client code, at least
For translation between dialects, you could use projects such as SQLGlot. The advantage of SQL is the standardization over many decades (yes, I know that it's still a mess with the different dialects).
SQLGlot is amazing. In many ways it helps erase the differences and bridge between dialects. It is so useful for moving complex queries between platforms.