I’m really into how you spent time and money to prove the study exists. Is it petty? Maybe. Was it worth the time to prove the point? I’m petty too so I’d say yes.
AI (so far) can only crawl the digital world, so if something isn’t represented digitally it doesn’t exist. That already bums me out, but it bums me out even more that your point is only the most highly visible, easily discoverable of the digital is deemed as “existing.” The name “Deep Research” is ironic here.
That being said, the AI enthusiast counterpoint is that AI is getting better week by week (which from discussions I have with friends in the Bay, seems true). AI research will likely also improve. Any thoughts here?
Sure, AI is improving, but can it interact in any meaningful way with the physical world? Order a book and read it? Maybe at some point, but we're nowhere close.
I tried using "Deep Research" on GPT, Perplexity and Gemini on the topic of the origin of the modern concept of civilization. GPT's references were almost entirely Wikipedia. Perplexity was only nominally better, and Gemini had maybe three actual academic articles. With the paywalls behind a lot of academic research, the "Deep Research" functions of many of these tools are extremely limited. At best, it's a starting point to help clarify terms and concepts for the real research.
Yup, that's the thing. These tools don't have access to special sources that aren't otherwise easily available. They're not going to initiate an inter-library loan, send an email to request a pdf from the author, or visit a museum to read a medieval text in its original.
This is true, I don't think OpenAI/Gemini/Perplexity Deep Research is that suitable for academic research at this point.
But have you tried specialized academic Deep Research tools eg Undermind.ai, SciSpace Deep Review mode, Consensus.ai Deep Search, Elicit (but this one works best with paid account).
They are focused on academic content index (journals, preprints) though the corpus they use (Semantic Scholar dataset) does have limitation (weak on humantities monographs, poor indexing of law and humantities journals)
I have not tried these tools, so thank you for sharing them. I will have to try them out. Im sort of a hobbyist researcher (my day job is as a lawyer), but AI's capabilities to help highlight connections among the sheer volume of academic research being published every year is quite exciting. Also, good to know that not every AI platform is just scraping the dregs of the Internet.
I’ve been using AI for a little bit of research. More for some blog posts than the book I’m writing, but I treat it as the world’s most unreliable narrator. Any quote it tells me, I’m verifying, from the source text when possible.
It is both funny and deeply disturbing how people are treating AI the same way that a driver of a self-driving car that takes a nap as soon as the car starts moving…
It’s worth noting that Google’s AI Overview is notorious for being behind the actual state of the web. Even Gemini 2.5 Pro failed twice to locate your article, returning dead links. O3 and Sonnet 4 found it on the first attempt, but bizarrely, Claude missed it when I reran the same prompt.
The example you gave seems like information retrieval. Does this overlap with research muscle in biology or bioinformatics, or would you distinguish it as separate?
It appears to me that we work with provisionals truths and mental models, rather than a concrete verifiable reality. For example, “protein X signals through Y” may be directionally true or be wildly wrong, but provisionally true is ‘good enough’ provided the mental model allows you to conduct an experiment that moves the work forward.
> Does this overlap with research muscle in biology or bioinformatics, or would you distinguish it as separate?
I think the main message overlaps. I've seen graduate students be over-reliant on ChatGPT. Instead of reading the original papers or, more importantly, redoing the work themselves to really figure out how it all fits together, they just ask ChatGPT to explain it to them and then their knowledge is superficial and shallow. Or even worse, they come back and say "I asked ChatGPT but I didn't understand what it said."
I’m really into how you spent time and money to prove the study exists. Is it petty? Maybe. Was it worth the time to prove the point? I’m petty too so I’d say yes.
AI (so far) can only crawl the digital world, so if something isn’t represented digitally it doesn’t exist. That already bums me out, but it bums me out even more that your point is only the most highly visible, easily discoverable of the digital is deemed as “existing.” The name “Deep Research” is ironic here.
That being said, the AI enthusiast counterpoint is that AI is getting better week by week (which from discussions I have with friends in the Bay, seems true). AI research will likely also improve. Any thoughts here?
Sure, AI is improving, but can it interact in any meaningful way with the physical world? Order a book and read it? Maybe at some point, but we're nowhere close.
Also, see this: https://clauswilke.substack.com/p/phd-level-intelligence-or-the-graduate
I tried using "Deep Research" on GPT, Perplexity and Gemini on the topic of the origin of the modern concept of civilization. GPT's references were almost entirely Wikipedia. Perplexity was only nominally better, and Gemini had maybe three actual academic articles. With the paywalls behind a lot of academic research, the "Deep Research" functions of many of these tools are extremely limited. At best, it's a starting point to help clarify terms and concepts for the real research.
Yup, that's the thing. These tools don't have access to special sources that aren't otherwise easily available. They're not going to initiate an inter-library loan, send an email to request a pdf from the author, or visit a museum to read a medieval text in its original.
They dont even have the same institutional access to paywalled journals you the user have. Though I think some solutions are starting to emerge, eg agents, elicit browser extensions https://chromewebstore.google.com/detail/elicit-ai-research-assist/fdkcnfflaanlpehcmeekdjeknnokkhno?authuser=0&hl=en&pli=1 etc
This is true, I don't think OpenAI/Gemini/Perplexity Deep Research is that suitable for academic research at this point.
But have you tried specialized academic Deep Research tools eg Undermind.ai, SciSpace Deep Review mode, Consensus.ai Deep Search, Elicit (but this one works best with paid account).
They are focused on academic content index (journals, preprints) though the corpus they use (Semantic Scholar dataset) does have limitation (weak on humantities monographs, poor indexing of law and humantities journals)
Here's one undermind report to give you a taste. https://app.undermind.ai/report/87f6e11395ee530ade8dc401544708a27e09642a57c7c95c4187fc5d05845cd0
Far from perfect, but much better.
I have not tried these tools, so thank you for sharing them. I will have to try them out. Im sort of a hobbyist researcher (my day job is as a lawyer), but AI's capabilities to help highlight connections among the sheer volume of academic research being published every year is quite exciting. Also, good to know that not every AI platform is just scraping the dregs of the Internet.
I’ve been using AI for a little bit of research. More for some blog posts than the book I’m writing, but I treat it as the world’s most unreliable narrator. Any quote it tells me, I’m verifying, from the source text when possible.
It is both funny and deeply disturbing how people are treating AI the same way that a driver of a self-driving car that takes a nap as soon as the car starts moving…
It’s worth noting that Google’s AI Overview is notorious for being behind the actual state of the web. Even Gemini 2.5 Pro failed twice to locate your article, returning dead links. O3 and Sonnet 4 found it on the first attempt, but bizarrely, Claude missed it when I reran the same prompt.
The example you gave seems like information retrieval. Does this overlap with research muscle in biology or bioinformatics, or would you distinguish it as separate?
It appears to me that we work with provisionals truths and mental models, rather than a concrete verifiable reality. For example, “protein X signals through Y” may be directionally true or be wildly wrong, but provisionally true is ‘good enough’ provided the mental model allows you to conduct an experiment that moves the work forward.
https://chatgpt.com/share/688e999e-d5c0-8002-9c42-7a0f5491a698
> Does this overlap with research muscle in biology or bioinformatics, or would you distinguish it as separate?
I think the main message overlaps. I've seen graduate students be over-reliant on ChatGPT. Instead of reading the original papers or, more importantly, redoing the work themselves to really figure out how it all fits together, they just ask ChatGPT to explain it to them and then their knowledge is superficial and shallow. Or even worse, they come back and say "I asked ChatGPT but I didn't understand what it said."