As you say, Deep Research is great for the right use cases (checking, vs origination). I'd add low effort exploration of adjacencies, too.
I've written about how the non-obvious implication of this is that it will (/should?!) change *how* we approach seeking answers – moving to a 'who to ask' paradigm, rather than a 'how to answer' one.
The people & orgs that win will be those that manage to incentivise *more* human collaboration, rather than just producing more AI-generated research & analysis.
I'd suspect it's much better at narrative-style literature review than data gathering. Would be interesting to see your thoughts if you tried it next time you needed something like this (e.g. research at the start of a project to find what you should build on).
Rightly or wrongly, I think of AI a bit like 'wisdom of the crowd' since it 'scrapes' (love that word!) whatever it can find off the internet. I suppose to be really useful to you (and your PHD students) the crowd would have to be large enough and diverse enough to provide quality input for the stupid bot to summarise (albeit in nicely written prose). If you are already working creatively on solving these problems perhaps there just arent yet enough resolved 'partial' solutions to feed the machine.
But science in itself is 'wisdom of the crowd' in the sense that it always relies on past findings that are available. AI can handle far larger sets of data/information than a person but it doesn't necessarily condense or translate that into anything meaningful, insightful or accurate. It's not about the lack of access to info because all PhDs already do their work relying on what's available (and of course their own lab research if that's applicable).
Interesting .. well thought out evaluation ...you could give the same briefing to human student they would come back with a lit review and piles of data.
The network you need would have to be trained further on at least 50% of the data you already have and tested on the balance of your data. You need that same close supervision when you retrain the network. By retrain I don’t mean that you start from scratch; just add more specific training. One more thing: your prompt is way too elaborate and demanding. I would have broken it down in parts. Neural networks are basically correlation machines, which build large quantities of simple feature extractors. I also don’t believe that unsupervised training works as well as supervised training. Use both, but use the supervised training first. They are like children. They need good teachers who can evaluate step by step how they’re doing on learning your subtasks before moving on to the next level of complexity. Just because we don’t fully understand how they work, doesn’t mean that we can just throw anything at them and expect them to turn it all into gold! By the way, I’m retired, but I started using backprop at about the same time Hinton invented it, so I’ve had plenty of time (and fast machines and great data) to observe and think about them! They’re extraordinary and getting better at an amazing rate, but they’re still children, not even as good as your worst PhD applicant for this kind of prompt!
First explain how reading the literature works. How citation lists are collected and how science is driven forward by identifying the gray areas and the areas of weak confidence. AI can do steps and assist but it will never propose the Planet that Wasn’t and seek alternate explanations and models to test via experimentation.
Adam, thank you for sharing your experience; I am still trying to put my mind on this issues. Two things that I find good guidance in here:
1. We don’t train PhD students so they can continually produce papers of similar quality to their first one.
2.We help train them so they will get better, and grow as scientists
We do assessments as part of a learning process, to consolidate knowledge and help us reach the next step in how we eventually use that knowledge
As you say, Deep Research is great for the right use cases (checking, vs origination). I'd add low effort exploration of adjacencies, too.
I've written about how the non-obvious implication of this is that it will (/should?!) change *how* we approach seeking answers – moving to a 'who to ask' paradigm, rather than a 'how to answer' one.
The people & orgs that win will be those that manage to incentivise *more* human collaboration, rather than just producing more AI-generated research & analysis.
Would love your thoughts – https://thefuturenormal.substack.com/p/chatgpts-deep-research-and-thinking
It seems Deep Research's quality is very variable depending on domain, without any obvious pattern. See: https://open.substack.com/pub/understandingai/p/these-experts-were-stunned-by-openai?utm_source=share&utm_medium=android&r=11048
I'd suspect it's much better at narrative-style literature review than data gathering. Would be interesting to see your thoughts if you tried it next time you needed something like this (e.g. research at the start of a project to find what you should build on).
Rightly or wrongly, I think of AI a bit like 'wisdom of the crowd' since it 'scrapes' (love that word!) whatever it can find off the internet. I suppose to be really useful to you (and your PHD students) the crowd would have to be large enough and diverse enough to provide quality input for the stupid bot to summarise (albeit in nicely written prose). If you are already working creatively on solving these problems perhaps there just arent yet enough resolved 'partial' solutions to feed the machine.
But science in itself is 'wisdom of the crowd' in the sense that it always relies on past findings that are available. AI can handle far larger sets of data/information than a person but it doesn't necessarily condense or translate that into anything meaningful, insightful or accurate. It's not about the lack of access to info because all PhDs already do their work relying on what's available (and of course their own lab research if that's applicable).
Interesting .. well thought out evaluation ...you could give the same briefing to human student they would come back with a lit review and piles of data.
Nailed it.
The network you need would have to be trained further on at least 50% of the data you already have and tested on the balance of your data. You need that same close supervision when you retrain the network. By retrain I don’t mean that you start from scratch; just add more specific training. One more thing: your prompt is way too elaborate and demanding. I would have broken it down in parts. Neural networks are basically correlation machines, which build large quantities of simple feature extractors. I also don’t believe that unsupervised training works as well as supervised training. Use both, but use the supervised training first. They are like children. They need good teachers who can evaluate step by step how they’re doing on learning your subtasks before moving on to the next level of complexity. Just because we don’t fully understand how they work, doesn’t mean that we can just throw anything at them and expect them to turn it all into gold! By the way, I’m retired, but I started using backprop at about the same time Hinton invented it, so I’ve had plenty of time (and fast machines and great data) to observe and think about them! They’re extraordinary and getting better at an amazing rate, but they’re still children, not even as good as your worst PhD applicant for this kind of prompt!
So did you try to change your prompt and see what happened?
First explain how reading the literature works. How citation lists are collected and how science is driven forward by identifying the gray areas and the areas of weak confidence. AI can do steps and assist but it will never propose the Planet that Wasn’t and seek alternate explanations and models to test via experimentation.