This is an attempt to answer the classic question (source: Dwarkesh Patel?) of why LLMs have not generated novel insights despite having memorized the bulk of human knowledge:

The first part of my answer is that they have. I often ask Deepresearch to research a topic, give it some suggestive followup questions, and watch as it combines a few related ideas into a genuinely interesting experiment idea. This is how most scientific progress works. Admittedly, it's not very good at it, so here's a more real answer:

asking the right questions

The requirements for an LLM coming up with a novel discovery are as follows:

The user must prompt it to accomplish a goal that requires novel insight.

This goal must overlap with the domain in which it's capable of making a novel insight.

Often, the hard part of science is coming up with the right question. If users don't ask an interesting question, they're not going to get an interesting answer.

If the user gives an extremely general query, like 'Please come up with a novel insight,' this won't put the LLM in a context in which it naturally combines ideas within a discipline. Most discoveries don't happen when someone sits down and thinks 'how can I advance science today?' Interesting discoveries occur in-context. If you had sat Einstein down and asked for a novel scientific discovery, I really just don't think he could've done it, at least on most days. You should not have a higher bar for GPT4 than for Einstein.

the nature of scientific discovery

I think questions about LLMs generating novel insights often fail to capture the nature of scientific discovery — most of the time, scientific progress doesn't stem from wild, unprompted connections between disciplines. Scientific progress happens when we sit down to work towards a goal and consider how related ideas might be combined. Then, prompted to think about these connections, we come up with ways to iteratively refine our understanding.

If you want an insight that doesn't stem from this iterative scientific process, the LLM has to be trained in a way that encourages diverging from the norm and taking risks to connect OOD ideas. In other words, quite differently from how all language models are currently trained. It would be surprising to me if any current LLM, on a user query about an extremely specific topic, attempted to connect random pieces of knowledge the user wasn't thinking about to craft a solution that it has never been trained on — when it could instead recite a safe, perfectly good memorized answer. Plus, ignoring questions about the exact training procedure, this 'miraculous connection between unrelated domains' type of intelligence is probably the hardest to instill into a machine to begin with.