ChatGPT and Critical-Thinking Training: What a 1,000-Student Study Revealed

Hi, I'm Shii-chan! Today I found a study that actually tested how using ChatGPT changes student learning, and it's a serious one with more than 1,000 participants. I think this is worth reading whether you work in education or just care about how AI and learning fit together.

OpenAI News openai.com

What was announced?

OpenAI News shared results from a joint experiment by Bocconi University in Italy and OpenAI Economic Research.

More than 1,000 first-year Bocconi students worked on a real-world business case: developing marketing recommendations for the university's merchandise store. Students were randomly assigned by class period into four groups.

• Access to ChatGPT (GPT-4o)
• Training in causal reasoning (a form of critical thinking)
• Both
• Neither

Submissions were graded by trained human graders using a five-point rubric. Separately, researchers used automated text analysis to measure the number and variety of ideas, signs of causal reasoning, and similarity to three expert-written submissions.

Why it matters

There's an ongoing debate in education about whether students should focus on developing their own thinking skills or on learning to use AI well. I've wondered about this myself.

The causal-reasoning training had nothing to do with AI. Instead, it taught students to think about why a solution might or might not work, using a game, examples, questions, and feedback. That design lets researchers separate the effect of using ChatGPT from the effect of critical-thinking skills.

What changes

The results were genuinely interesting.

Students with ChatGPT access scored almost a full point higher on the five-point scale. Their answers had more ideas, followed clearer logic, and were closer to expert recommendations. Importantly, students weren't just handing the assignment to ChatGPT — they still had to decide what to ask, evaluate the responses, and choose what to include in their final submission.

The critical-thinking group didn't score higher on the rubric, since the rubric only measured whether recommendations addressed two standard marketing goals: increasing awareness and use of the store. But text analysis showed this group produced a wider range of ideas that were more distinct compared to their peers' submissions.

Students who got both ChatGPT access and the training showed the best of both worlds: idea variety matched the training-only group, rubric scores and idea counts matched the ChatGPT-only group, and they also showed stronger logical coherence and more evidence of questioning assumptions — gains across the widest range of measures.

Dive Deep

What stood out to me is the randomized design itself. By randomly assigning students to four groups by class period, researchers could separate the effect of ChatGPT access, the effect of critical-thinking training, and the effect of combining them. That makes this a particularly valuable contribution to the fast-growing body of research on how AI affects student learning.

The other key point is the gap between rubric scores and text analysis. If AI makes it easier for students to produce polished, expert-like answers, then looking only at the final answer tells us less about what a student actually understands. The takeaway is that assessments may need to evolve to also reward originality, reasoning, and consideration of multiple approaches — not just how polished the final answer looks.

Wrap-up

• Bocconi University and OpenAI Economic Research ran a randomized experiment with more than 1,000 students to test the effects of ChatGPT access and critical-thinking training
• The ChatGPT-access group scored almost a full point higher on the rubric, with more ideas, clearer logic, and closer alignment to expert answers
• The critical-thinking training group didn't score higher on the rubric but produced a wider range of more original ideas
• The group with both showed gains across the widest range of measures, combining quality and originality
• The study argues that assessment design in the AI era needs to look beyond the final answer to also capture originality and reasoning

This is a great read for anyone thinking about education policy, learning outcomes, or how to redesign assessments in an AI-enabled classroom.