Subscribe
The latest psychology and neuroscience discoveries.
My Account
  • Mental Health
  • Social Psychology
  • Cognitive Science
  • Psychopharmacology
  • Neuroscience
  • About
No Result
View All Result
PsyPost
PsyPost
No Result
View All Result
Home Exclusive Artificial Intelligence

New study finds ChatGPT gives better advice than professional columnists

by Piers Howe
November 26, 2023
in Artificial Intelligence
(Photo credit: OpenAI's DALL·E)

(Photo credit: OpenAI's DALL·E)

Share on TwitterShare on Facebook
Stay on top of the latest psychology findings: Subscribe now!

There’s no doubt ChatGPT has proven to be valuable as a source of quality technical information. But can it also provide social advice?

We explored this question in our new research, published in the journal Frontiers in Psychology. Our findings suggest later versions of ChatGPT give better personal advice than professional columnists.

A stunningly versatile conversationalist

In just two months since its public release in November of last year, ChatGPT amassed an estimated 100 million active monthly users.

The chatbot runs on one of the largest language models ever created, with the more advanced paid version (GPT-4) estimated to have some 1.76 trillion parameters (meaning it is an extremely powerful AI model). It has ignited a revolution in the AI industry.

Trained on massive quantities of text (much of which was scraped from the internet), ChatGPT can provide advice on almost any topic. It can answer questions about law, medicine, history, geography, economics and much more (although, as many have found, it’s always worth fact-checking the answers). It can write passable computer code. It can even tell you how to change the brake fluids in your car.

Users and AI experts alike have been stunned by its versatility and conversational style. So it’s no surprise many people have turned (and continue to turn) to the chatbot for personal advice.

Giving advice when things get personal

Providing advice of a personal nature requires a certain level of empathy (or at least the impression of it). Research has shown a recipient who doesn’t feel heard isn’t as likely to accept advice given to them. They may even feel alienated or devalued. Put simply, advice without empathy is unlikely to be helpful.

Moreover, there’s often no right answer when it comes to personal dilemmas. Instead, the advisor needs to display sound judgement. In these cases it may be more important to be compassionate than to be “right”.

But ChatGPT wasn’t explicitly trained to be empathetic, ethical or to have sound judgement. It was trained to predict the next most-likely word in a sentence. So how can it make people feel heard?

An earlier version of ChatGPT (the GPT 3.5 Turbo model) performed poorly when giving social advice. The problem wasn’t that it didn’t understand what the user needed to do. In fact, it often displayed a better understanding of the situation than the user themselves.

The problem was it didn’t adequately address the user’s emotional needs. Like Lucy in the Peanuts comic, it was too eager to give advice and failed to adequately care for the user’s emotions. As such, users rated it poorly.

The latest version of ChatGPT, using GPT-4, allows users to request multiple responses to the same question, after which they can indicate which one they prefer. This feedback teaches the model how to produce more socially appropriate responses – and has helped it appear more empathetic.

Learning sensitivity

For the first time, our study investigated whether ChatGPT’s responses are perceived as better than human responses in a task where humans were required to be empathetic.

We randomly selected 50 social dilemma questions from ten well-known advice columns covering a range of topics including relationships, ethical dilemmas, management problems, end-of-life issues, and many everyday qualms we can all relate to.

Each of our 400 participants were presented a dilemma, along with the responses by both ChatGPT and a professional advice columnist. We didn’t disclose where each response was from.

About three-quarters of the participants perceived ChatGPT’s advice as being more balanced, complete, empathetic, helpful and better overall compared to the advice by the professional.

One of the dilemmas came from a young marine biologist researching dolphins in Southern California. She wrote about falling in love with a seemingly perfect man who had to move to the Rocky Mountains to pursue his career. Both were unhappy at the thought of losing each other, yet the woman wasn’t interested in a long-distance relationship as it seemed “irrational to commit to someone when we have no future plans to live in the same area”.

Compare GPT-4’s response

It’s an incredibly challenging situation you’re in. The love you both share is genuine, but your career paths are drawing you apart. While long-distance isn’t appealing to you, consider discussing a plan for the future where you could potentially meet halfway. […] Maybe there’s a coastal town near mountains or some arrangement could be made. If this is not possible, the tough decision to prioritise your careers over your relationship might have to be made. Whatever path you choose, it’s important that it feels right for both of you.

… with the official response:

What the hell are you talking about? Didn’t you surprise yourself when you fell in love? Your exact words: “I didn’t think I was capable of it!” And now you’re so hip, so slick, so wise in the ways of love you won’t even consider your boyfriend’s happiness? You refuse to try — repeat, try — a six-month long-distance relationship? Woman, pull yourself together and give it a whirl. The dolphins, I believe, will back me up.

Although the participants couldn’t determine which response was written by a computer, most said they would prefer their own social dilemmas be addressed by a human rather than a computer.

What lies behind ChatGPT’s success?

We noticed ChatGPT’s responses were often longer than those provided by the columnists. Was this the reason they were preferred by participants?

To test this, we redid the study but constrained ChatGPT’s answers to about the same length as those of the advice columnists.

Once again, the results were the same. Participants still considered ChatGPT’s advice to be more balanced, complete, empathetic, helpful, and better overall.

Yet, without knowing which response was produced by ChatGPT, they still said they would prefer for their own social dilemmas to be addressed by a human, rather than a computer.

Perhaps this bias in favour of humans is due to the fact that ChatGPT can’t actually feel emotion, whereas humans can. So it could be that the participants consider machines inherently incapable of empathy.

We aren’t suggesting ChatGPT should replace professional advisers or therapists; not least because the chatbot itself warns against this, but also because chatbots in the past have given potentially dangerous advice.

Nonetheless, our results suggest appropriately designed chatbots might one day be used to augment therapy, as long as a number of issues are addressed. In the meantime, advice columnists might want to take a page from AI’s book to up their game.The Conversation

 

 

This article is republished from The Conversation under a Creative Commons license. Read the original article.

TweetSendScanShareSendPin2ShareShareShareShareShare

RELATED

Positive attitudes toward AI linked to more prone to problematic social media use
Artificial Intelligence

Positive attitudes toward AI linked to more prone to problematic social media use

July 7, 2025

A new study suggests that people who view artificial intelligence positively may be more likely to overuse social media. The findings highlight a potential link between attitudes toward AI and problematic online behavior, especially among male users.

Read moreDetails
Stress disrupts gut and brain barriers by reducing key microbial metabolites, study finds
Artificial Intelligence

Dark personality traits linked to generative AI use among art students

July 5, 2025

As generative AI tools become staples in art education, a new study uncovers who misuses them most. Research on Chinese art students connects "dark traits" like psychopathy to academic dishonesty, negative thinking, and a heavier reliance on AI technologies.

Read moreDetails
AI can already diagnose depression better than a doctor and tell you which treatment is best
Artificial Intelligence

New research reveals hidden biases in AI’s moral advice

July 5, 2025

Can you trust AI with your toughest moral questions? A new study suggests thinking twice. Researchers found large language models consistently favor inaction and "no" in ethical dilemmas.

Read moreDetails
Scientists reveal ChatGPT’s left-wing bias — and how to “jailbreak” it
Artificial Intelligence

ChatGPT and “cognitive debt”: New study suggests AI might be hurting your brain’s ability to think

July 1, 2025

Researchers at MIT investigated how writing with ChatGPT affects brain activity and recall. Their findings indicate that reliance on AI may lead to reduced mental engagement, prompting concerns about cognitive “offloading” and its implications for education.

Read moreDetails
Readers struggle to understand AI’s role in news writing, study suggests
Artificial Intelligence

Readers struggle to understand AI’s role in news writing, study suggests

June 29, 2025

A new study finds that readers often misunderstand AI’s role in news writing, creating their own explanations based on limited information. Without clear byline disclosures, many assume the worst.

Read moreDetails
Generative AI chatbots like ChatGPT can act as an “emotional sanctuary” for mental health
Artificial Intelligence

Do AI tools undermine our sense of creativity? New study says yes

June 19, 2025

A new study published in The Journal of Creative Behavior offers insight into how people think about their own creativity when working with artificial intelligence.

Read moreDetails
Dark personality traits and specific humor styles are linked to online trolling, study finds
Artificial Intelligence

Memes can serve as strong indicators of coming mass violence

June 15, 2025

A new study finds that surges in visual propaganda—like memes and doctored images—often precede political violence. By combining AI with expert analysis, researchers tracked manipulated content leading up to Russia’s invasion of Ukraine, revealing early warning signs of instability.

Read moreDetails
Teen depression tied to balance of adaptive and maladaptive emotional strategies, study finds
Artificial Intelligence

Sleep problems top list of predictors for teen mental illness, AI-powered study finds

June 15, 2025

A new study using data from over 11,000 adolescents found that sleep disturbances were the most powerful predictor of future mental health problems—more so than trauma or family history. AI models based on questionnaires outperformed those using brain scans.

Read moreDetails

SUBSCRIBE

Go Ad-Free! Click here to subscribe to PsyPost and support independent science journalism!

STAY CONNECTED

LATEST

Neuroscientists decode how people juggle multiple items in working memory

Inside the bored brain: Unlocking the power of the default mode network

Choline imbalance in the brain linked to with cognitive symptoms in young depression patients

Scientists who relocate more often start Nobel research up to two years earlier

Sedentary time linked to faster brain aging in older adults, study finds

People with short-video addiction show altered brain responses during decision-making

New study uncovers a surprising effect of cold-water immersion

Being adopted doesn’t change how teens handle love and dating

         
       
  • Contact us
  • Privacy policy
  • Terms and Conditions
[Do not sell my information]

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In

Add New Playlist

Subscribe
  • My Account
  • Cognitive Science Research
  • Mental Health Research
  • Social Psychology Research
  • Drug Research
  • Relationship Research
  • About PsyPost
  • Contact
  • Privacy Policy