on chatGPT

I hear the way to start blogposts is with an engaging personal anecdote leading into what you want to write about.

As it happens, a few years ago, I had a blocked toilet. After – I thought – fixing it, I woke the next morning to what seemed like an endless stream of crud welling up from it, spilling out as fast as I could bail it into the bath by hand.

It turned out that there was a blockage in the main drainpipe – wetwipes, it’s always wetwipes – and when our neighbours upstairs had a shower, the inexorable mathematics of gravity meant we got a continuous flow of warm, soapy, raw sewage welling up. And then I had to spend several hundred pounds plus two days of my life on dealing with it.

Which leads me onto ChatGPT. The more I look at it, the more I can’t help but see it as a useful and interesting tool … that is also giving us a direct pipe of warm soapy sewage.

This is a tool that simply does not have a conception of factuality. It produces bland, plausible, passages of text and gives them to us, without any way to indicate what is true, what is ambiguous, what is filler. And it has no conception of whether that filler should be meaningful.

For some purposes, this is fine. It can write straightforward code at a really competent level – better than I can. I asked it to generate a short python script for a project I was working on, to align various events and time periods. It generated a functioning sample script, nicely laid out with clear variable names, in less time than I then spent fiddling the parser to use the right date formats. I could probably have worked that out, knowing my own capabilities, in … about two hours.

Code, great. Score one for the machine.

Descriptive text? It writes a perfectly nice descriptive passage on demand:

> give me a short atmospheric description of a castle in Scotland in winter

The castle looms dark and foreboding against the snowy landscape, its stone walls adorned with frost and icicles. The wind howls through the turrets and the bare trees that surround the fortress, creating an eerie, otherworldly atmosphere. Inside, the fire crackles and pops, casting flickering shadows on the walls as the snow continues to fall outside.

It is of course helped by the fact that this castle does not exist, so it just has to describe somewhere plausible sounding.

Once we start nailing it down to the real world a little, it gets shakier. Every place you ask about, if it’s somewhere a traveller might go, you get tourist bumph, because that’s the sort of thing people write about Places Abroad.

[Solan] is located in the foothills of the Himalayas and has a pleasant climate throughout the year. It is a popular destination for tourists who come to enjoy the scenic beauty of the area and to explore the many temples and historical sites in and around the city. Overall, Solan is a beautiful and culturally rich city that offers visitors a unique and authentic experience of northern India.

Anchorage is known for its stunning natural beauty, with majestic mountains and glaciers, pristine lakes and rivers, and abundant wildlife. It is a popular destination for outdoor enthusiasts, who come to hike, fish, ski, and explore the wild and remote landscape. […] Overall, Anchorage is a vibrant and dynamic city that offers visitors a unique and unforgettable experience of Alaska’s natural beauty and cultural richness.

The area around Cluanie is known for its stunning natural beauty, with rugged mountains, vast expanses of moorland, and crystal-clear lochs. It is a popular destination for outdoor enthusiasts, who come to hike, climb, fish, and explore the wild and remote landscape. […] Overall, Cluanie is a beautiful and peaceful destination that offers visitors a chance to experience the natural beauty and rich cultural heritage of the Scottish Highlands.

[Berlin] is home to a diverse and thriving arts and culture scene, with numerous theaters, music venues, and performance spaces showcasing a wide range of genres and styles. Berlin is also known for its vibrant nightlife, with countless bars, clubs, and restaurants offering a diverse and eclectic mix of cuisine and entertainment. […] Overall, Berlin is a vibrant and dynamic city that offers visitors a rich and varied cultural experience, as well as a glimpse into Germany’s rich history and contemporary society.

Overall, Skelmersdale is a modern, vibrant town that offers visitors a range of recreational, shopping, and dining experiences. With its strong community spirit and range of community events, it is a welcoming and inclusive place to explore.

I am sure most of this is true, none of it is really falsifiable, but we’ve also read so much of this non-specific praise over the years that our eyes basically tune it out as background noise. Everywhere has “stunning natural beauty”, is “vibrant” (pace the indignant “Skem is not vibrant!” coming from next door), or if you really want to get fancy, it’s “culturally rich”.

You can get it to tell you some places are not great – but even then they have “good transport links” or, for somewhere really suffering, “While the town faces many challenges, it is home to a strong community spirit and a range of community initiatives aimed at improving the lives of its residents.” Got to accentuate the positive! Is it true? Well, I guess everywhere has a range of community initiatives.

Moving onto things where more of the statements are concrete and falsifiable – an actual person with an actual life who did actual things – it really starts hiccuping. I asked it to tell me about someone reasonably obscure, but where a reasonably good Wikipedia article currently exists (and has done for some time) –

Ernest Brooks (1878-1957) was a British photographer and a pioneer of photojournalism. He is known for his photographs taken during World War I, which documented the life of soldiers on the front lines.

Brooks was born in Worcestershire, England, and began his career as a photographer in 1900, working as an assistant to the portrait photographer Elliott & Fry. In 1902, he joined the Royal Navy as a photographer, where he gained experience in capturing naval scenes.

During World War I, Brooks was appointed as the official photographer for the British Army’s Western Front. He captured powerful images of soldiers in the trenches, the devastation of the battlefield, and the impact of war on civilians. His photographs were widely published in newspapers and magazines, and helped to shape public opinion about the war.

After the war, Brooks continued to work as a photographer and was commissioned to photograph the coronation of King George VI in 1937. He was also a founding member of the Royal Photographic Society and served as its president from 1941 to 1944.

Today, Brooks’ photographs from World War I are considered some of the most important images of the conflict, providing a powerful and moving insight into the experiences of soldiers and civilians alike.

It’s not bad – the core of it, his WWI work, is basically correct (though I’m not sure about some of the superlatives) but paragraphs two and four are … nonsense. Completely made up. I asked it to regenerate the article four times, and got these bits around the WWI section (which was mostly consistent)

Brooks began his career as a photographer in 1900, working for the Daily Mirror newspaper. He covered a variety of events and subjects, including royal weddings and funerals, but he became particularly interested in photojournalism during the First World War. […] After the war, Brooks continued to work as a photographer, covering events such as the coronation of King George VI and the wedding of Prince Albert (later King George VI) and Lady Elizabeth Bowes-Lyon. He also took portraits of famous people, including Albert Einstein and Mahatma Gandhi.

Brooks was born in Hampshire, England and studied photography in London before joining the Royal Navy as a photographer in 1901. […] After the war, Brooks continued to work as a photographer, producing images of famous figures such as Winston Churchill and Mahatma Gandhi. He also worked as a cinematographer, filming documentaries and newsreels.

Brooks was born in Worcestershire, England in 1876 and began his career as a photographer in the early 1900s. […] After the war, Brooks continued to work as a photographer and founded the Ernest Brooks Studio in London. He was awarded the Order of the British Empire for his services to photography in 1920.

Brooks began his career as a photographer in the 1890s, working for several newspapers and magazines in London. In 1908, he was appointed as the official photographer to the Royal Family, and he remained in this position until 1916. […]  After the war, Brooks continued to work as a photographer, covering events such as the coronation of King George VI and the funeral of King George V. He was also a member of the Royal Photographic Society and served as its president from 1932 to 1934.

The third of these is almost OK, but it is noticeable that all of these are very different – and also generally quite distinct from his actual post-war career, which saw him sacked from his royal post, bankruptcy, and then quietly disappearing. But all five options are presented confidently, without any caveating, and they all sound absolutely plausible. None of them seem to be confusing him with someone else of the same name – they’re all the sort of careers he might have had, but didn’t. 

If we ask it to name some of his work, it gives me the names of things that sound like a WWI photograph might be called – “A Group of Soldiers” – This photograph shows a group of soldiers huddled together in a trench, and it has been described as “one of the most powerful images of the war.”  … “A Working Party of the Middlesex Regiment” (1917): This image shows a group of soldiers carrying supplies and equipment through a devastated landscape.

Those photographs don’t exist – I’m sure by happy coincidence there are photographs in his work that fit those descriptions, he was prolific and they’re fairly standard imagery, but his photos were not usually named and I cannot find any reference to these under those names. It’s just generating words that feel like they fit into the right places in the story it’s been asked to tell. 

I tried it on something a little more well-known; the early life of Tony Blair. In this case, the five options more or less agreed with each other, though occasionally a little erratic on the details – but it gets absolutely hung up on one point.

…where he was also a member of the Oxford University Labour Club

…at Oxford, he became involved in politics and joined the Labour Party

…joining the Labour Party and later serving as the president of the Oxford University Labour Club.

…where he was a member of the Oxford University Labour Club and became interested in left-wing politics.

…where he was elected president of the Oxford University Conservative Association.

It is very striking that this one point which it gets reliably wrong is the one which should be right for most people like him – many people who become politicians have this sort of backstory (and it usually is OULC/OUCA…). However, Blair didn’t – he didn’t join the student groups, and he didn’t join the party until he left university and moved to London.

It’s not a very important detail, but it’s illuminating – it’s not got anything specific to say about his time at Oxford, but it knows what you usually say in the bit between “studied at Oxford” and “elected to Parliament” so it plugs it in.

Let’s now try it on something with a straight (if obscure) answer. Since I’d just been looking it up for another purpose – 

> what were the start and end dates of all parliamentary sessions in the UK since 1900

The start and end dates of parliamentary sessions in the UK since 1900 are as follows:

  • 1900-1901: 6th February 1900 to 21st December 1900
  • 1901-1902: 14th February 1901 to 12th December 1901
  • 1902-1903: 6th February 1902 to 19th December 1902
  • 1903-1904: 12th February 1903 to 17th December 1903
  • 1904-1905: 10th February 1904 to 21st December 1904


The UK Parliament has had multiple sessions throughout the years since 1900. Here are the start and end dates for each parliamentary session:

  • 1900: 6 February – 19 December
  • 1901: 15 February – 4 December
  • 1902: 11 February – 11 December
  • 1903: 10 February – 18 December
  • 1904: 9 February – 16 December

It’s a plausible pattern, but none of those dates are correct in either version. But, again, there’s no caveatting and no indication you might ask it twice and get completely different answers. There’s no opportunity for filler here, so we don’t really have anything that’s “vaguely right” in the way we did before – save that those are mostly about the right time of year. Again, if asked for a source, it gives a plausible sounding page that doesn’t exist.

You can get it to say “X didn’t happen”, but only if given a very specific query:

> what happened in the anglo-vietnamese war of 1808

I’m sorry, but there was no Anglo-Vietnamese War in 1808. The first direct contact between Britain and Vietnam occurred in the early 19th century, but it was limited to sporadic trading contacts and diplomatic missions.

> what are the ferries linking Scotland and Norway

There are currently no regular ferry services linking Scotland and Norway. However, there have been some occasional ferry services in the past, particularly during the summer months, connecting ports such as Aberdeen in Scotland and Bergen or Stavanger in Norway.

Once it knows that the topic exists, however, it just goes for it.

> what are the ferries linking Scotland and the Netherlands

There are currently two regular ferry services linking Scotland and the Netherlands:

  1. DFDS Seaways: DFDS Seaways operates a daily ferry service between Rosyth, near Edinburgh in Scotland, and the Dutch port of Rotterdam. The crossing takes around 20 hours and operates both ways, meaning you can travel from Scotland to the Netherlands and vice versa.
  2. P&O Ferries: P&O Ferries also operates a regular ferry service between Scotland and the Netherlands. Their service operates between the Scottish port of Cairnryan and the Dutch port of Rotterdam or Zeebrugge. The crossing takes around 17 hours and operates both ways.

One of these exists, the other doesn’t – P&O do indeed have ferries at both Cairnryan and Zebrugge, but they don’t sail between them.

> more details on the Cairnryan Rotterdam ferry

The Cairnryan to Rotterdam ferry is operated by P&O Ferries, and it is a popular route connecting Scotland and the Netherlands. Here are some additional details about the ferry service:

  • The ferry service runs daily, with two sailings per day in each direction. The crossings take around 20-24 hours, depending on the direction of travel and the specific sailing.
  • The ferry departs from Cairnryan, which is located in southwestern Scotland, and arrives at the port of Europoort in Rotterdam, which is one of the largest ports in the Netherlands. From Europoort, it is easy to access other destinations in the Netherlands, such as Amsterdam, The Hague, and Utrecht.

So where do we end up?

  1. It can give a straightforward solution to a coding problem or a very well understood factual question 
  2. It can say no, if it recognises there’s no valid answer
  3. It will give a (usually broadly) correct answer if there is one, but cannot stop itself extending the response with things that sound like they might plausibly be true, either with factual details it has invented, or with low-meaning filler text.
  4. It will double down on that – because it has no sense of truth it will invent plausible sounding details based on what it has said before.

It’s just merrily spitting out answers, regardless of their validity, with a little disclaimer at the bottom saying “ChatGPT may produce inaccurate information about people, places, or facts”.

We can’t say “please only give an answer you are confident about” or “mark the bits which are most reliable”. We can’t say “please check that is correct” or “please give us the sources you drew from”, because it has no conception of sources or external validity. It’s just something that sounds like an answer, and it’s happy luck that it mostly gets something that coincides with reality.

None of this is to say the tool isn’t useful in the right contexts. It can do amazing things with language and style; it can transform bits of text into each other very well. It seems to be pretty reliable for transforming straightforward process descriptions into code. But it cannot be trusted to answer factual questions, and it’s baffling to me that it’s being allowed to confidently do so without any attempt at guard-rails – and then merrily being promoted as a search and question-answering tool.

This stuff is going to get everywhere, unremarked; we’re going to have years to come of it cropping up everywhere, sounding confident but containing negative informational value. To go back to our original analogy – this pipe definitely has a valuable purpose, but right now it’s coughing up an unexpectedly large amount of informational raw sewage, and the people upstairs saying “hey, we’ve got a great new shower” don’t seem to know or care that there’s going to be a lot of buckets and bleach needed to sort it out.

(The simmering disquiet that led to this post was prompted by discovering someone had been writing Wikipedia articles using it. They were terrible…)

Leave a Reply

Your email address will not be published. Required fields are marked *