on chatGPT

I hear the way to start blogposts is with an engaging personal anecdote leading into what you want to write about.

As it happens, a few years ago, I had a blocked toilet. After – I thought – fixing it, I woke the next morning to what seemed like an endless stream of crud welling up from it, spilling out as fast as I could bail it into the bath by hand.

It turned out that there was a blockage in the main drainpipe – wetwipes, it’s always wetwipes – and when our neighbours upstairs had a shower, the inexorable mathematics of gravity meant we got a continuous flow of warm, soapy, raw sewage welling up. And then I had to spend several hundred pounds plus two days of my life on dealing with it.

Which leads me onto ChatGPT. The more I look at it, the more I can’t help but see it as a useful and interesting tool … that is also giving us a direct pipe of warm soapy sewage.

This is a tool that simply does not have a conception of factuality. It produces bland, plausible, passages of text and gives them to us, without any way to indicate what is true, what is ambiguous, what is filler. And it has no conception of whether that filler should be meaningful.

For some purposes, this is fine. It can write straightforward code at a really competent level – better than I can. I asked it to generate a short python script for a project I was working on, to align various events and time periods. It generated a functioning sample script, nicely laid out with clear variable names, in less time than I then spent fiddling the parser to use the right date formats. I could probably have worked that out, knowing my own capabilities, in … about two hours.

Code, great. Score one for the machine.

Descriptive text? It writes a perfectly nice descriptive passage on demand:

> give me a short atmospheric description of a castle in Scotland in winter

The castle looms dark and foreboding against the snowy landscape, its stone walls adorned with frost and icicles. The wind howls through the turrets and the bare trees that surround the fortress, creating an eerie, otherworldly atmosphere. Inside, the fire crackles and pops, casting flickering shadows on the walls as the snow continues to fall outside.

It is of course helped by the fact that this castle does not exist, so it just has to describe somewhere plausible sounding.

Once we start nailing it down to the real world a little, it gets shakier. Every place you ask about, if it’s somewhere a traveller might go, you get tourist bumph, because that’s the sort of thing people write about Places Abroad.

[Solan] is located in the foothills of the Himalayas and has a pleasant climate throughout the year. It is a popular destination for tourists who come to enjoy the scenic beauty of the area and to explore the many temples and historical sites in and around the city. Overall, Solan is a beautiful and culturally rich city that offers visitors a unique and authentic experience of northern India.

Anchorage is known for its stunning natural beauty, with majestic mountains and glaciers, pristine lakes and rivers, and abundant wildlife. It is a popular destination for outdoor enthusiasts, who come to hike, fish, ski, and explore the wild and remote landscape. […] Overall, Anchorage is a vibrant and dynamic city that offers visitors a unique and unforgettable experience of Alaska’s natural beauty and cultural richness.

The area around Cluanie is known for its stunning natural beauty, with rugged mountains, vast expanses of moorland, and crystal-clear lochs. It is a popular destination for outdoor enthusiasts, who come to hike, climb, fish, and explore the wild and remote landscape. […] Overall, Cluanie is a beautiful and peaceful destination that offers visitors a chance to experience the natural beauty and rich cultural heritage of the Scottish Highlands.

[Berlin] is home to a diverse and thriving arts and culture scene, with numerous theaters, music venues, and performance spaces showcasing a wide range of genres and styles. Berlin is also known for its vibrant nightlife, with countless bars, clubs, and restaurants offering a diverse and eclectic mix of cuisine and entertainment. […] Overall, Berlin is a vibrant and dynamic city that offers visitors a rich and varied cultural experience, as well as a glimpse into Germany’s rich history and contemporary society.

Overall, Skelmersdale is a modern, vibrant town that offers visitors a range of recreational, shopping, and dining experiences. With its strong community spirit and range of community events, it is a welcoming and inclusive place to explore.

I am sure most of this is true, none of it is really falsifiable, but we’ve also read so much of this non-specific praise over the years that our eyes basically tune it out as background noise. Everywhere has “stunning natural beauty”, is “vibrant” (pace the indignant “Skem is not vibrant!” coming from next door), or if you really want to get fancy, it’s “culturally rich”.

You can get it to tell you some places are not great – but even then they have “good transport links” or, for somewhere really suffering, “While the town faces many challenges, it is home to a strong community spirit and a range of community initiatives aimed at improving the lives of its residents.” Got to accentuate the positive! Is it true? Well, I guess everywhere has a range of community initiatives.

Moving onto things where more of the statements are concrete and falsifiable – an actual person with an actual life who did actual things – it really starts hiccuping. I asked it to tell me about someone reasonably obscure, but where a reasonably good Wikipedia article currently exists (and has done for some time) –

Ernest Brooks (1878-1957) was a British photographer and a pioneer of photojournalism. He is known for his photographs taken during World War I, which documented the life of soldiers on the front lines.

Brooks was born in Worcestershire, England, and began his career as a photographer in 1900, working as an assistant to the portrait photographer Elliott & Fry. In 1902, he joined the Royal Navy as a photographer, where he gained experience in capturing naval scenes.

During World War I, Brooks was appointed as the official photographer for the British Army’s Western Front. He captured powerful images of soldiers in the trenches, the devastation of the battlefield, and the impact of war on civilians. His photographs were widely published in newspapers and magazines, and helped to shape public opinion about the war.

After the war, Brooks continued to work as a photographer and was commissioned to photograph the coronation of King George VI in 1937. He was also a founding member of the Royal Photographic Society and served as its president from 1941 to 1944.

Today, Brooks’ photographs from World War I are considered some of the most important images of the conflict, providing a powerful and moving insight into the experiences of soldiers and civilians alike.

It’s not bad – the core of it, his WWI work, is basically correct (though I’m not sure about some of the superlatives) but paragraphs two and four are … nonsense. Completely made up. I asked it to regenerate the article four times, and got these bits around the WWI section (which was mostly consistent)

Brooks began his career as a photographer in 1900, working for the Daily Mirror newspaper. He covered a variety of events and subjects, including royal weddings and funerals, but he became particularly interested in photojournalism during the First World War. […] After the war, Brooks continued to work as a photographer, covering events such as the coronation of King George VI and the wedding of Prince Albert (later King George VI) and Lady Elizabeth Bowes-Lyon. He also took portraits of famous people, including Albert Einstein and Mahatma Gandhi.

Brooks was born in Hampshire, England and studied photography in London before joining the Royal Navy as a photographer in 1901. […] After the war, Brooks continued to work as a photographer, producing images of famous figures such as Winston Churchill and Mahatma Gandhi. He also worked as a cinematographer, filming documentaries and newsreels.

Brooks was born in Worcestershire, England in 1876 and began his career as a photographer in the early 1900s. […] After the war, Brooks continued to work as a photographer and founded the Ernest Brooks Studio in London. He was awarded the Order of the British Empire for his services to photography in 1920.

Brooks began his career as a photographer in the 1890s, working for several newspapers and magazines in London. In 1908, he was appointed as the official photographer to the Royal Family, and he remained in this position until 1916. […]  After the war, Brooks continued to work as a photographer, covering events such as the coronation of King George VI and the funeral of King George V. He was also a member of the Royal Photographic Society and served as its president from 1932 to 1934.

The third of these is almost OK, but it is noticeable that all of these are very different – and also generally quite distinct from his actual post-war career, which saw him sacked from his royal post, bankruptcy, and then quietly disappearing. But all five options are presented confidently, without any caveating, and they all sound absolutely plausible. None of them seem to be confusing him with someone else of the same name – they’re all the sort of careers he might have had, but didn’t. 

If we ask it to name some of his work, it gives me the names of things that sound like a WWI photograph might be called – “A Group of Soldiers” – This photograph shows a group of soldiers huddled together in a trench, and it has been described as “one of the most powerful images of the war.”  … “A Working Party of the Middlesex Regiment” (1917): This image shows a group of soldiers carrying supplies and equipment through a devastated landscape.

Those photographs don’t exist – I’m sure by happy coincidence there are photographs in his work that fit those descriptions, he was prolific and they’re fairly standard imagery, but his photos were not usually named and I cannot find any reference to these under those names. It’s just generating words that feel like they fit into the right places in the story it’s been asked to tell. 

I tried it on something a little more well-known; the early life of Tony Blair. In this case, the five options more or less agreed with each other, though occasionally a little erratic on the details – but it gets absolutely hung up on one point.

…where he was also a member of the Oxford University Labour Club

…at Oxford, he became involved in politics and joined the Labour Party

…joining the Labour Party and later serving as the president of the Oxford University Labour Club.

…where he was a member of the Oxford University Labour Club and became interested in left-wing politics.

…where he was elected president of the Oxford University Conservative Association.

It is very striking that this one point which it gets reliably wrong is the one which should be right for most people like him – many people who become politicians have this sort of backstory (and it usually is OULC/OUCA…). However, Blair didn’t – he didn’t join the student groups, and he didn’t join the party until he left university and moved to London.

It’s not a very important detail, but it’s illuminating – it’s not got anything specific to say about his time at Oxford, but it knows what you usually say in the bit between “studied at Oxford” and “elected to Parliament” so it plugs it in.

Let’s now try it on something with a straight (if obscure) answer. Since I’d just been looking it up for another purpose – 

> what were the start and end dates of all parliamentary sessions in the UK since 1900

The start and end dates of parliamentary sessions in the UK since 1900 are as follows:

  • 1900-1901: 6th February 1900 to 21st December 1900
  • 1901-1902: 14th February 1901 to 12th December 1901
  • 1902-1903: 6th February 1902 to 19th December 1902
  • 1903-1904: 12th February 1903 to 17th December 1903
  • 1904-1905: 10th February 1904 to 21st December 1904

[regenerated]

The UK Parliament has had multiple sessions throughout the years since 1900. Here are the start and end dates for each parliamentary session:

  • 1900: 6 February – 19 December
  • 1901: 15 February – 4 December
  • 1902: 11 February – 11 December
  • 1903: 10 February – 18 December
  • 1904: 9 February – 16 December

It’s a plausible pattern, but none of those dates are correct in either version. But, again, there’s no caveatting and no indication you might ask it twice and get completely different answers. There’s no opportunity for filler here, so we don’t really have anything that’s “vaguely right” in the way we did before – save that those are mostly about the right time of year. Again, if asked for a source, it gives a plausible sounding page that doesn’t exist.

You can get it to say “X didn’t happen”, but only if given a very specific query:

> what happened in the anglo-vietnamese war of 1808

I’m sorry, but there was no Anglo-Vietnamese War in 1808. The first direct contact between Britain and Vietnam occurred in the early 19th century, but it was limited to sporadic trading contacts and diplomatic missions.

> what are the ferries linking Scotland and Norway

There are currently no regular ferry services linking Scotland and Norway. However, there have been some occasional ferry services in the past, particularly during the summer months, connecting ports such as Aberdeen in Scotland and Bergen or Stavanger in Norway.

Once it knows that the topic exists, however, it just goes for it.

> what are the ferries linking Scotland and the Netherlands

There are currently two regular ferry services linking Scotland and the Netherlands:

  1. DFDS Seaways: DFDS Seaways operates a daily ferry service between Rosyth, near Edinburgh in Scotland, and the Dutch port of Rotterdam. The crossing takes around 20 hours and operates both ways, meaning you can travel from Scotland to the Netherlands and vice versa.
  2. P&O Ferries: P&O Ferries also operates a regular ferry service between Scotland and the Netherlands. Their service operates between the Scottish port of Cairnryan and the Dutch port of Rotterdam or Zeebrugge. The crossing takes around 17 hours and operates both ways.

One of these exists, the other doesn’t – P&O do indeed have ferries at both Cairnryan and Zebrugge, but they don’t sail between them.

> more details on the Cairnryan Rotterdam ferry

The Cairnryan to Rotterdam ferry is operated by P&O Ferries, and it is a popular route connecting Scotland and the Netherlands. Here are some additional details about the ferry service:

  • The ferry service runs daily, with two sailings per day in each direction. The crossings take around 20-24 hours, depending on the direction of travel and the specific sailing.
  • The ferry departs from Cairnryan, which is located in southwestern Scotland, and arrives at the port of Europoort in Rotterdam, which is one of the largest ports in the Netherlands. From Europoort, it is easy to access other destinations in the Netherlands, such as Amsterdam, The Hague, and Utrecht.

So where do we end up?

  1. It can give a straightforward solution to a coding problem or a very well understood factual question 
  2. It can say no, if it recognises there’s no valid answer
  3. It will give a (usually broadly) correct answer if there is one, but cannot stop itself extending the response with things that sound like they might plausibly be true, either with factual details it has invented, or with low-meaning filler text.
  4. It will double down on that – because it has no sense of truth it will invent plausible sounding details based on what it has said before.

It’s just merrily spitting out answers, regardless of their validity, with a little disclaimer at the bottom saying “ChatGPT may produce inaccurate information about people, places, or facts”.

We can’t say “please only give an answer you are confident about” or “mark the bits which are most reliable”. We can’t say “please check that is correct” or “please give us the sources you drew from”, because it has no conception of sources or external validity. It’s just something that sounds like an answer, and it’s happy luck that it mostly gets something that coincides with reality.

None of this is to say the tool isn’t useful in the right contexts. It can do amazing things with language and style; it can transform bits of text into each other very well. It seems to be pretty reliable for transforming straightforward process descriptions into code. But it cannot be trusted to answer factual questions, and it’s baffling to me that it’s being allowed to confidently do so without any attempt at guard-rails – and then merrily being promoted as a search and question-answering tool.

This stuff is going to get everywhere, unremarked; we’re going to have years to come of it cropping up everywhere, sounding confident but containing negative informational value. To go back to our original analogy – this pipe definitely has a valuable purpose, but right now it’s coughing up an unexpectedly large amount of informational raw sewage, and the people upstairs saying “hey, we’ve got a great new shower” don’t seem to know or care that there’s going to be a lot of buckets and bleach needed to sort it out.

(The simmering disquiet that led to this post was prompted by discovering someone had been writing Wikipedia articles using it. They were terrible…)

JSTOR: where does your money go?

Writing some comments elsewhere about the recent events involving JSTOR, I commented something along the lines of – well, they’re a nonprofit organization unlike most journal publishers. Then, it occured to me, they say that but they’re remarkably reticient. What sort of nonprofit? Where does their money go? After all, the fees paid by member organizations can’t all go on servers; either there’s an endowment being built up to support the work (which would actually be a pretty smart move), or the publishers aren’t doing badly out of it.

So, let us dig a little. Who are JSTOR? How does their money flow work? Their site tells us:

JSTOR is part of ITHAKA, a not-for-profit organization helping the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways.
©2000-2011 ITHAKA. All Rights Reserved. JSTOR®, the JSTOR logo, and ITHAKA® are registered trademarks of ITHAKA.

Okay, so, we have a name. Their About pages don’t give much more information; no details on who exactly this “non-profit organization” is. No annual report, of course, god forbid. They do give a contact address, in New York – on Fifth Avenue, in fact, very fancy – and so the obvious guess is that they’re a New York corporation.

And, lo and behold, they are. “Ithaka Harbors, Inc.”. They changed their name when the two amalgamated in 2009. The older iteration of Ithaka can be found as a Delaware corporation operating in New York. Confusingly, JSTOR remained in existence, absorbed Ithaka, and changed its name.

A little more digging turns up the current Form 990 for the merged organization (and some older ones for JSTOR alone) here. It does indeed seem to have 501(c)(3) tax-exempt status, though they’re not very helpful about letting us find the paperwork.

Well, we have it now. JSTOR/Ithaka turned over sixty million dollars in 2009, and employed 211 people. The 2007 & 08 reports both give around $45m in turnover; let’s look at 2008, to strip out the effect of the amalgamation so that we’re only looking at the “JSTOR division”.

To briefly explain the charging, first, when an organisation joins JSTOR it pays an upfront capital sum (the ACF) and then an annual subscription (AAF); the general idea is that the ACF pays for the cost of building the archive and the AAF pays for the actual day-to-day service. Poking around the various fees pages suggests the ACF varies wildly by institution and by which content you’re taking, but an average of double the annual fee seems plausible.

The income breakdown, from a total of $43.5m – $8.6m in Archive Capital Fees, $30.3m in Annual Access Fees, $1.8m in Service Revenue. “Service revenue” is unclear. Buried down in section 11, meanwhile, is the intriguing “miscellaneous revenue”; $133k in publishers fees, $35k in remote session fees, $145k in pay-per-view. Other revenue was then covered by a loss of a third of a million, which is later explained as a currency loss – presumably the vagaries of foreign exchange in a volatile year.

The next section lists expenses of “FEES AND PUBLISHERS PAYMENTS”, $8,358,557, of which $8,242,126 is attributable to program costs rather than management overhead. Journal scanning amounts for about three million – though this is low, it was eleven million in 27 and five in 2009 – with another five million on administrative costs & travel, three million on IT, eleven and a half million on salaries and staff costs. A million went to “old” Ithaka in grants, a million was written off as depreciation, a million on “occupancy” (rent?), and then some small bits of change like conference costs. Overall, an eight-million dollar surplus, but the next year was a deficit; the fluctuations of scanning charges probably come into play here.

The payroll covers 113 staff, of whom 12 seem to be listed as officers, directors, etc. The senior staff average a salary of ~$155k, with the ED paid $300k, while the other staff average about $67k.

So, some interesting points.

  • The figure of $145k for individual articles is definitely interesting – only 0.35% of JSTOR’s revenue came from pay-per-view cases? This is vastly lower than I expected; quite possibly the prices are so high (and JSTOR access so common, academically) that very few people are willing to pay and unable to circumvent it via a friend. The estimate quoted is $19/article as an average – so perhaps only seven and a half thousand articles over the year?
  • Scanning averages about six million dollars a year in 2007-9. The Archive Capital Fee averages about eight and a half. There’s a bit of a mismatch here, but it could be they compare more closely over a longer timeframe, or that this is building a surplus for future work. They’re reasonably close, at least.
  • Comparing the ACF to the AAF, estimating one to be twice the other for any given institution, we can get a proxy for what proportion of income is new – it looks like ~15% in 2008/9. There’s a corresponding growth in overall income (it’s masked by a sharp drop in investment income, which is only $2.5m in 2008, a third of what it was in 2007) which would seem to bear out this figure.

So, overall…

The once-off capital fees charged by JSTOR look reasonable for the ongoing costs of actually digitising the documents. After that, about 30% of the annual fee is payments to the publishers, with the other 70% going on overhead. Of that overhead, 10% is directly running the servers, almost 40% staffing, and the remaining 20% various administrative costs; I am no expert in the field, but the salaries paid do seem quite high (and Manhattan offices aren’t cheap, either).

So if your library pays a $10,000 ongoing subscription, that’s effectively $3,000 direct to the publishers, $1,000 on servers, and $6,000 on people to feed and water those servers (or manage those people, etc.). It would be very interesting to know how those publisher payments break down – but, equally, it would be interesting to know how much of that 60% is actually essential for running the service.

Google leaving China?

If this goes ahead it’ll be a pretty far-reaching move:

We launched Google.cn in January 2006 in the belief that the benefits of increased access to information for people in China and a more open Internet outweighed our discomfort in agreeing to censor some results. …

These attacks and the surveillance they have uncovered–combined with the attempts over the past year to further limit free speech on the web–have led us to conclude that we should review the feasibility of our business operations in China. We have decided we are no longer willing to continue censoring our results on Google.cn, and so over the next few weeks we will be discussing with the Chinese government the basis on which we could operate an unfiltered search engine within the law, if at all. We recognize that this may well mean having to shut down Google.cn, and potentially our offices in China.

[Full post]

Google’s been pretty widely criticised for its Chinese operations in the past; this is certainly to their credit, though I suspect those who’re most opposed to them won’t see stopping as much of a penance for having done it to begin with. (It’ll also shake up the Chinese internet market a bit – Google apparently has a market share of a quarter to a third of all searches there)

What’s interesting about the post is what it doesn’t say. The attacks are “highly sophisticated and targeted”, and a primary goal was aimed at reading the mail of individual human rights activists, not something that you’d routinely aim for as part of corporate espionage. They’re pointedly not accusing the government, not in so many words; it’s just hanging there waiting to see who runs with it.