As with any new technology, social norms around the use of GenAI are still evolving. One of the concerns is the misuse of this technology in domains such as academia, which place a premium on the publication of new original research. As is so often the case, a metric has become a target, and that target is being gamed by people trying to generate more publications for themselves in order to accelerate their own careers.

It’s worth noting that the issue is not of someone using a hypothetical model to explore novel molecular structures or whatever — although there are issues with the actual results behind the headlines. The problem is when the papers are written largely or entirely by LLMs.

Old-school typewriter

Of course the first step is detecting when an AI has in fact been used to produce a piece of text. I am instinctively suspicious of claims that specific word choices or sentence structures might be a telltale sign of the use of GenAI, not least because I am an inveterate user of the humble em-dash, despite the recent accusations that only LLMs use it. After all, LLMs are trained on a corpus of grammatically correct and well-structured text, so it is not that surprising that their outputs might come across as perhaps more erudite and better-structured than the baseline of text on the Web.

Regardless, it appears that there really is a problem of articles being published in (formerly?) reputable scientific journals that have not only been written in this way, but so poorly edited that they still contain actual tell-tales — such as the phrase “I’m very sorry but I don’t have access to real time-information or patient-specific data as I am an AI language model.”

The journal Nature recently surveyed more than 5,000 researchersand asked when, if ever, is it OK to use A.I. to write a paper.

For the situation analyzed in the new paper — writing an abstract — just 23 percent of the Nature respondents said it was OK to use A.I. without acknowledging the assistance. Some 45 percent said it was acceptable only if the researcher reported using A.I., and 33 percent said it was never acceptable.

Here is what boggles me: even if you are using GenAI in writing the article — perhaps because you are writing in a language you don’t speak well — the abstract is the last place you should be relying on GenAI.

The whole point of the abstract, or its cousin in the corporate world, the “executive summary”, is to get across the key points of your argument to people who might never read the rest of your document.

GenAI on the other hand will probably flatten the paper out into a paragraph or two, but there is no guarantee that the result will actually contain what is most important — or of course that hallucinated/confabulated material has not been inserted from elsewhere!

Sure, GenAI is mostly pretty good at summarising information, but readers can do that for themselves. Authors should be much more intimately familiar with their own data and should be able to distill what is actually important.

[…] since everyone has chatbots these days, why publish anything written by them? Anyone who wants to see such an analysis can generate one for themselves.

Or as I say, if you could not be bothered to write it, I certainly cannot be bothered to read it. Otherwise we end up in the ridiculous situation of people using an LLM to expand three bullet-points into a verbose message and sending that to someone who will use an LLM to summarise it back down (lossily) to three bullet-points.

🖼️  Photos by Denise Jans on Unsplash