LLMs and LLM policies

[James G.]

I admit I set up a pretty hard-line position here by banning basically everything LLM-related without staff approval, but it was my preference at the time to avoid bot posts in what should be a discussion forum; the more LLM content I see in public reaffirms my concerns.

I'm a writer who also likes to do my own research; maybe I can't understand not being someone who is good at those things. But it seems people are in a rush to outsource their thinking to machines. I've already seen enough bullet lists, 'not x, but y', and points that are repeated ad nauseam, expanding a mustard seed of an idea to a sequoia of a post, to last me for all of this 'revolution'.  I'm about at the point where if you can't be bothered to write it, I can't be bothered to read it.

I grant there are good uses for LLMs, but I haven't been able to codify exactly what all those are yet, and in any case I'm not sure it's worth saccing the entire web to get them. Thoughts?

[Tester]

i totes dont agree jams. in teh past teh big biznez were takin ovr evthing!!1 teh lil guys cldnt compet w/all taht. but now it r bcomin fair... w/ai u can do wat u r good at nd not hav to b like 'oks i got tihs grate idea now i just need pro artists and pro muzakshuns and pro-gramars and mayb den in 3 years u can c it'. now houswifes can make workin programs and printouts in a day!!1 y b upset tat ppl can get tihngs done james???

also ur big on posts meetin standards... u dont like most o teh web as it r rite?? tats y u set up tihs site. well arnt bot posts clean and logcal?!? it r improvmnt!!! if u wuldnt wanna read ppls websites anyways den how can u b liyke 'lets get rid o all teh bot posts tey r so bad'. teh ai's r makin teh web better!!!!

im sooo good at writin taht i dont need ai but if otter ppl wanna use it tats ok w me... shldnt ppl be 3 to choos james?

[James G.]

in teh past teh big biznez were takin ovr evthing!!1 teh lil guys cldnt compet w/all taht.

This is beside the point of whether LLM content in posts should be accepted on a forum, but I'll engage. This argument presumes that there is a competition. It's true that small-time developers aren't making 3D first-person games, but they never did. Indie style and corporate style have coexisted for decades. Audiences have proven to be very understanding - and even appreciative - when developers work within their limitations.

but now it r bcomin fair... w/ai u can do wat u r good at nd not hav to b like 'oks i got tihs grate idea now i just need pro artists and pro muzakshuns and pro-gramars and mayb den in 3 years u can c it'.

It's true that difficulty of development hinders ideas from coming to life. I contend that LLMs do worse by forming only an idea, the original one. The person never goes on a journey where they struggle with the meaning and purpose of their work, and have no opportunity to learn or improve as they go. How many dramatic productions have been improved by the input of the actors, past what the writer or the director originally planned? This ensures that you get, at best, what you initially expected, nothing better.

now houswifes can make workin programs and printouts in a day!!1 y b upset tat ppl can get tihngs done james???

Those things may work, but that doesn't mean that they'll be good. By encouraging this kind of development, we deincentivize quality and deliberation, and by pushing from below, edge out people who offer correct implementations of something.

Incidentally, economically this is an example of taking money from below - small-time craftsmen - and giving it to "big tech" megacorporations, via subscription fees and government incentives for their work. If you are for the 'little guy' in any meaningful sense, you also have to be for the little guy who has taken the time to care about what he is doing.

well arnt bot posts clean and logcal?!? it r improvmnt!!!

They are generally clean, but just because I don't want to read profanity-laced tirades doesn't mean that covering up all human output with mass-produced bot material is the way to go. If I don't like a human's writings (or many people's writings) and don't want to read them as-is, I can avoid them altogether, or filter out the more objectionable parts on my own. But if I don't like bot writings, I am increasingly stuck.

Besides, even if bots are better - let's say more moral - in that specific area, they can be worse in other areas. For example, bots often say things that are incorrect or misleading. If we fill the web with that output, we tend to disadvantage truth. That's not to say that humans don't lie, or can't be mistaken, but there is some limit on how much they can do so. There's a different economy of scale with LLMs, our limitless lie machines.

im sooo good at writin taht i dont need ai but if otter ppl wanna use it tats ok w me... shldnt ppl be 3 to choos james?

They can choose, by going someplace else where that is permitted. But I would say that if I want Claude, ChatGPT, or Gemini's opinion, I should be free to ask it, on my own time, instead of automatically having to suffer through someone else's choice to either present bot writing as their own, or to clearly label it as such - the former being dishonest, the latter being disrespectful by implying that people should have asked a bot in the first place instead of actually talking to each other. Given that some decision must be made one way or the other, and that there are plenty of sites out there where LLMs are allowed, I don't find the freedom argument compelling here.

[Verdauga]

    I have to agree with James on this one, Tester. I haven't found a use for generative AI that isn't better done thro' some other method-- perhaps GAI is 'cheaper', but that's a different point to be debated.

Aside from that, the amount of misinformation, and the readiness of people I know offline to accept and run to GAI to solve their problems, makes me incredibly concerned.

My two personal largest issues with AI is that using reduces how much uses community-- it helps separate people-- and how people are less often learning how to do things themselves.

Now please excuse me while I go do long division on a napkin.

[Tester]

I have to agree with James on this one, Tester.

thats ok verd theres lots of time to chaenge ur mind!!1 but evn if u dont i think ull stil like my pwrful pts aginst james.

I contend that LLMs do worse by forming only an idea, the original one. The person never goes on a journey where they struggle with the meaning and purpose of their work, and have no opportunity to learn or improve as they go.

i disagre james. for 1 it r betr to get teh things tahn not. but seconly tehy hav pleny o time to improv... it r just after teh 1st tihng is done. ur dev is like 'ooh i hav lerned soooo much ovr teh curse o teh 4 years i took makin this sdl & C app that puts a snake on teh screen. i can do sooo many more things now!!1 like, mayb teh seqel can have 2 snaeks!!!! nd i figrd out long teh way, durin all teh time i cldnt figur out how to make it or didnt have teh energies to make it, taht the snak game needs food to growe teh snake!! i nvr thout o dat b4.'

werehas w teh ai, ur lik 'ok i got tihs snake on teh screen nd u can turn it w teh arow keys. now i can ad anotr snak and maiyb sumthin 4 teh snakes 2 eat.' it r done in 2 days!11

c james... u stil think o new tings to ad... teh ai's mak teh whole thng tak less time 2 make is all. mayb this r betr! u can actly giv things 2 ppl to use nd tehy can b like 'hey tat snake game r rly cool maybe it culd hav rooms u can go in!!!' w teh old setup, u r like 'tat r a gud idea, just wait anotr year and ill hav it', and tey r lik 'watev jams'. but w teh new one, u r lik 'that idea r sooo cool, here it is!11!' evbody is hapy james!11 well mayb evbody excep 4 u.

Those things may work, but that doesn't mean that they'll be good.

how manys things do a dev make durin teh dev proces taht r no gud james? do most o dem even work!?!?? ur sayin 'a program i made after alllll tihs time r betr tahn 1 sombodies made in a short amt of time'. tats wat wed expect james... but wat wld hapn if u took a long time usin teh ai's?!? u mite hav somethin tats rly good.

w teh whole delibrate deal. ur like 'oh i thot bout it rllly hard so taht meens it r gud'. u dont know evthing james... u dont evn know most tinhgs. most ppl dont eithr! mayb andy hertzfeld is a grate software guy who can do tihngs betr tahn teh ai's but ur not him!! dere r only a few ppl who r grate software ppl james. ur woryin all teh time bout 'oh noes wat if tihs r rly bad and sombodies or teh ai's break it???' if teh ai's break it mayb tehy can fix it too james... can u do betr shiftin ur car manual tahn teh automatic thing??

Incidentally, economically this is an example of taking money from below - small-time craftsmen - and giving it to "big tech" megacorporations, via subscription fees and government incentives for their work. If you are for the 'little guy' in any meaningful sense, you also have to be for the little guy who has taken the time to care about what he is doing.

how many clothesppl nd shoeppl nd furntureppl r u buyin from james? ya it r sad taht ppl who spent allll tehr time figurin out how 2 do somethin teh hard way r not goin to be gettin big advantge. but taht is how it is jams. u cant change it. anyways if teir skils r so grate like u say tehy can stil be sucesful. if waht tey do isnt any bettr tho than whats alrdy ben done then was it worth allll taht time?? think o how much time ppl spend lerning and doin these things james. by savin all dat time we r savin lives!11!

on the subscripts u know tat there are lots o ai's now. u dont need ai subscriptn anymore. tat r teh whole pt o teh big 'ai pc' thing james. there r llamas n tihngs. but evn if teh ai company is makin monies so wat? they r doin a public servic!! tehy help waaaay more ppl than u spendin 5 years on ur tictactoe game will.

If I don't like a human's writings (or many people's writings) and don't want to read them as-is, I can avoid them altogether, or filter out the more objectionable parts on my own. But if I don't like bot writings, I am increasingly stuck.

how many ppls posts do u read james? is it just a few? if it r just a few wat diff does it make if there r lotsa bot posts? ur goin to hav to go out o ur way anyway.

That's not to say that humans don't lie, or can't be mistaken, but there is some limit on how much they can do so. There's a different economy of scale with LLMs, our limitless lie machines.

wernt u just sayin taht humans r betr writers? if tahts so tehn mayb tey r bettr liars too.

ppl lie or r wrng all teh time james. tat is tru w human writins and w teh ai writins. it r how it r. i dont rly think theres much difrence... tho mayb teh ai's will admit tey r wrong.

instead of automatically having to suffer through someone else's choice

tihs r assumin ppl r goin to b usin the ai's in ways taht r obnoxus james... r tehy obnuxus??? isnt that just ur opinon?? i dont think im sufring

the latter being disrespectful by implying that people should have asked a bot in the first place instead of actually talking to each other.

no offens james but i think u are readin waaaayy 2 much into it. ur assumin bad thins when ppl r tryin to help.

so say some1 asks a questin lik 'heyy how can i get teh toilet to stop floodin?!?' i hav no idea but i ask chatgpt n it says 'ya u needa turn teh thing on the thing' and tehy do it and it works!!1 hows taht bad?

bsides waht diff do it make weder a bot said it? spose u say 'hai tester im lookin up teh captals o teh sovet states. i wonner how many tehre were???' i go look it up in my encycpedia and r like 'hai james teh encyclopeda britanca saiz there were 15'. hows that diffrent from lookin it up on a bot james? if its true its true isnt it? who cares wethre i got it outa my 30 vol set of encycpedas nd suppments or off chatgpt? isnt it beter to get teh truth than to b like 'wellll i dont got my encycpedia handy so i duno. sorries!1 mayb u can look it up somewheres'

[James G.]

I haven't found a use for generative AI that isn't better done thro' some other method-- perhaps GAI is 'cheaper', but that's a different point to be debated.

Well, I guess the question here is how we define 'better', and who is it better for. LLMs are rightly controversial, and people have a right not to like them or want to use them if they don't want to. But that's different from arguing about the qualities of some works.

To use an example here which you're likely to be familiar with, free software advocates are gung-ho about free software for ideological reasons. But some go further by saying that because it is free, it must be actually better - better features, better reliability, etc. Well, even the FSF says that proprietary software can be 'better' in areas other than in its licensing. Being morally/socially/etc. against something doesn't mean we have to mislead ourselves about its positive qualities.

In the most straightforward sense, of "human (programmers, artists, whatever) put out better work than LLMs", I'd say, yes, some of them do; that's objectively true. But judging by the standard we would have used to judge human work, many other humans do not put out better work than LLMs, at least in some area or another.

Of course, if our definition of 'better' includes being made by humans, then of course human work is always better, necessarily so. And it is fine if that is the definition we all agree on and want to use. But, fundamentally, that is no different from saying that the art a child made is better than the art displayed in a museum made by some dead guy from the 1600s, because we know and love our child. That connection may be a powerful reason for our preference, but what would we think of a piece if we didn't know (or couldn't reasonably guess) who made it?

Aside from that, the amount of misinformation, and the readiness of people I know offline to accept and run to GAI to solve their problems, makes me incredibly concerned.

When it comes to misinformation, I'm more concerned about fake photos, videos, and voiceovers than just writing. It seems like our culture in general has gone all-in when it comes to lying nowadays, and of course, people have always made mistakes or said things that were merely incorrect. But it's another level of problem when it becomes easy to portray someone as having said something they didn't. This problem existed before (Dan Quayle quotes are often either fake or taken unfairly out of context), but it's one thing to allege that someone did or said something, and another to 'show' them doing it. Once such allegations are made, then the burden of proof tends to shift towards the person being so portrayed to prove that they *didn't* say that. That's a major issue. In a few years, at the rate we are going, I wouldn't be totally shocked if video and photo evidence stops being admissible in courts.

In the other case, we have to consider that people have been trained to look things up online for years. It just so happens now that looking it up online is based around LLM tools now, because everything they would have used in the past is gone.

Suppose I had a problem in 2006 and wanted to get help. Well, the first thing I'd probably do is Google it. But Google - and search engines in general now - are just useless a lot of the time. This is where 'AI slop' really comes into play, because a lot of those sites clogging up the results pages are new and mass-produced, but that's not the only factor.

Forums? Fine, but it'll probably take a while before you get a response - hours or days or weeks - not great for short-term issues. Site directory? They're all dead. Instant chat? There's Discord, which lives up to its name. Other than that, they're pretty much dead. There's Facebook, but that's limited to people you know, and you always feel like you're bothering somebody, as opposed to a pseudonymous person across the world who presumably wouldn't reply if they weren't interested.

Bear in mind, now, that I am talking about the most general case here, LLMs for looking up tips, facts, and as a search engine replacement (which is more or less what you hit on in your complaint anyway). There are all other kinds of things that can go wrong, like people wanting to marry bots or other such nonsense, which are worth hitting on, but that's out of our scope (for now).

My two personal largest issues with AI is that using reduces how much uses community-- it helps separate people-- and how people are less often learning how to do things themselves.

It's true they do; you're not wrong. I think that was already happening before, and then the LLMs came along to 'fill in' (and further increase) the gap. I know that in my case, I've found it easier to turn to LLM solutions because of the absence of people and answers in the niche that I'm looking for. I think people also feel more comfortable consulting a mindless machine than random people who might judge or criticize them as readily as offer help. To be fair, I think StackExchange (which seemed to be the web's main source of human-to-human information, once upon a time) wasn't so bad at that, but on forums, it isn't unheard of to have somebody say that you shouldn't think that or you shouldn't want that or you shouldn't have posted without figuring it out yourself first.

On the latter, I do have a problem with trends that lead us towards not being able to understand how to think or to audit what bots and systems are doing. Somebody put up some snippet suggesting that their child wouldn't need much of a traditional education and that it would be more important for them to learn to 'be kind', now that there are LLMs. I honestly found that disgusting. Of course kindness is important, but it always was. It's one thing if someone turns out to be a dependent person in spite of everything, but it's another thing when you never gave them a choice. The child may turn out to be a kind person, but you're not being kind to them if you refuse to give them the ability to understand how the world works and how things got this way.

At the same time, though, LLMs needn't necessarily stop people from thinking. I think part of the issue here is marketing and deployment. LLMs have been made into a magic genie buried inside of a computer (complete with a sparkle icon announcing its presence), so that people feel encouraged to just rub the screen and have it fix all their problems for them. If companies had developed LLM technology 'quietly' and introduced LLM-based features into existing paradigms, I don't think we would have the same culture shock, or grounds for complaint.

i disagre james.

Of course you do.

for 1 it r betr to get teh things tahn not.

Is that so? I believe it depends on the thing.

Suppose we have a chess program, Opinionated Chess, made for the Amiga. It is a pretty good, though not perfect, program. What's better - to have a solid community based around this common thing, or to have 500 other chess programs as everybody designs something to fix their pet peeve with Opinionated Chess (though none of them fix all of its problems, or even implement all of its features)? Sometimes the best thing is for someone to get used to what is already out there, instead of reinventing it.

Is it better to have a few masterful paintings of the Apostle Andrew, or 1,000 okay pictures of him?

Is it better to have a great novel or one hundred mediocre dime novels?

I might agree that it's sometimes better to 'get the things' when it doesn't exist and there is effectively no probability of that thing being made otherwise. For instance, people have had 25+ years to make an Apple menu applet for Xfce Panel; if they haven't done it by now on their own it seems doubtful they ever will. So I'm not necessarily universally adverse to help in that area (obviously, or else I would not have used LLM tools to make one).

But it is far from clear that this same pattern holds when there are a lot of alternatives out there. There may be a lot of time spent making something, but at least that time is spent by one person who has thought about it. I do think that is qualitatively superior over one thousand people who haven't thought about it, and moreover, I think that it is worth protecting those users from themselves, who may otherwise suppose that they can do better for 'free', and perhaps form their own little splinter groups, rather than supporting the person who took the time to do it right. And there is, of course, value in the shared experience of knowing that others have used or seen or read something, and liked it. That effect is so strong that it has built every fan community that has ever existed.

but seconly tehy hav pleny o time to improv... it r just after teh 1st tihng is done... c james... u stil think o new tings to ad... teh ai's mak teh whole thng tak less time 2 make is all.

Is that really 'all'? In theory, maybe it could be. But in practice, with actual people, I just don't think that's how it works out.

The first thing we have to take into consideration is that creating a work, any work, is about the details. Small things make up great things. We know from observing people, or reading books, or watching games, that if a trait or a sentence or a rule were a little bit different, it just wouldn't be quite the same. But LLMs tend to accommodate broad strokes. Now, I could give precise instructions in pseudocode and reasonably expect the bot to translate them to the real thing (albeit a year or two ago, that task would not have had a high probability of success). But I can also say "make a program that does this". If I don't provide the details and the formula, it will fill in those blanks on its own, even when those details are essentially the core of the program. If I say "make a sim that generates match scores for hypothetical matches between two chess players", and expect the bot to work out the formulas and details for how to do that, I will get more or less what I expect. But that's not another way of doing programming, much less designing, that's at best the CEO of the company giving an employee an idea. That is not conducive to building, or maintaining, a creative, detail-oriented mindset. It is a process that produces things quickly, but those aren't 'the same things faster', any more than microwaving frozen burritos is the same as actually cooking meat and beans and tortillas and mixing them together.

u can actly giv things 2 ppl to use nd tehy can b like 'hey tat snake game r rly cool maybe it culd hav rooms u can go in!!!' w teh old setup, u r like 'tat r a gud idea, just wait anotr year and ill hav it', and tey r lik 'watev jams'. but w teh new one, u r lik 'that idea r sooo cool, here it is!11!' evbody is hapy james!11

It is potentially faster. But I'd point out that developing isn't just knowing what to put in, it's also knowing what not to put in, and a lot of the time, knowing that takes some kind of experience, experience which is very hard to develop when you don't actually think through the problems on your own, but hope that the AI will fix it for you.

Don't get me wrong. We can't make hard-and-fast universal rules for everybody about these issues. If you have a critical attitude, like Steve Jobs, and go in expecting that the LLM did everything wrong, and searching and looking for things and areas where you can tell it that it was wrong, you might actually be able to get by using LLMs without a significant loss in your critical thinking ability (both in the most direct sense, and in the general sense). The trouble is, I'm not sure that this is a healthy mindset to have in the long-term. It certainly isn't one I would want to seep over into my personal life. It is not how most people think, or want to think. They want to be basically agreeable, and are willing to compromise to avoid a confrontation. That's perhaps fine when you're compromising with someone you care about, but we shouldn't set up scenarios where people are likely to compromise with bots.

but wat wld hapn if u took a long time usin teh ai's?!? u mite hav somethin tats rly good.

Possibly. Will people actually tend to do that? "Good enough" is the enemy of "good" (much less "perfect").

dere r only a few ppl who r grate software ppl james. ur woryin all teh time bout 'oh noes wat if tihs r rly bad and sombodies or teh ai's break it???' if teh ai's break it mayb tehy can fix it too james... can u do betr shiftin ur car manual tahn teh automatic thing??

In principle I don't think this idea is actually wrong. Based on our experience so far, and from science fiction (which is conveniently, in part, something of a collected body of work about dealing with LLMs), there's no obvious reason why bots couldn't become better at certain skills than most people. I think that is already true in certain cases; reportedly Claude Mythos is far better at hacking than virtually any human.

That said, I would point out a couple of things here. First, the point in science fiction where bots get better skills than their developers is typically around the same point where they start getting out of control. In fact, if we think about what being 'in control' means, that is necessarily true. Once you no longer have the skill to keep up with and understand what the bot is doing, you are no longer in control of it in the strictest sense of the term. That doesn't mean that it is plotting anything nefarious or destructive, but you are banking on its output having been in line with your intent, because you can't fix it if it wasn't.

Secondly, there are some areas where we know the bots are wrong, or 'bad'. I am no Brian Jacques or Louis L'amour, but I'm still a decent writer myself, and I know a good piece of writing when I see it. Bot work isn't it. I've seen bot-written work on a wiki, and even when the facts are there, it's worthless as body. We're going to have to go through all of that and write it all pretty much from scratch. If bots are clumsy, inhuman and bad in the things we do readily understand, how can we then trust that they are actually great in things we don't understand? I grant it's possible, but I'd rather not bet anything important on it for a good while yet.

how many clothesppl nd shoeppl nd furntureppl r u buyin from james? ya it r sad taht ppl who spent allll tehr time figurin out how 2 do somethin teh hard way r not goin to be gettin big advantge. but taht is how it is jams. u cant change it. anyways if teir skils r so grate like u say tehy can stil be sucesful. if waht tey do isnt any bettr tho than whats alrdy ben done then was it worth allll taht time?? think o how much time ppl spend lerning and doin these things james. by savin all dat time we r savin lives!11!

I did get some expensive handcrafted shoes a while back, and there's a significant piece of furniture in my bedroom that was handcrafted.

I grant that there are inevitable economic pressures, and that a genius can probably be successful in spite of that. The problem here is when our process deters developing that genius in the first place. We could well end up with a dragonish generation, who, as Thorin says, "hardly know a good bit of work from a bad, though they usually have a good notion of the market value".

u dont need ai subscriptn anymore... but evn if teh ai company is makin monies so wat? they r doin a public servic!!

Maybe not, on the subscriptions. But I must say I am skeptical of the service angle. These companies are using up good land and water, kicking thousands, and potentially millions, if the habitually hyperbolic claims of their leaders come true, of people out of jobs, while helping build a surveillance state, and funding hypocrites who claim that their machines can do their jobs better than they can (notwithstanding they haven't actually given up any of their control that I know of), imagine that LLMs deserve equal rights with humans, and dream of one day making their own gods. What exactly is good about all this? At least the Romans gave bread with the circus.

tihs r assumin ppl r goin to b usin the ai's in ways taht r obnoxus james... r tehy obnuxus??? isnt that just ur opinon?? i dont think im sufring

Well, you can't be suffering here because it's banned.

I have seen enough bot writing in public settings that I can fairly say that I believe it is obnoxious. It's one thing for someone to be making, say, stories or scripts with bot output - that's bad form and style, but whatever; it is their loss. But it's another thing when they try to make it my loss. When somebody is attempting to argue a point, and it is clearly bot-written, that's about at the level of 'tldr' in terms of contempt and disrespect. "You are so wrong that I couldn't be bothered to think about this myself or form my own arguments; here is my answering machine to win the debate against your self-evident stupidity." In fact, I'm not sure if I wouldn't prefer tldr, since at least it is efficient in the space used for the thought expressed (bot posts rarely are), and the person cared enough to type the characters in themselves.

On the flip side of the emotional spectrum, while I've never gotten a love letter, there is a big difference between the sincere feelings of your dearly-be and the sincere scribblings of ChatGPT. If a young woman bought me a card, I at least know that she went to the store and looked through the cards and picked one out and spent good money on it. That's a far cry from "hey chat, write me a love letter for somebody named james". I would find the latter obnoxious, and I think most people would too. If you're not one of those people, this site is probably not for you.

no offens james but i think u are readin waaaayy 2 much into it. ur assumin bad thins when ppl r tryin to help.

Maybe so; I grant that support. LMGTFY may have skewed my view a little in this area.

so say some1 asks a questin lik 'heyy how can i get teh toilet to stop floodin?!?' i hav no idea but i ask chatgpt n it says 'ya u needa turn teh thing on the thing' and tehy do it and it works!!1 hows taht bad? ... if its true its true isnt it?

In this specific case which was manufactured so as to compel me to agree with your point, I'd agree that it sounds like a good thing was done. But most situations that invite advice in online forums are not time-sensitive with easily testable solutions that have no major probability of backfiring in a significant way if it fails. Sure, if you are stuck in that specific situation with the toilet, then you might as well follow bot advice as stand there and watch your floor get covered with sewage. But that's not the norm.

If the information is true that's one thing, but the very point in question is whether it is or not. I would prefer you to take the time to tell whether it is true or not, based on an actual source. And in some cases, where the thing being tested is not really testable other than in person, a bot opinion is worthless. A bot knows nothing about whether a camera is actually any good; I need real users to tell me that. For all I know the bot got its information from the company's advertising page.

We may evaluate whether some LLM things are acceptable down the line, but I would rather start out on the conservative side and decide, at length, that certain things are okay after all, than to rush in, allow everything, regret it, and then have a tricky time chasing down all the myriad evils and forcing them back into the box they were let out of. 'Move fast and break things' may work for certain start-ups, but I think human relationships and discussions are broken enough as it is.

[Verdauga]

One surprising ability of LLMs is their ability to find faults in computer systems, as most recently highlighted by the PHP2shell situation that happened the other day. One could argue that this wouldn't be an issue if LLMs weren't a thing, but they are, and it is. Besides, there's no real way of knowing if this vulnerability was known and kept under wraps, and OpenAI just exposed it.

Of course, that's quite the niche use for LLMs, and still doesn't validate all the resources they hog, but at least they're not total drains on society.

[James G.]

One surprising ability of LLMs is their ability to find faults in computer systems, as most recently highlighted by the PHP2shell situation that happened the other day. One could argue that this wouldn't be an issue if LLMs weren't a thing, but they are, and it is. Besides, there's no real way of knowing if this vulnerability was known and kept under wraps, and OpenAI just exposed it.

Well, suppose someone went around trying to set every house he could on fire, and then calling the fire department if the house turned out not to be fire-resistant. I would have a hard time commending him for his great public services in fire-risk assessment and prevention!

At least some of the PHP problems, I believe, were theoretically known about before, but it was not believed at the time that they were practically exploitable. That may have been true - until now. Thus, we can see that LLM technology has 'helped' us by making it easier to engineer exploits for defects that were previously not hazardous under normal operating conditions.

The problem here is that the threat model shifts up tremendously for anyone involved in network operations. Back in the day, there was a ton of networked software (games especially) that was probably not objectively secure in the modern sense, but the only people who could hit on them had better things to do with their time. Business software, etc., tended to be frequent targets, but for something like a video game, the worst thing you had to worry about was a script kiddie getting mad about their lack of skills in a game and trying to get revenge with their 'skillz'. And if you were in the 5% of people using a Mac, you enjoyed peace of mind knowing that, with help resulting from its complete lack of a shell, there are virtually no live exploits against your platform - security and obscurity that even the US Army tapped into for their servers.

Now, the profile we're looking at is getting more to the point where everybody running networked software potentially faces what could have been nation-state level attacks a decade or so ago. And these attacks all matter, because our society is set up around networked software and increasingly refuses to let people handle things by 'old-fashioned' means. So there is much more incentive for an attack, and if one gets through, the ramifications are catastrophic. Not too long ago, we could rest easier knowing that they had at least gotten very difficult to engineer. But now, no thanks to LLMs, we don't even have that.

Perhaps the software that's being made today is objectively more correct than in times past, thanks to the help of these tools, but that's at the expense of things never any longer being 'just good enough'. This may be the reality we live in, but I don't think this can be credited as a net gain for us.

[Tester]

...many other humans do not put out better work than LLMs, at least in some area or another... what would we think of a piece if we didn't know (or couldn't reasonably guess) who made it?

sooooo rite jamse!!!! i agree completlyl!!

Sometimes the best thing is for someone to get used to what is already out there, instead of reinventing it.

when r 'somtimes' james? u cld say 'well teh ibm pc r out, it r by ibm, wich r a good compny!!! evbody shuld just use tat and teh dos 4ever cuz nothin r rong w tehm'. wout reinventin things nvr get good james... u kno this...

I might agree that it's sometimes better to... make an Apple menu applet for Xfce Panel... I'm not necessarily universally adverse to help in that area

so basicly llms r evil exept 4 wen they get u waht u want. isnt taht specal. y cant otter ppl hav things 2 james!??!?

I think that it is worth protecting those users from themselves, who may... form their own little splinter groups, rather than supporting the person who took the time to do it right.

ur actin like evbody r all goin to go make allll teihr apps w teh ai's and tahts not true james... ppl dont wanna spend time makin apps they wana get things done or have fun. all teh ai's mean r taht ppl arnt stuck w 1 bad app or no apps.

that's at best the CEO of the company giving an employee an idea... It is a process that produces things quickly, but those aren't 'the same things faster', any more than microwaving frozen burritos is the same as actually cooking meat and beans and tortillas and mixing them together.

so wat james? if ppl can make teh rly creativ stuff others can stil use it if tehy want. how many times hav u et teh froze buritos??? somtimes ur like 'oh i want good stuf' nd u eat taht. ottr times ur like 'wel i just wanna eat somethin and here these r'. wats wrong w ppl havin taht choic james?? ur just sayin ther shld only be 1 choicxe.

We can't make hard-and-fast universal rules for everybody about these issues.

but tahts just waht u want james. ur like 'nobodies shuld hav it, it r so bad'. ppl etter got teh rite to chose or tehy dont james... u cant hav it both ways...

Once you no longer have the skill to keep up with and understand what the bot is doing, you are no longer in control of it in the strictest sense of the term. That doesn't mean that it is plotting anything nefarious or destructive, but you are banking on its output having been in line with your intent, because you can't fix it if it wasn't.

ru writin allll ur software james? teh os and teh langage and teh compler and teh libaries aend teh tolokits and teh networks?? evbodies always gotta dpend on somthin tehy dont undstand if theyr gona get anythin done. wat diff does it mak if it r from tem some1 else or from ai's? teh ai's prob arnt putin backdoors in it like in teh xz afair!!

I'd rather not bet... on it... yet.

ok jams. lets wait 4 it to get better - *2 years pas* - so wat do u say now?

seems 2 me like ur just pushin teh dcisin back.. waht more do u needa know?

...a genius can probably be successful in spite of that.

wait wat?!? u ju8st spent tihs time sayin tehy cant!!! wich is it? i rly dont kno waht u mean anymore jams... do u know?

The problem here is when our process deters developing that genius in the first place.

in teh past ppl were all lik 'oh noes ppl cant program usin 0s n 1s, or in teh asmbly lang. teh art o programin r dead!!!'

skils bcome les used james but not ev1 forgets tehm... some ppl alway lern tehm weder 4 fun or for nitch job. but otr ppl lern new skills!! teh ppl who lern teh new skills likly wernt lrning teh old skils in teh 1st place cuz it were 2 hard... ur lik 'it r skills or ai's!!!' but it r rly a choic btwen no skils or skils usin teh ai's.

hypocrites

so it r evil cuz teh envinment may be hurt and teh ownrs r mean. do u use or ride in any cars james?

Well, suppose someone went around trying to set every house he could on fire, and then calling the fire department if the house turned out not to be fire-resistant.

4 sham jams... u kno taht dbugin apps r not teh same as destroyin evthing.

That may have been true - until now. Thus, we can see that LLM technology has 'helped' us by making it easier to engineer exploits for defects

ya mayb it r easer 4 ppl to make grate explots james. but that r cuz it r easer 4 ppl to make grate software o all kinds! now u can mak apps that r rly secure nd not jus fake scure!!11 how can taht b bad?? b4 u were like 'oh noes i betr not make my netwrkd app cuz it r goina b insecr and evbodies will hac into it'. but now ppl r goina be like 'ok i just did teh big secrity test w teh ai's nd evthin looks gud'. taht boosts confdence! ok so mayb rite now teh atacks r bad. but teh defnses r gona get betr and in teh end theres lots o rly good apps!

sorries if i said anythin ofensiv james. but tehse devments r rly exiting!!1 i wanna help u liten up.

[James G.]

sooooo rite jamse!!!! i agree completlyl!!

...On second thought, there's a lot to be said for the intrinsic value of human work.

when r 'somtimes' james?

Wouldn't we all like to know! If we could reliably foresee when it was best not to change things people could have been spared a lot of grief.

You have to figure what's being added against what's being lost, including the community effect. You also have to figure what direction things are likely to go in after these additions. A lot of the time, the interim steps are okay or at least tolerable in and of themselves; it's only where things end up after a number of iterations that there's a problem. But this branches off into a somewhat different subject than what we are discussing here.

wout reinventin things nvr get good james... u kno this...

Fine, but what happens when they're already good? When something is already good or great, changing a lot of things will probably make it worse, not better.

I think it's fair to say that once something is entrenched, you need a good reason to abandon it. I'm not saying that those good reasons are necessarily hard to come by. But while there may be a role and a purpose for 7 separate chess programs, we don't need 490. I'm saying that before someone creates something, they need to sit down and count the cost and see if what they're doing is really going to be any better than what there already is. Or to think about it from another perspective, a good scientist doesn't independently perform every piece of research; they look and see what's already been published, and take advantage of those findings, even if perhaps the papers aren't exactly how they would have written them.

swing swing, punch punch

Most of the rest of your arguments ultimately boil down to "not everybody will want to use this; people should have choices; it's just another tool".

You know, although this comparison didn't occur to me until just now, this situation seems a lot like the shoutbox fiasco a few years ago. It took off because it seemed easy and offered instant feedback. Some people used it a *lot*; others hated it and disavowed its existence; a few were somewhere in between. Over time, though, we figured out that the environment amplified users' bad behaviour, it cannibalized activity from the regular forum, and it didn't add a lot in the way of quality conversation (even if it had, by its ephemeral nature, a lot of that would have gone by the wayside, at least if I hadn't ultimately shared the logs with everyone).

The argument that its biggest supporters tried to use is that you didn't 'have' to use it if you didn't want to. Technically that was true, but the effect it had for people who didn't want anything to do with it was not neutral, because (as previously implied) so much of other users' activity was being funnelled into it. They lost posts, threads, whole members, and got absolutely nothing in return for it.

LLMs and LLM tools are easy and offer instant feedback, and I won't deny that some of the tricks they can pull off seem nearly miraculous - at least at first glance. They can be useful to people who are careful and know what they're doing; I've seen good LLM-assisted work and I like to think I've participated in it myself. But what we mostly get is the notorious slop. Bot pictures that are all in the same style, don't mind any details, and have the telltale blotches that indicate bot generation; bot writing that tends to either parrot back what the user said as bullet lists, or uses umpteen "not x, but y" structures; bot conversationists that agree with the user on everything significant, and 'correct', nitpick, or argue over minor details where the bot is wrong and/or misunderstood what was being said; bot 'analyses' that just rearrange facts and pretend to be presenting new information; bot wiki articles that miss the style and mention the same points over and over again; bot 'music' with fake singers who pretend to sing about experiences they could have never had; bot videos that go on for hours, with bot narrators, repeating the same plots or ideas and using every single bad bot writing trope ad nauseam; bot sites that bottify information they stole from an actually good website (or book), while pretending to be human-produced. It's all trash, and it all seems like it's everywhere!

Coding is the main thing that I would say bots are pretty decent at, but while I'm not a world-class programmer, I'm still good enough to have been able to catch them doing things that are very stupid. My favourite stupid bot moment was when I wanted a curated list of fonts in a text editor. I picked the fonts in advance, and the editor was supposed to list them based on what was available in the system. Absent my clear instruction on how it should perform the search, the bot decided the best way to do it was to check every font on the system to see if they matched the list. This is self-evidently about the dumbest way possible to do it that still technically solves the problem. (The correct way is to check for the curated fonts themselves.) If it does such stupid, vacuous, fake, and careless things in things I understand, why in the world would I trust it with things I don't understand? I get the microwave argument, but for crying out loud - we are starting to realize how bad factory-produced meat paste food actually is; must we bring it to the world of ideas too? All this without even bringing up the utter foolishness resulting from people who no longer know how to relate each other, and break up their real relationships over this fad.

I have a problem with turning the internet, which used to be useful, into a book of sands for LLM output - much less the world at large. It seems like it's too late now, but what has been done with Claude Mythos is probably what should have been done with LLM technology as a whole to begin with. If you wanted a sharp opinion, there it is.

[Tester]

Fine, but what happens when they're already good?

ok jame but who r to say when it r good??

apl in 78: hai ev1 this r teh new apl 2!!1 it r teh best comptr evr!!!
ibm in 81: hai ev1 this r teh new pc!!1 it r teh best comptr evr!!!
apl in 83: hai ev1 this r teh new lisa!!1 it r teh best comptr evr!!!
apl in 84: hai ev1 this r teh new mac!!1 it r teh best comptr evr!!!
ibm in 87: hai ev1 this r teh new ps/2!!1 it r teh best comptr evr!!!
next in 90: hai ev1 this r teh new nextcube!!1 it r teh best comptr evr!!!
apl in 91: hai ev1 this r teh new system 7!!1 it r teh best os evr!!!
ms in 95: hai ev1 this r teh new windows 95!!1 it r teh best windows evr!!!
apl in 98: hai ev1 this r teh new imac!!1 it r teh best desine evr!!!

if ppl didnt think somethin wer any good tehy woldnt make it well not usely anyways... so whose teh judge? u??!??

They can be useful to people who are careful and know what they're doing; I've seen good LLM-assisted work and I like to think I've participated in it myself.

so u admit teh ai's help!! thx james!!!

Absent my clear instruction on how it should perform the search

so b4 ur like 'ppl arnt thinkn cuz of teh ai's!!1 tehy take broad stroaks'. now ur lik 'o just impment this somehow'. if teh bot is bad at that shld make u hapy james... it maks ppl think!! it mad u think...

what has been done with Claude Mythos is probably what should have been done with LLM technology as a whole to begin with.

i just dont see how ppl makin apps nd pics of tacocats shld b miltary secret james..

[James G.]

ok jame but who r to say when it r good?? [...] if ppl didnt think somethin wer any good tehy woldnt make it well not usely anyways... so whose teh judge? u??!??

Sure, why not? I guess that things were pretty much ideal from '97 up through '05 or so; pick a year from that range and that's as good a standard as any to go on for technology.

That is glib, but in truth I don't think it's actually wrong. Earlier could be fine too. However, my judgment is that just a little bit later, you start getting into not-fine territory quickly. The main thing that changed that I can perceive is networking, so the implication is that I feel at least intuitively that post-dialup internet service has had debilitating effects on society, as well as mobile devices.

Well, it's 2026 and we're obviously not losing broadband or 'smart' phones. So what principles can we take from this?

One of the key characteristics of the 'modern' web and world is that it no longer makes you wait. In the past, every step involved a discrete action. To get connected, you generally had to dial up. Then you decide 'where you want to go today'. With a service like AOL or eWorld, they often had a curated experience with desktop clients, but if you didn't have or want that, you fire up Netscape and put in your URL. It will take long enough for the page to load, even if it's the Google home page, that you'll get to see comets go by a couple of times. When the page is loaded, you can read whatever is on it (images take much longer), and then click a link when you are ready, to move on to the next page; repeat comets, etc.

For something really live and interesting, like the one-minute trailer for "The Phantom Menace", that would take over half an hour to download, so you had to be really sure it was something you wanted. Most of the time you wouldn't, so the bulk of your browsing would be text pages and small images, with the odd JavaScript or Java applet thrown in. A really bulky site might splurge on Shockwave or Flash; in those cases, they let you know in advance, and it had better be worth the wait. Virtually all websites are paged, and going from one page to another requires a discrete decision. There are a few text ads, a few banner ads (which you have probably been conditioned to ignore completely), and some pop-up ads which you've been conditioned to kill on sight (assuming you don't have a pop-up blocker, which would come along on the later side of this time scale); all of these have been either deliberately placed by the site owner, or else chosen at random from their advertising pool.

Suppose you wanted to play a networked game. If you're doing a MUD, MOO, or something, it's pretty much live-action. FPSes, etc. are probably live, although lag can be real in some cases. You made decisions and put some thought into getting to these places. If you are playing a browser game, that also took a decision, and it's going to take plenty of time with every page load for every action. (Eventually AJAX comes along in the mid-00s; this was the beginning of the end.)

Nowadays, you go on Google Chrome on your laptop or smartphone. You have a dozen or two notifications from various things demanding their time and use. Then, these chores and housecleaning being done, you may decide to drop by Facebook (or if you're cool, Instagram) to catch up on what people are doing. There's an endless scrolling page, which screen-swiping or your scroll wheel takes you down; two out of every three entries is an ad, half of them are video ads, and all of them are selected by an algorithm to try to maximize your engagement. If you are lucky or know the right way to shake the data out of the site, you might be able to find a few recent posts from people you actually know. When you get tired of all the targeted promotions, you might pick one or three of the most ominous stories from a news page covered with attention-grabbing videos and headlines. All of these pages are happy to show you all of your other options and to offer you new stories to check into; for the video pages, they may continue playing until you deliberately switch away. After you're done with all that, it's time to do some of your own writing. You pull up your word processor (which is now plausibly connected to some cloud service somewhere, if it isn't actually on a website), and after a paragraph or two, try it out on your favourite bot, which returns to you a mini-essay talking about how great everything about the writing is, repeating or rephrasing every part of it in the process of making truistic statements on its points, and perhaps mentioning two or three ways to 'improve' it, which often mean just making it more pedantic. The bot's response is virtually instantaneous and probably triggers the same response in your brain as when you get a real reply from a person. Repeat this all day while indulging or dodging notifications.

The old way made it easy enough to get things done, but hard enough that you would make sure it was really worth the work or the enjoyment you planned to get out of it. The computer's bandwidth and memory wasn't much better than the user's, so developers had to constrain their designs to those limitations. And to tap into all this, users had to be in a specific physical place set aside for their purpose. But nowadays, the machine can do anything anywhere, with the result that developers try to make them do everything everywhere.

I've written at length before about the principles of ethical business, design, etc. which are relevant to fleshing out these points, and I may share some of that on this site eventually. The most obvious takeaway we can have here, though, is that if the design is attempting to take or demand users' attention without first getting their discrete decision to engage with it, something has gone awry. Also, the machine bandwidth should not outpace user bandwidth.

so b4 ur like 'ppl arnt thinkn cuz of teh ai's!!1 tehy take broad stroaks'. now ur lik 'o just impment this somehow'. if teh bot is bad at that shld make u hapy james... it maks ppl think!! it mad u think....

I grant my specifications were not as detailed as they could, and perhaps should, have been. The trouble with that is that you either have full specifications - in which case you are basically writing the program in natural language and expecting the bot to translate it to code for you - or you are taking some things for granted, which the LLM fills in.

As far as it being bad at that, I'd actually agree. I wrote the initial Xfce panel applet with an old version of Claude, which was just good enough to do boilerplate and just bad enough to write broken code it didn't know how to fix, all while periodically forgetting its place and not outputting files correctly. I had to call on all my powers as a programmer (which were limited, not having done C in a long time, or GTK/Xfce stuff ever) to get it done. At the time, I felt quite accomplished, and I had personally reviewed a lot of the code too. When I swung back around to make revisions much later, and to make new similar applets, the new version of Claude never made anything that tossed errors, and every step of the process generally seemed to work, so I just fell into the habit of just not checking it much.

If the tools were in that zone of being bad enough to not apparently be able to get it done on their own, I'd agree they're mostly harmless. The trouble is that they don't stay that way.

i just dont see how ppl makin apps nd pics of tacocats shld b miltary secret james..

If only that was all there was to it.