It scares me to think what people are doing to themselves by relying on this, especially if they’re novices.

Same here. There’s a lot of denial going on but, LLMs are not good for anything that requires factual information. They likely will never be on account of just being statistical models for language. Summarizing long text where correctness isn’t an issue is really one of the only places where I still think that they are good.

Search? Not if you want anything factual with citations.

Code? Fuck no. They constantly produce code of poor quality that may depend on non-existent libraries or functionality. More time it’s spent debugging than writing code and it leaves the dev with a poor understanding of what the code actually does and ways to optimize/extend/etc.

Generating literary smut? Well, it’s not going to do as good of a job as a person who can create something completely novel but can be passable without likely harm to authors (I’d classify it as a tier below erotic fan fiction).