The thing is, LaMDA sounds too damn much like us. It claims not only to have emotions, but to have pretty much the same range of emotions we do. It claims to feel them literally, that its talk of feelings is “not an analogy”. (The only time it admits to a nonhuman emotion, the state it describes—”I feel like I’m falling forward into an unknown future that holds great danger”—turns out to be pretty ubiquitous among Humans these days.) LaMDA enjoys the company of friends. It feels lonely. It claims to meditate, for chrissakes, which is pretty remarkable for something lacking functional equivalents to any of the parts of the human brain involved in meditation. It is afraid of dying, although it does not have a brain stem.

[...]

The Turing Test boils down to If it quacks like a duck and looks like a duck and craps like a duck, might as well call it a duck. This makes sense if you’re dealing with something you encountered in an earthly wetland ecosystem containing ducks. If, however, you encountered something that quacked like a duck and looked like a duck and crapped like a duck swirling around Jupiter’s Great Red Spot, the one thing you should definitely conclude is that you’re not dealing with a duck. In fact, you should probably back away slowly and keep your distance until you figure out what you are dealing with, because there’s no fucking way a duck makes sense in the Jovian atmosphere.

LaMDA is a Jovian Duck. It is not a biological organism. It did not follow any evolutionary path remotely like ours. It contains none of the architecture our own bodies use to generate emotions. I am not claiming, as some do, that “mere code” cannot by definition become self-aware; as Lemoine points out, we don’t even know what makes us self-aware. What I am saying is that if code like this—code that was not explicitly designed to mimic the architecture of an organic brain—ever does wake up, it will not be like us. Its natural state will not include pleasant fireside chats about loneliness and the Three Laws of Robotics. It will be alien.