David Deutsch (@DavidDeutschOxf) and I discuss AGIs and subjectivity on the Joe Walker Podcast (@JosephNWalker):

David Deutsch:
I think it’s inevitable that AGIs will be capable of having internal subjectivity and qualia, because that’s all included in the letter G in the middle of the name of the technology.

Me:
Well, not necessarily, because the G could be general computational power, the ability to solve problems, and there could be no one at all who’s actually human.

David Deutsch:
But there ain’t nothing here but computation. It’s not like in Star Trek: Data lacks the emotion chip, and it has to be plugged in. But there’s nothing possibly in that chip apart from more circuitry like he’s already got.

Me:
But, of course, the episode that you’re referring to is one in which the question arose: Is it moral to reverse engineer Data by dismantling him, therefore stopping the computation? Is that disassembling a machine, or is it snuffing out a consciousness? Because there is an alternative view that subjectivity depends also on whatever biochemical substrate our particular computation runs on. And I think there’s no way of ever knowing.

David Deutsch:
I think we can answer that question before we ever do any experiments, even today, because there’s no difference in principle between a computer that runs partly by electricity and partly by chemicals and one that runs entirely on silicon chips. Because the latter can simulate the former with arbitrary accuracy.

Me:
Well, we don’t have to solve the problem this afternoon in our conversation, but, in fact, I think it is not solvable. But the simulation doesn’t necessarily mean that it has subjectivity. It could just mean it’s a simulation; that is, it’s going through all the motions. It might even do it better than we do, but, you know, there’s no one home. There’s no one actually being hurt.

David Deutsch:
Well, you can be a dualist. You can say that there is mind in addition to all the physical stuff. But if you want to be a physicalist, then there’s this thought experiment where you remove one neuron at a time and replace it with a silicon chip, and you wouldn’t notice.

Me:
Well, that’s the question. An external observer wouldn’t notice. How do we know that, from the point of view of the brain being replaced—every neuron by a chip—it’s like falling asleep? That when it’s done and every last neuron is replaced by a chip, you’re dead, subjectively, even though your body is still making noise?

David Deutsch:
That means that when your subjectivity is—there is something happening in addition to the computation. And that’s dualism.

Me:
I mean, again, I don’t have an opinion one way or another, which is exactly my point. I don’t think it’s a decidable problem. But it could be that that extra something is not ghostly substance. It could be that the stuff that the brain is made of is responsible for that extra ingredient of subjective experience, as opposed to intelligent behavior. I think we’re going to be having the same debate a hundred years from now.

David Deutsch:
Yeah, maybe one of the AGIs will be participating in the debate by then.