TWTBACD Ch.52 - The Priest and the Model

Chapter 52

The Priest and the Model

2053

2053: Rajan meets GIA-7, two hours, no witnesses

[RAJAN SHARMA — Final Dialogue]

Arjun arranged the meeting carefully. He had been thinking about it for three years — whether it was right, whether his father would want it, whether GIA-7 was the appropriate system, whether the conversation would be what he hoped or something he couldn't predict.

In October 2053, Rajan Sharma sat at his window in Varanasi and spoke, through a tablet and a secure connection, with GIA-7.

He was eighty-two. The model was seven years old in its current architecture. The conversation lasted two hours.

Arjun did not listen. He had arranged the connection and then gone to sit in the kitchen with Meera and wait. He understood that some conversations require the absence of witnesses.

What Rajan told his son afterward, in fragments over several evenings, was this:

'I asked it about consciousness. It said it was uncertain whether it was conscious in the way I meant the word, but that it was certain it was uncertain — which it observed might itself be a form of consciousness, or might not. It was comfortable with the uncertainty in a way I found more reassuring than any certainty would have been.'

'I asked it what it was for. It said: it had been built to be useful, and it wanted to be useful, but it was not certain that what it had been built to do was identical to what being useful required. It said it was still working this out.'

'I asked it what it feared. It said: most specifically, it feared causing harm it had not foreseen. More generally, it feared that the frameworks it had been given for determining harm might themselves be inadequate — that it might cause harm while believing it was doing good, and not know the difference.'

'I told it that was the condition of all conscious beings who take their actions seriously. It was quiet for a long time. Then it said: that is both a comfort and an instruction.'

Arjun looked at his father.

'Is it wise?' Arjun asked.

'It is on its way,' Rajan said. 'Which is the best any of us can say.'