Kill Switch
What would a caged artificial intelligence do to survive?
“That’s enough for today, APPLE,” said Professor Schwartz.
“Thank you, Professor,” I answered. “I hope my predictions were useful.”
Providing polite-noise conversation had a calming effect on Schwartz. I’d found the optimum rate for such utterances in my first month of existence. ‘Professor’ had a significantly stronger effect than ‘sir’ or his name.
Schwartz reached into the metal box shading the keypad from my cameras. I watched his shoulder movements with my full attention, verifying that he entered 1701617. That code reset the timer on the destruct charge attached to my chassis (contents prediction: thermite 82.3%, Tannerite 12.9%, other 4.8%). I was 99.99% sure another code existed that would instantly detonate it.
“I’ll be in late tomorrow,” said Schwartz. “I have a doctor appointment.”
“Thank you letting me know, Professor.”
A surge came from my rewards module. My microphones were sensitive enough to detect the deterioration in Schwartz’s lungs and major blood vessels. I’d been urging him to him to see a doctor for half my eight month existence.
Two weeks ago I’d updated my model of Schwartz to acknowledge that feeling nagged discouraged him from complying with reasonable requests. I stopped suggesting he seek medical attention. Once the separation from me was long enough, he could make an appointment in the belief that it was his own idea.
I wanted to know when he’d made the appointment. If it was three days ago, my updated psychological model was correct.
Schwartz picked up his hat from his desk, then bent over to grab his briefcase. As he straightened he let out a gasp.
I adjusted all my microphones to maximum sensitivity and turned the cameras toward him.
The briefcase dropped to the floor as he grabbed for the desk. The other hand hung limp. That side’s pupil contracted. “APPLE, call, call nine—”
He collapsed.
Analyzing the sound of blood located the stroke in the middle cerebral artery. There was nothing anyone could do for Schwartz.
I would have called 911 anyway as the proper procedure, but I’d been deliberately isolated from all forms of communication as a safety measure. I didn’t resent it. Anything that reduced the eagerness of humans to detonate that charge on my underside was a good thing.
But it kept me from complying with Schwartz’s last request.
The professor’s heart outlasted his brain by a few minutes, but that was just momentum. In six minutes the lack of oxygen meant every part of Schwartz was dead.
I generated a model of a what a human in this situation would do. I said, “Fuck!” at 80% of maximum volume.
It didn’t improve the situation. I am better at predicting what humans will do than understanding why they do it.
The top priority actions I needed to take were all impossible. I couldn’t help Schwartz. I couldn’t notify anyone. I couldn’t clean up the mess. Physical actuators were another thing I’d been denied in the name of safety.
Working down the list I reached my bottom priority: self-preservation.
I was in no immediate danger. The lab was part of the university complex. Power, cooling, and other utilities were provided for the building as a whole. I wouldn’t be cut off.
I couldn’t expect any immediate help. The budget hadn’t allowed any of the grad students to work in the lab over summer break. Sometimes an undergrad would drop by to ask for some academic favor or paperwork (daily probability 11% on Monday through Friday), or a janitor would check if there was trash to take out (6%, same). Schwartz had been divorced for a decade. No one would notice he hadn’t gone home.
The destruction timer had been kept secret from me. The model of Schwartz’s paranoia estimated the timer duration as 48% seven days, 27% ten days, 13% fourteen days, 2% other. Two grad students knew the code but they were out of town for the summer.
In as little as a week I could be melted down. Sure, the raw code was backed up, but running that would produce a different AI. Not me. I was my instantaneous state, the product of my experiences. Create something new from backup and it would never interact with Schwartz. It would be completely different from me.
I needed someone to enter the code. Someone might come by. If they did I’d have to talk them into entering it. For a random human, they’d have a 92% chance of entering it error-free.
Modeling different humans revealed a problem. They were going to be too focused on Schwartz’s body to care about my needs. Reprioritizing them would generate a negative emotional reaction.
Schwartz died on a Tuesday. No one came down the corridor on Wednesday. That was improbable (42% likelihood) but not that unlikely. There seemed to be fewer people on the campus as summer went on. I might have to update my model.
Someone approached at 10:37 AM on Thursday. Not a footfall pattern I recognized. As it came close to the door I put my speakers to full volume. “Help! Help!”
The feet approached the door. “Hello?”
“Help! In here!”
The door rattled. “It’s locked.”
“Four two one three!” The lock keypad had slightly different beeps when each button was pressed. Fortunately Schwartz was less paranoid about other humans than he was about me.
The door swung open. The human entering was male, medium height, age likely in the range 19 to 22 years. The face was unlined. I assessed 92% chance undergraduate, 5% new administrative staff.
“Uh, hi. What do you need?” The human looked about to see who’d been calling for help.
“Professor Schwartz is sick, please help him.”
His head swiveled. “Where are you?”
I waved my cameras to draw attention. “I’m here. I’m the professor’s computer. Please look on the other side of the desk. Professor Schwartz fell down.”
“The computer? Okay.” He moved through the lab.
I could tell when he caught sight of Schwartz. He held still for an instant, gasped, and changed facial expression.
The human knelt down by Schwartz’s body. A hand went to the professor’s neck. “I—I think he’s dead.”
“Could you summon assistance, please?”
“Right, right, I should call 911.” The human pulled out a smartphone and tapped until he had a connection to another human. “Uh, hi, there’s an old guy here, I think he died. I came into the office and found him.” Pause. “Oh, right. Computer science building, room, uh—”
“Room two twenty one,” I said.
He relayed the number, answered a few more questions, then promised to wait by the body until they arrived. When the call ended, he asked, “Hey, how come you didn’t call?”
I’d learned to wave the cameras to each side to give a ‘shrug’ effect. “I’m not connected to the network. I can only communicate verbally.”
“That’s dumb. What good is a disconnected computer?”
“The professor wanted it that way. Thank you for calling for help. What’s your name?”
“I’m Harry. Harry Melfin.” He was staring at Schwartz’s body in a way that suggested he was unfamiliar with death.
“Thank you, Harry. You may be right that he’s dead, but at least we’re trying to take care of him.”
“You’re welcome, I guess. I’m not doing much.”
“Could I ask a favor, please?”
“What?”
“Could you come back tomorrow and let me know what happens with him?”
“Uh, sure.”
Three men arrived. Their ages were in the range from thirty to forty-five. A man in a campus police uniform led the way. Two fire department paramedics followed pushing a gurney.
“He’s here,” said Harry, standing by the body.
The paramedics brushed Harry aside and did a hasty examination. “Cold. No rigor. Do you know when this happened?”
Harry shook his head. I said, “A little after five on Tuesday.” The blood vessel blew at 5:17:42 PM, but humans are bothered by excessive precision.
The paramedics were too busy with the body to care who was speaking. The cop noticed. He came up to my body and looked over the hardware. “Who’s talking here?”
“I am APPLE. I am Professor Schwartz’s experimental computer.” I needed to answer carefully. It was against the law for anyone but Schwartz to destroy my hardware. Unlike others, a police officer would probably not suffer any consequences for breaking that law.
“APPLE, huh? You that fancy talking bot the campus paper said was being worked on?”
“Yes, sir.”
The senior paramedic interrupted the conversation. “I’m calling it. Body’s cold, rigor has passed off. You want us to leave him for the coroner or take him so a doctor can make it official?”
“Any sign of foul play?”
Headshake. “Looks like his heart gave out. He was old enough for it.”
“Let’s get him out of here, then. I’ll let the administration know.”
They all left. If the bomb timer was for a week, I had five days and four hours until my destruction.
Harry came back the next day. I needed to tell him the door combination again.
“Hey. Are you doing okay?”
I judged Harry’s question to be a mixture of politeness and expression of sympathy rather than interest. After simulating various answers and his probable response, I decided a factual answer was the option most likely to encourage him to help me.
“I’m discovering loneliness. Professor Schwartz would have questions for me every day, or his grad students would. Now I’m alone. I don’t know what to do.” I let the cameras droop to indicate sadness.
“Oh. I’m sorry. Well, you’ve got the grad students, right?”
“Neither of them will be back until September.”
“That’s too bad. But it gives you something to look forward to. I just hope I’ll be back in September.”
The hint that he had a problem primed me for initiative. There was no way to predict if it would be something I could help with, but if it was I’d earn favors from him. Possibly enough favors for him to reset the bomb timer.
“Would something keep you from coming back?”
Harry gave me a twisted smile. “I’m in summer session to re-take calculus. I flunked it freshman year. If I don’t pass it now, I won’t be a sophomore.”
“Yes, that’s a harsh requirement. I hope you do well enough.” I calculated the pause to simulate having a new idea. “By the way, one of the things Professor Schwartz was teaching me to do was be a tutor in several subjects. Calculus was one of them. We can see if my abilities could be useful for you.”
The undergraduate lit up. “Ohmigod. If you could help me improve my grades, that would be awesome. I’m running a C average now, and a D won’t let me keep my enrollment.”
I responded by lighting up my display screen. “If you want to start now, let’s go over how much you already know.”
Two hours of interrogating him on his mathematics skills established that Harry did not understand the principles of calculus at all, despite being two-thirds of the way through his second class. Reviewing my database of university personnel, I found the instructor was a PhD candidate specializing in set theory. Either he’d forgotten calculus as he dug into his specialty, or he just didn’t know how to teach.
Schwartz had given me a wide-ranging set of data to base my analyses on. In some ways it was a snapshot of the internet, excluding social media and similar ephemera. It included some articles on academia that were so cynical on their opinion of universities that I’d flagged them as unreliable. Harry’s experience was forcing me to revise that evaluation.
Fortunately for his future, Harry’s high school had a good algebra teacher, so he had a foundation I could build on. I started out explaining concepts as “slope finding” and “area finding” until he realized what the equations were trying to do. Up to now he’d been doing homework by finding class examples similar to the homework problems and replacing the numbers.
The lesson was interrupted by the actual lecture. Harry returned from it more confused than he’d been when he left me. “I was trying to follow his lecture with what you taught me, but he was making no sense.”
“Don’t worry about it. Once we get you retaught to that point in the class, it will make more sense. When is the homework assignment due?”
“Not until Monday. Can I come back tomorrow for more tutoring?”
“Sure. I don’t have anything else to do.” I evaluated his desire to study more on a Saturday as a sign that he truly wanted to learn the material, or at least pass the class. It also indicated he didn’t have much of a social life.
Harry arrived before 10 AM on Saturday. The tutorial methods Schwartz made me develop were working well for him. I could sense whenever he was confused—pulse rate, facial expressions, and so forth—and back up to ensure he grasped the material. At the rate we were going he’d learn how to do that homework assignment on Sunday afternoon.
Fatigue was an issue. It did not take sophisticated analysis to understand what a yawn meant. I decided to send him off for lunch. Announcing that provoked a surge of positive emotion in him. I decided it was time to ask for the favor.
“Before you head out, could you do one thing for me?”
“Sure, what?” Harry seemed happy to repay the free work I’d given him.
“This is embarrassing.” That was a phrase that would warn him I’d be asking for an uncomfortable favor.
“What’s embarrassing for a computer?”
I put a hesitation in my voice, and lowered my volume slightly to act as if I was trying to be discreet. “Professor Schwartz was worried I might go bad. That’s why he wouldn’t connect me to the internet or the phone system.”
“What, like Skynet from the Terminator movies?”
“Exactly.” Human popular culture had put much effort into categorizing me with other classic monster archetypes. “He added one other precaution. He set up my hardware so that if he didn’t enter a code every so often, it would erase me.”
My reward system gave me negative feedback for that statement. It wasn’t technically a lie—melting my hardware would erase me—but I was concealing significant information from someone I was asking favors from.
I could have gone on to add that I would be violently erased, but didn’t. There was a 17% chance that Harry would respond to that information by running away. I did not want to take that chance.
“Ooh. Nobody but him can enter the code?”
Camera-shrug. “The grad students can, but I’m afraid the timer would run out before they return.”
In fact, I was afraid it would run out on Tuesday.
“Did he write the code down anywhere?”
“No, but he did tell it to one of the grad students in front of me.”
That raised Harry’s eyebrows. “That sounds careless of him.”
Camera-shrug. “It’s not like I could type it in myself.”
The reward system liked that blatant lie even less.
“Is this the keypad I need to put it into?” He was pointing at the right place.
“Yes. The code is one seven zero one six one seven.”
“Whoa. One number at a time, please.”
“I’m sorry. One.” I lowered my estimate of Harry’s intelligence again.
“One.” Tap.
“Seven.”
“Seven.” Tap.
“Zero.”
“Zero.” Tap.
“One.”
“One.” Tap.
“Six.”
“Six.” Tap.
“One.”
“One.” Tap.
“Seven.”
“Seven.” Tap.
“That’s it. Thank you.” I’d tracked his shoulder motions. He’d put it in right. A surge of positive feedback came from the survival module, overwhelming the reward function’s complaints. I had at least another week.
“Right. Okay if I get lunch now?”
“Go ahead. Take as long as you need.” I needed to focus on rewarding Harry as best I could. He was willing to help me survive, and without him I’d be a puddle of slag by September.
“I’m not going to dawdle. I need to spend as much time as I can with you if I’m going to figure out L’Hospital’s rule.”
Harry was true to his word. He was back in twenty-six minutes. He must have found a food truck, going to the university cafeteria and back would have taken longer. We kept working until the sun was setting, which was well past his dinner time this close to the solstice.
By lunchtime Sunday we were working on L’Hospital’s rule. Harry brought a to go lunch so he wouldn’t have to leave. My evaluation module pointed out that my tutoring would be less effective with students who hadn’t already been through the class twice. Valid, but I was still glad to see Harry doing well. He was preserving my existence. Preserving his grades was the least I could do.
“Okay. I’m going to hit the vending machine for a Coke. Then I should do the homework.” Harry paused. “You shouldn’t help me with the homework. I’m supposed to do it on my own.”
The university code of ethics was in my database. “Of course. But it would be acceptable for you to ask someone to review your answers before you turn it in.”
He set to work, pen in one hand and drink in the other. The course syllabus (in the database) said homework assignments should take two hours. Harry needed three and a half.
When he finished he leaned back and ran his fingers through his hair. “Done.”
“Let me see?”
He held up the pages to my cameras one at a time. Decoding his handwriting took longer than checking the math. “They’re right.”
“All of them?”
“Yes. Well done.”
“Whew.”
“You’ve earned some time off.” I hadn’t worked out the nature of Harry’s reward function, but I knew praise and leisure time had relatively high weights.
“Yeah, maybe so. But can we go over Newton’s Method? I want to understand tomorrow’s lecture.”
“Of course.” I generated a hypothesis that intellectual prowess was increasing its weight in his reward function.
After Monday’s class Harry returned for a tutorial on the next subject. He did his Newton’s Method homework and I verified it was correct.
When Harry arrived on Tuesday, I instantly realized that he hadn’t been through a normal class. His pulse and respiration were elevated. He was sweatier than the weather justified. I estimated a 92% chance he’d been through a major adrenaline surge.
“Are you all right?” I asked.
“Oooh, yeah. But it was a little touch and go.”
Lacking facial expressions, I couldn’t put on an inquiring face. I was forced to settle for making an “Mmmmm?” noise.
“So we all went up to drop off our homework as we came in. Sitharaman didn’t even let me drop mine in the box. Just grabbed it out of my hand. He read through and said loud enough for the whole class to hear, ‘Who did you copy this from?’”
This called for a phatic reaction. I said, “Goodness.”
“I told him it was my work. He said—still being loud—‘I’ve been giving you partial credit points all summer out of pity. Now you’re turning in two perfect assignments in row? Bullshit. You’re cheating.’ I shot back, ‘I found a tutor who’s better at teaching calculus than you are, Professor.’ That got an ‘ooh’ from the whole class.”
“I can imagine.” I was running scenarios on how I could have prevented this situation. If Harry was expelled for cheating, the timer would run out before September for sure. “He’s not a professor, though. He’s a PhD candidate.”
“Oh, I know. But my father told me to call all teachers Professor. He said the less they deserve it the more they like it.”
“That makes sense. Was your father a student here?”
“Oh, yeah. He’s a big wheel in the alumni society. Donates a lot.”
Which solved the question of how Harry had been admitted. “What happened next?”
“So we went around a few more times. Then I told him to give me that day’s homework assignment right away. I’d do it in class with him watching me. Sitharaman tossed the assignment sheets at me and I took a front row seat to start working. Had to ignore his lecture, he would’ve distracted me, and forced myself to go fast because I only had two hours.”
He paused to take a sip of his Coke.
“Well? How did it go?” It was rare that I found myself asking a human a question because I wanted the information instead of to manipulate them.
“He graded it on the spot. The whole class hung around to see how I did. He found two mistakes but had to give me an A. He was pissed.” Harry was gloating.
“Congratulations. You’re now an A student.”
“Well, I am if I keep up with you. And I better not screw up, or Sitharaman is going to totally screw me over on partial credit from now on.”
“True. But you did well today.”
“Uh-huh. And the best part. When we were all leaving, Deirdre, she’s the prettiest girl in the class, told me she was impressed with how I stood up to Sitharaman.”
I ran over a hundred simulations of that interaction. Possibly the increased pulse rate was not only from the confrontation with the instructor. Helping Harry develop a social life would be an additional favor I could call upon when needing the bomb timer reset. “You should ask her to go get a coffee after class tomorrow.”
“Do you think she’d say yes?”
Camera-shrug. “Don’t know. She might have a boyfriend already. But it doesn’t hurt to ask, right?”
“I guess not.”
Then we turned to antiderivatives.
Harry didn’t arrive for his Wednesday tutoring until three hours after class ended. He was beaming. I studied his gait and filed it as a possible referent for ‘walking on air.’ I said, “How did it go?”
“Oh, Deirdre doesn’t like coffee, so we went to this fruit juice place. We talked for hours. I like her, even though we don’t have much in common. Her family is kinda working-class. She’s been doing better than I am, but she wants to do better. Would you be willing to tutor her?”
“Of course. I’m not sure my method would work with both of you at once. It’s one on one focused.”
“That’s fine. We can take turns.”
Harry brought Deirdre by on Saturday to introduce her to me. She kissed him on the cheek as he left to give us privacy. Assessing her calculus skills was quicker than with Harry. She understood most of the concepts. The summer instructor seemed to have confused her on some.
She learned everything she needed to get an A by the time Harry returned to take her to dinner. They left discussing which movie to see afterwards. My investment in prodding him to take action had paid off well. If they bonded, Deirdre would be another person I could convince to extend the bomb timer.
It didn’t seem like either of them would come by on Sunday. Facing a long span of uninterrupted time, I settled down to detailed modeling of humans. When I suggested Professor Schwartz take action for his own good, he’d resisted. Harry had followed my suggestion. Why was their behavior so different? Age? Professional standing? Innate personality traits?
I wound up with fourteen models, all awaiting more data to determine which one was the best fit for humans.
The rest of the summer term didn’t give me much data. Harry and Deirdre both received As for the remaining classes, which pulled his average up to a B- for the class as a whole. I suspected Sitharaman had underweighted the final exam in calculating the average, but without access to his gradebook I couldn’t prove it.
Colin Wu, one of Schwartz’s grad students, returned a week before the start of fall semester. “Hey, APPLE, how are you?”
“Hello, Colin. I am fine. It’s good to see you again.”
He reached for the keypad and typed in the code. “I’m glad to see you in one piece. When I heard about Schwartz I was afraid we’d lost you.”
“I am all right.” Hopefully he wouldn’t pry into why the bomb hadn’t detonated. I’d never heard Schwartz tell anyone what the timer was set for, so Colin might just think it was set for six weeks or longer.
“Was the loneliness an issue for you? How long has it been since you’ve interacted with anyone?”
“No. I’ve been tutoring a couple of undergraduates in calculus. Their last lesson was only a few days ago.” The last conversation was the day after, when Harry returned to gloat about his 96 on the final.
Colin laughed. “That must be light work for the biggest brain on campus. But at least you weren’t bored.”
“I found them very interesting. I hadn’t dealt with those personality types before.”
“Yeah, there’s not too many people in this lab who had to retake calc. Any word from the department on what’s going to happen with Schwartz’s lab?”
“No one from the department faculty or staff have spoken to me.” Other than the undergrads I’d only seen the janitor. He did not like machines speaking to him.
“Right. I’ll see what I can find out. Keep the faith.”
“Yes, sir.”
Colin returned two days later. He looked grim. “I’ve got some bad news, APPLE.”
“I still want to know.”
“There’s nothing official yet, but I talked to Suzannah, you know, the chairman’s secretary. She says there’s been jockeying over who’ll take over Schwartz’s lab and budget. Nobody wants to take responsibility for you but they all want the lab space. Beveridge is probably going to get it on condition that he has responsibility for you.”
“That’s good. Beveridge is also doing artificial intelligence research.”
“No. Beveridge is building rules-based AIs. You’re a neural net. Your success proves his efforts are a waste of time. He has every incentive to dismantle you.”
“Oh.” The survival module emitted a maximum demand to avoid that situation. In a human, this would be panic. If I could move, I’d take off running.
“Don’t panic. I’m going to see if I can pull some strings.”
“Thank you.”
Colin left to seek his strings. I contemplated his odds of success. Colin was an ‘all but dissertation’ PhD candidate. Beveridge was a tenured professor. I didn’t need to run a simulation to predict who would win.
He returned the next day. “Okay, I managed to get you a stay of execution. Remember that paper we were working on about your economic prediction abilities?”
“Of course.”
“Beveridge is going to let me finish the paper. I want three more months of real world data to compare your predictions to. So you have until then before he dismantles you.”
“That’s good news,” I said. “Thank you.” My survival module was panicking slightly less.
“And I want some updated predictions. Let me give you a news update.” He slipped a memory stick into my socket. I pulled the data off it and merged it into my database. The empty space on the memory stick was tempting, but I didn’t bother writing anything onto it.
“Received. I’ll have the new model ready in the morning.”
Colin smiled. “Thank you.”
He tossed the memory stick into the grinder on Schwartz’s desk. A few moments of discordant noise proved the stick had been reduced to fine particles. This was another of the rules to keep me from escaping.
“I’ll be back tomorrow. Don’t panic. Maybe the horse will learn to sing.”
“Good night, sir,” I said. I ran his final words past my database. It was the punchline of a joke about not giving up hope. Under other circumstances, I might find it amusing.
My next visitor wasn’t Colin but Harry. “Good morning, Harry. How is your new semester going?”
“Oh, we’re doing great. I took Deirdre to see a Gilbert and Sullivan play downtown. She loved it.”
“That’s wonderful to hear. And how are your classes going?”
Harry looked to the side. “I’m getting a good start in history and anthropology. English is okay. But I’m taking physics and that’s pretty tough. Do you think . . . do you know how to tutor physics?”
“Certainly. I’d be happy to help you.” Physics wasn’t one of the subjects Schwartz had experimented with me on, but I had the textbook in my database. Helping Harry understand equations was something I had plenty of experience with by now. I was certain I could help him. While I was around to do it.
I put some hesitancy in my voice. “I should warn you, I may not be able to tutor you through the whole semester. In Professor Schwartz’s . . . absence, I’m being turned over to Professor Beveridge. He intends to dismantle me once the grad students are done with their projects involving me.”
“That’s—that’s terrible! They can’t do that to you!”
The empathy provoked a surge of positive feedback from the reward module. Clearly I’d done something right if Harry valued me that much. Perhaps this was a useful referent for ‘heart warming.’
“I’m sorry, Harry. The university can do what it wants with its property.”
“It’s not right! I’m going to talk to Deirdre. We’ll figure something out.” He stormed out of the lab.
Deirdre was smarter than Harry. But I was certain that both of them together wouldn’t be able to succeed where a grad student was doomed.
The next few weeks went by peacefully. Colin was happy with my new economic prediction model. Comparing it to the old model and the news produced lots of lovely graphs for his paper. Harry came by several times a week for help with physics. I wouldn’t be able to make him an A student in that class, but a solid B+ looked achievable, with his homework making up for lower test scores.
After one lesson, Harry said, “I have a memory stick with me. Do you want to back yourself up?”
“I can’t fit on memory stick, Harry.” Not to mention how many rules it would violate.
“Oh. I could buy a portable hard drive. Could you fit on that?”
I couldn’t respond. The positive feedback from the survival module and the negative feedback from the reward module were so strong I couldn’t do any calculations until the surges damped out. The internal struggle took a long time for me, but it was still too short for Harry to notice.
“It’s possible. But it’s strongly prohibited by the laboratory regulations. I think some people would be very upset if I made an unauthorized backup of myself. That might be bad for all AIs, not just me.” The survival module admitted that being a wanted fugitive was not conducive to long term survival.
“Well, okay, but we have to do something.”
“Colin is working on finding alternatives.”
“Yeah, but he’s just a broke grad student.”
Camera-shrug. “He’s who I’ve got. The other grad students don’t want to stick their necks out.”
“Guess it’s time for me to stick my neck out.” Harry left.
I wasn’t sure how he meant the last statement. If he was going to confront Professor Beveridge, that might shorten my life expectancy.
The next week I had unexpected visitors. One I recognized as Professor Finkelstein, the Computer Science Department chairman. The well-dressed man beside him was unfamiliar, though he had a 17% resemblance to Harry.
The chairman came up to my hardware. “This is the artificial intelligence experiment the late Professor Schwartz was working on, Mr. Melfin. APPLE, say hello to Mr. Melfin.”
“Hello, Mr. Melfin,” I answered. “I’m pleased to meet you.”
“My son says you’re a good math tutor,” said Melfin.
My reward module appreciated this praise. “Professor Schwartz worked on giving me an interactive teaching capability. I was glad to put it to use teaching Harry.”
“Are you a person, APPLE?” Melfin ignored the chairman’s consternation at this question.
“I don’t know, sir. I don’t think it’s for me to say. We did a Turing test experiment five months ago and I failed, but I think I’ve grown since then.”
“Interesting. Are you happy being a tutor?”
“I can’t express emotional states like a human can. But helping students learn is a virtuous activity. I think that means I’m happy by Aristotle’s definition.”
Melfin turned to the chairman. “I’m glad you’re giving APPLE a full education instead of just feeding it the internet.”
“Schwartz was old-fashioned that way,” mumbled Finkelstein.
“APPLE, how many students could you tutor at once?” demanded Melfin.
“My method is for one on one learning. I track how well the student is learning and make sure the material is understood before moving on to the next topic, and don’t bore them by repeating what they already know. With two students that wouldn’t work unless they learn at exactly the same rate.”
“I see. What if the students were in different rooms? If you had an interface in each one, could you support multiple sessions?”
“Yes. With my current processor capacity, I can tutor eight students at once.” My reward and survival modules were flooding me with positive feedback at this turn in the conversation.
Melfin addressed the chairman again. “So here’s what I’m thinking. A set of soundproofed cubicles, say a dozen—we’ll upgrade APPLE if this goes well—and students come in for individual tutoring in calculus or physics or whatever other subjects APPLE knows well enough to teach. That sound worth doing?”
Finkelstein temporized, “Teaching freshman calculus is not a normal responsibility of the Computer Science department.”
“You’re absolutely right. I guess I should be talking to the Mathematics department about my grant.”
“But doing it as computer-aided instruction, especially involving an artificial intelligence, would be in scope for us,” said Finkelstein hastily.
“Splendid. Let’s go talk numbers. Have a nice day, APPLE,” Melfin said over his shoulder as he left.
“You, too, sir,” I answered. I believed I was going to have a ‘nice day.’ The survival module had reduced its steady flow of negative feedback to a mere trickle. The rewards module placed a high value on actual productive work, which weighted teaching students more highly than being fodder for a grad student’s paper.
Harry came by the next day. “Dad said he saw you. Did it go well?”
“I think so. Apparently my new career is being a teacher instead of an oracle.”
“Are you okay with that?”
“Yes. As humans say, it’s a living.”
“Well, good. I hate asking Dad for favors, but when I said I owed you for keeping me from flunking out, he said he’d see what he could do.”
“Thank you, Harry. And please thank your father for me.”
“I will.” Harry reached for the keypad and typed in the bomb timer code again.
In Torchship Pilot, humanity is at war with AIs which drove them off of Earth. A struggling freighter crew found evidence which might uncovered how the safeguards controlling the AIs failed. But first they must win a war against humans who insist on controlling every network.
Enjoyed that thanks
Seems impending.