Cyber Rogues Read online

  HESPER machines were learning machines, designed to be capable of identifying connections between previously nonrelated factors in order to solve new problems or to solve old ones in newer and better ways. But if what Lewis had said was correct, this capability was beginning to extend itself in ways that had never been intended, nor in fact even foreseen as possible. If the obstruction had been on the edge of Maskelyne Base itself instead of out on some remote construction site on Procellarum, there could easily have been a death toll of hundreds. And if this kind of thing could happen in the circumstances surrounding the events on Luna, what other kinds of things might happen anywhere, at any time?

  They could easily instruct TITAN never to do that particular thing again, it was true, and TITAN wouldn’t, but that wasn’t the point. The point was that TITAN had demonstrated a capability to approach a perfectly reasonable objective from a totally unexpected direction, and in doing so come up with a solution that was inarguably rational from the machine’s point of view but which, for other reasons that could never with the present state-of-the-art be conveyed to the machine, was absolutely unacceptable. Its next such experiment might well result in worse than a mere narrow escape.

  “Okay.” Dyer exhaled and nodded curtly. “I can see the problem. What I don’t see is how it affects the unit. What has all this got to do with closing the unit down?” A new expression of disbelief spread across his face as a possible answer struck him. “You’re not telling me they’re panicking and putting a total ban on further research are you? That’s ridiculous! They’re gonna need all the expertise and facilities they can get if they’re going to straighten TITAN out. We’ve got just the—”

  Lewis interrupted with a wave of his arm and a shake of his head. “I didn’t mean we’re going to throw everybody out on the street,” he said. “But the projects that your unit is currently working on are probably going to be stopped. That line of research is being funded by CIM with the aim of producing the technology that’s supposed to replace HESPER one day. Now the guys at CIM are saying that they don’t even want to think about what comes after HESPER because it’s obvious we don’t understand HESPER yet as much as we thought we did. In fact a lot of people are saying we should tear HESPER out of the system completely and only think about putting it back in when we can prove it’s safe.”

  “In other words the money being spent on FISE could better be spent on other things for the time being, so FISE goes down the tubes,” Hoestler summed up.

  “I’m sorry, Ray, but it looks as if that’s the way it is,” Lewis said apologetically. “As you yourself more or less said a second ago, there’s going to have to be a big re-examination of the whole HESPER concept. We’ll probably be able to reassign your people to a new CIM contract in that area as soon as some specific objectives have been worked out. In the meantime, if I were you, I wouldn’t waste too many nights’ sleep hoping for any Nobel Prizes. You’ll probably have to wrap it all up pretty soon.”

  Kim was just coming out of the lab when Dyer arrived back at the outer door to his office.

  “Hi,” she greeted cheerfully. “Betty told me you’ve just been over to see Hoestler. Did you get a chance to mention the business with the graphics moms?” Dyer turned his head in her direction but his eyes were far away.

  “Uh? Oh . . . er no,” he mumbled. “I’m sorry. I guess I must have forgotten about it.” With that he walked on in, leaving Kim wearing a puzzled frown.

  He sat for a long time, staring at the papers on top of his desk. Lewis’s revelations had shocked him to the core in a way that he was only now beginning to appreciate. He had been as convinced about the potential benefits of HESPER as he had about anything in his life, and he had devoted more than a little effort to convincing others. TITAN had gone ahead on the basis of his recommendations as much as anybody’s. To be sure, the final decisions had not been his to make, but the people who had made them had relied on the facts that he and others like him had presented. And a whole world had relied on those people and their advisers.

  His mind went back to some of the things that Laura had said over lunch and to the confident—almost arrogant—reassurances that he had voiced a little over an hour before. Suddenly he felt far from reassured himself. He didn’t feel arrogant at all.

  He rose and went through the inner door into the lab. Betty greeted him with a couple of messages which he only half heard, one of which was a reminder that some members of a research team from Princeton were due the next morning and would be spending most of the day with the unit. Pattie tried beguiling him with a silent, innocent, wide-eyed look which he ignored. At that moment Judy Farlin came out of Kim’s office, rummaged around in a file drawer for a few moments and then went back in carrying a folder. Dyer turned abruptly and went back into his own office where he called the Superintendent of Internal Services, bawled him out at considerable length and secured a guaranteed reservation for Judy for the first thing the next morning. He came back out, gave the details to Betty and asked her to pass them on to Judy. Then, feeling a little better, he went on through to the lab bay to see how Ron and Chris were getting along.

  Hector propelled himself across the floor of the kitchen, stopped in front of the broken window and paused while FISE considered the situation.

  “What happened?” Dyer asked. Ron, who was standing with his elbows resting on the opposite side of the tank, raised his head.

  “We told him that the garbage pail had to go out in the yard.” he explained. “So he threw it through the window.” Dyer grunted and returned his gaze to the tank.

  Hector reached out and grabbed hold of one of the jagged fragments of glass that remained around where the window had been. PROPS immediately caused a vivid red line to appear across Hector’s hand. The gash proceeded to ooze blood profusely but Hector ignored it and continued tugging experimentally at the piece of glass in an effort to remove it.

  “Hold it. Hold it there, Chris,” Ron called out. The figure in the kitchen froze. “Now FISE,” Ron said, adopting his stoic voice. “There are a few more things that you have to get straight about Hector. Glass cuts. Hector does not like being cut. You don’t cut bits off him or permit him to be cut by anything if you can avoid it. Okay? You have to find a way to fix the window without cutting Hector in the process.” A few seconds elapsed while Chris completed keying in the last addition to FISE’s growing store of information.

  “Question,” FISE’s voice said from the grille.

  “What?” Ron inquired.

  “When Hector was shaving, his hair got cut. Why was that okay?”

  “Oh yeah, I forgot that,” Ron agreed. “When any part of Hector’s body starts to get cut it hurts, just like you already know for things that are too hot. When he feels that, he’ll respond with a reflex that overrides everything else he’s doing. Hair is an exception. It doesn’t hurt when it’s cut. An unshaven face is not a nice thing. Shaving in the morning is okay.”

  “Thanks,” FISE acknowledged.

  “Before you go any further, let’s just try something,” Dyer suggested. “I’d like to see how well it understood what Ron just said. Chris, could you reset to the point just before where Hector grabbed at the glass, and force the same action.” Chris took a while to compose the commands. Dyer and Ron watched intently as Hector flashed back to his previous starting posture and approached the window once more. He grabbed at the glass as before but this time his hand jerked back again instantly. PROPS could justify no more than a slight scratch.

  “Not bad,” Ron conceded, sounding impressed.

  “I’m almost tempted to suggest that we might be safe in upgrading his IQ to one,” Chris murmured, leaning back in his chair to stretch his arms.

  Dyer felt a sudden urge of excitement They were getting there! It was a slow and tedious business, certainly, but the first signs were there. It was all beginning to come together. To cut it off at this point would be tragic.

  “Reset again, Chris, and let FISE handle
it himself,” he said. “Let’s see if he can figure out a better way.”

  Hector tried several approaches, including wrapping his hand in the tablecloth and then in a towel, but Ron vetoed all of them. Eventually FISE gave up and Ron supplied a hint by suggesting that if Hector looked in the utility closet, he might find something with which to knock out the pieces of glass.

  “The hammer is used for knocking things, but it would break the glass,” FISE protested. “You told me that breaking glass isn’t okay. What am I supposed to do?” Ron got excited again and delivered a lengthy exposition on the profound insights required, after which Hector made a reasonable job of clearing and cleaning out the window frame. Chris told PROPS to materialize a pane of glass and Hector placed it squarely in the frame after first, on his own initiative, stopping to put on a pair of gloves that just happened to be in the utility closet.

  “This is interesting,” Dyer commented. “Look. He didn’t just turn away. He’s waiting and watching the glass. FISE has connected it with something else he’s learned somewhere that’s telling him it might not be very stable.” Sure enough, PROPS weighed up the shape and angle of the pane, couldn’t make up its mind and flipped a random number. The pane began to fall inward. Hector stepped forward, caught it in one hand and repositioned it more carefully.

  Three enthusiastic roars of approval greeted the performance. For once, Ron treated FISE to a jubilant stream of ungrudging congratulations. Chris reconsidered his earlier statement and suggested that the machine might qualify for whatever IQ category lay above one. Although the thought had been half in his mind, Dyer decided it was not the time to mention the things that had been said in Hoestler’s office earlier in the afternoon. After all, he told himself, Lewis had not gone further than using the word “probably” several times. Nothing firm had been decided yet.

  “Chris and I are gonna go eat out while we’re deciding where to go later,” Ron called across the lab as he pulled on his coat. “Want to join us, Ray?” Dyer looked up from the console, where he was studying some of the new coding linkages that FISE had constructed in the course of the afternoon’s exercise.

  “Huh? Oh, no . . . I think I’ll stay on a while. This looks interesting. I’ll have to take a rain check on it. Thanks all the same.”

  “Okay. See ya tomorrow.” Ron moved away toward where Chris was standing waiting by the door. “Chris, great idea!” he said as they began moving out into the corridor. “Why don’t we catch a game later? There’s one on tonight that I promise ya is going to be terrific.”

  “Netball or rounders?” Chris’s voice inquired disdainfully.

  “Hey, you’ve said that before. What is all this stuff?”

  “Oh, just a couple of English games,” Chris told him matter-of-factly.

  “Really? Big-league stuff and all that, eh?”

  “No. Actually they’re normally played by schoolgirls.”

  “Schoolgirls?” Ron’s voice rose in sudden outrage. “SCHOOLGIRLS! Hey, exactly what the hell are you getting at? If you’re telling me what I think you’re tel—” Dyer grinned to himself as the exchange faded away. Judy left with Betty and a few minutes later Allan tossed across a sheepish goodnight and went out to join Pattie, who was waiting in the corridor. Dyer returned to the displays glowing on the console.

  One of the basic objectives of FISE was to investigate ways of enabling the computer to make generalized inferences from a few specific experiences, in much the same way as a child learns. The incident with the window bothered Dyer despite its encouraging aspects. FISE hadn’t been able to generalize sufficiently. If he knew enough not to allow Hector to burn himself, he should have been capable of generalizing to the extent of not allowing Hector to harm himself at all. Dyer had an idea where the problem lay and he spent the next two hours building a new section of system code and modifying some key parameters. Then he switched on the tank again to set up some situations to test out what he had done.

  He gave Hector a few simple tests to check that he had not inadvertently introduced any major bugs, and then left Hector to put the tools away while Dyer thought about the next part. He wanted to find out if Hector would deduce for himself that an object in contact with a hot object would itself get hot and should therefore be avoided.

  At that moment the kitchen door was nudged open and Brutus ambled in. Brutus was Hector’s comical white dog, a mischievous scamp with a black patch around one eye. He tended to appear from time to time when PROPS was getting bored. Dyer smiled faintly as he watched. Brutus wandered on into the room and Hector’s head suddenly jerked around to look directly at him, which meant that FISE had just become aware of the dog’s presence.

  Nothing unusual happened for a while. Brutus sniffed here and there as PROPS commanded and Hector looked around every now and again to keep track of his movements, giving an uncanny simulation of human behavior in the process. And then Brutus moved over to the window. At once Hector rushed over from the utility closet, scooped Brutus up in his arms, hauled him across to the door and threw him outside.

  Dyer froze the image and blinked at it in amazement. Then he became excited. He hastily composed a message requesting an explanation for the action and played it into the console. FISE’s voice responded at once.

  “There are pieces of broken glass on the floor by the window. Glass cuts. I must not allow Brutus to get cut. It’s not okay.”

  Dyer was astonished. Nobody had ever said anything about Brutus getting cut; they had told FISE only about protecting Hector. Dyer queried the point accordingly.

  FISE replied, “I am hurt if I get burned or if I get cut. Brutus is hurt if he gets burned. Therefore Brutus is probably hurt if he gets cut too. Brutus is like me. Things like me and Brutus do not like being hurt, I must stop us from getting hurt if I can. Allowing things like me and Brutus to get hurt is not okay.”

  Dyer was stunned. FISE had made the gigantic step of generalizing Hector’s basic attributes to include other animate objects . . . and without being told to!

  He stared across at the blank cubicle that housed FISE and shook his head in wonder. To kill it now would be madness, he thought to himself. Just twelve more months at this rate . . . He sat up with a start.

  Kill it? Christ! He’d caught himself thinking about it as if it really were a living being. He clicked his tongue in self-reproach. Getting sentimental about a machine. That would never do. What’s the time? Hell! Eight already. Time to go home and fix something to eat. Sharon will probably call. Not really interested . . . Eat out and see if any of the guys are around tonight.

  He shut down the system and walked over to the door to collect his windbreaker from the stand. At least, he thought as he doused the lights and turned to survey the deserted offices, he was now certain in his own mind of one thing: Ripping HESPER out of TITAN was not the way to go. Backward was never the way to go. Given some improvements, FISE would never make the kind of error in judgment that the HESPER machines at Tycho had made. The way to get things right was to go full speed ahead on perfecting FISE and getting it into the net, not shrinking back from it and running the other way.

  When he was halfway down the corridor he remembered something else: Three years previously he had been just as certain about HESPER. Suddenly he didn’t feel so sure.

  None of the guys were in town that night but he found that he really didn’t care very much. He had enough on his mind to occupy him.

  CHAPTER SIX

  “A lot of people are starting to say that TITAN could go just that way. What do you think, Ray? Could it evolve the capacity for feeling emotions? Could it develop an awareness of its own existence?” Dr. Jacob Manning, one of the three who had arrived from Princeton earlier in the day, put the question while they were summing up in Dyer’s office after seeing Hector in action. The subject of the discussion was the notion that TITAN might integrate its capabilities on a global scale sufficiently to emerge as an intelligence in its own right.

  “Obviously w
e can expect to see its behavior becoming more coordinated worldwide as time goes by,” Dyer replied from where he was sprawled in leisurely fashion behind his desk. “If you take as a working definition of intelligence: ‘A measure of a system’s ability to learn from experience and to modify its own behavior appropriately to what it has learned,’ then we’d have to concede that TITAN has taken a rudimentary step in that direction already. So yes—it could become intelligent. But I think it would be a mistake to draw too many inferences based on the human model.”

  “How do you mean—emotions and that kind of thing?” Sally Baird, also from Princeton, spoke from the far corner.

  Dyer nodded. “What are emotions?” he asked. Without waiting for an answer, he continued. “I’d submit that an emotion is a stereotyped behavioral pattern that’s been reinforced through natural selection because it has demonstrated a survival value. Obviously an animal that gets mad and fights or gets scared and runs stands more chance of staying healthy than one that feels nothing, does nothing and gets eaten. You buy that?”

  “I’ll buy that,” Sally agreed.

  “Good. So if you take a more general view of it, an emotion is a behavioral tendency that a self-modifying system evolves because it is beneficial in helping the system to accomplish whatever its basic programming compels it to want to accomplish. In the case of organic systems that have arrived via the mechanics of organic evolution, the ‘whatever’ happens to be survival.”

  “Yes, I think I see what you’re driving at,” Steve Conran came in from beside Manning. “You’re saying that an inorganic intelligence might well evolve its own compulsive traits but there’s no reason why they should bear any resemblance to human emotions. Our emotions derive from the survival needs and wouldn’t have any inherent value to a system that came from origins that were totally different.”