text: Are you sure?
  This action might not be possible to undo. Are you sure you want to continue?
  Karl Winklmann Carolyn J. C. Schauble
  Department of Computer Science University of Colorado at Boulder
  September 24, 2002
  c 2002 Karl Winklmann, Carolyn J. C. Schauble
  2
  3
  Preface
  Unlike most textbooks on theory of computation these notes try to build as What is diﬀerent much as possible on knowledge that juniors or seniors in computer science tend to have. For example, the introduction of undecidability issues in the ﬁrst chapter starts with examples of programs that read and process other programs that are familiar to the students: Unix utilities like grep, cat, or diff. As another example, the proof of Cook’s Theorem (NP-completeness of Boolean satisﬁability) [Coo71] is done in terms of “compilation”: instead of the usual approach of turning a nondeterministic Turing machine into a Boolean expression, we compile a nondeterministic C++ program into machine language, and the resulting “circuit satisﬁability” problem into the problem of determining satisﬁability of Boolean expressions. All material is presented without unnecessary notational clutter. To get to strong results quickly, the text starts with the “machine model” students already know as plain “programs.” When a proof requires a substantial new concept, the concept is ﬁrst presented in isolation. For example, before we prove that all problems in P can be transformed into Sat, we show transformations for a few concrete examples. This allows the students to understand the concept of transformation into Boolean expressions before we present the generic argument that covers a whole class of such transformations. Pedagogically this makes sense even though the examples get superseded quickly by the more general result. Students should know the basic concepts from a course on algorithms, including Prerequisites the diﬀerent types of algorithms (greedy, divide-and-conquer, etc.) and they should have some understanding of algorithm design and analysis and of basic concepts of discrete mathematics such as relations, functions, and Boolean operations. Introductions to such concepts can of course be found in many books, e.g., [Sch96] or [Dea96]. These notes are not meant for self-study. The concepts are often too subtle, Don’t try this at home and these notes are generally too terse for that. They are meant to be read by students in conjunction with classroom discussion.
  Karl Winklmann Carolyn Schauble Boulder, August 5, 2002
  4 .
  . . . . . . . 7 7 11 12 13 15 21 25 25 2 Finite-State Machines 31 2. . . . 1. . . 1. . . . . . . .1 Context-Free Grammars . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 A Proof By Contradiction . . .4 More “NP-Complete” Problems .3 A Proof of Undecidability . . . . . . . . . . . . . . . . . . . .1 Strings. . . . . . . . . . . . . . . . . . .7 The Unsolvability of the Halting Problem 1. . . . . .2 The Myhill-Nerode Theorem . . . . .1 State Transition Diagrams . 1. . . . . . . . . . . . . . . . 3. . . . . . .3 The Pumping Lemma for Regular Languages . . . . . . . . . . . 62 3 Context-Free Languages 3. . . . . . . . . .4 A Characterization of Context-Free Languages . 89 . . . . . . . . .5 The “Holy Grail”: P = NP? . .3 The Pumping Lemma for Context-Free Grammars 3. . . . . . . . . .3 The NP-completeness of Satisfiability . . 4. . . . . . . . . . . . . . . . 3. . . . . . . . . . . . . . . .2 NP . . . . Properties. . . . . . . . . . . 98 . . . 31 2. . . . .Contents 1 Computability 1. . . . . . . . . . . . . . . .6 The Undecidability of All I/0-Properties . .1 The State of Knowledge Before NP-Completeness 4. 1. . 41 2. 4 Time Complexity 4. 1. . . . . . . 1. . . . . . . . . . 69 69 71 76 84 89 . . . . . . . . . . . .4 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Transformations . 93 . Programs . . . . . . . . . . . Bibliography . . . . . . . .8 G¨ odel’s Incompleteness Theorem . . 101 111 5 . . . . . . . . . . . . . . . . . . 4. . . . . . . . . . . . . . . . . 4. . . . . . . . . . . . . . . . . . . . . . . . . 95 .2 Parsing . . . . . . .
  6 CONTENTS .
  Examples of such “tools” are editors. Can this be made precise? Theory of computation attempts just that. compilers.Chapter 1 Computability We write programs in order to solve problems. The program “oddlength.cpp” determines whether or not its input consists of an odd number of characters. But if a problem is ”too hard” and our program is ”too simple. For example. 1. If we ran the program in Figure 1. Yes or No (assuming that the program 7 . The sophistication of our program needs to match the intrinsic diﬃculty of the problem. Some such programs determine properties of programs they are operating on. The approach is mathematical: we try to prove theorems. lint. interpreters. debuggers. even without any restrictions on the programs we will be able to prove in this ﬁrst chapter that many problems cannot be solved.1 Strings. The central piece of mathematics in this ﬁrst chapter is a proof technique called “proof by contradiction. Properties.” this won’t work. later we will restrict the kinds of data structures the programs use. utilities such as — to pick some Unix examples — diff. a compiler among other things determines whether or not the source code of a program contains any syntax errors.g. In this ﬁrst chapter we do not place any restrictions on the programs. Maybe surprisingly. We will study classes of programs. trying to characterize the classes of problems they solve.cpp oddlength < phonelist the response would be one of two words. cat. Programs Programs often operate on other programs. e. by issuing the two commands: CC -o oddlength oddlength. Figure 1. rcs.” The central piece of computer science is the notion that the input to a program can be a program.1 on some text ﬁle. more and lots of others.1 shows a much more modest example. grep.
  We could of course run this program on the source texts of programs. COMPUTABILITY Figure 1.8 void main( void ) { if (oddlength()) cout << "Yes". } return( toggle ). writing this program was a “constructive” proof of the following theorem. else cout << "No". 2 The “input” to a function is that part of the standard input to the program that has not been read yet at the time the function gets called. To show that there was a program as claimed by the theorem we constructed one. } CHAPTER 1.cpp” that determines whether or not its input consists of an odd number of characters was stored in oddlength.1: A program “oddlength.) In theory of computation the word “decidable” is used to describe this situation. . (Equivalently. } boolean oddlength( void ) { boolean toggle = FALSE.cpp and that there was indeed a text ﬁle phonelist and that this all is taking place on a system on which these commands make sense1 ).cpp The response in this case would be No because it so happens that the source code of this program consists of an even number of characters. even on its own source text: CC -o oddlength oddlength. To avoid clutter we leave out the usual “#include” and “#define” statements and assume that TRUE and FALSE have been deﬁned to behave the way you would expect them to. Mathematically speaking.cpp oddlength < oddlength. there is a boolean function that returns TRUE or FALSE depending on whether or not its input2 contains an odd number of characters. while (getchar() != EOF) { toggle = !toggle. as in 1 For our examples of programs we assume a Unix environment and a programming language like C or C++. Theorem 1 There is a program that outputs Yes or No depending on whether or not its input contains an odd number of characters.
  will not terminate}. The bad news about Selflooping is spelled out in the following theorem. In terms of sets. sets property Π is the same as x being in the set {z : z has property Π}. when run on x. PROGRAMS void main( void ) { char c. . Let’s say a program has property Looping if there exists an input on which the program will not terminate. having property Looping means being a member of the set LLOOPING = {x : x is a syntactically correct program and there is an input y such that x. Concern about whether or not a program contains an odd number of characters More interesting is not usually uppermost on programmers’ minds. Theorem 1 (yet another wording) Oddlength is decidable.2 and 1. when run on y . The program in Figure 1. An example of a property of properties of programs much greater concern is whether or not a program might not always terminate. The notion of a “property” is the same as the notion of a “set”: x having Properties vs. Let’s make that one input the (source code of the) program itself.3.1. Theorem 2 Selflooping is not decidable. The programs in Figures 1.2: A program “x. Selflooping means being a member of the set LSELFLOOPING = {x : x is a syntactically correct program and.3 both have this property. while (c != EOF) { while (c == ’X’). PROPERTIES. STRINGS.cpp” that loops on some inputs Theorem 1 (new wording) Membership in the set LODDLENGTH = {x : |x| is odd } is decidable. This suggests yet another rewording of Theorem 1 which uses standard terminology of theory of computation. In terms of sets. let us simplify matters and worry about only one input per program. will not terminate}. } } 9 Figure 1.1. If we use the name Oddlength for the property of consisting of an odd number of characters then the theorem reads as follows. Let Selflooping be the property that a program will not terminate when run with its own source text as input.1 does not.2 and 1. c = getchar(). Instead of being concerned with whether or not there is some input (among the inﬁnitely many possible inputs) on which a given program loops. Two examples of programs with property Looping are shown in Figures 1. // loop forever if an ’X’ is found c = getchar().
  programs The deﬁnitions of Looping and Selflooping use the notion of a “syntactically correct program. This program can be run on some input by cat phonebook | grep Boulder | sort grep Boulder phonebook | sort or by ( grep Boulder | sort ) < phonebook . all) inputs What this means is that there is no way to write a program which will print "Yes" if the input it reads is a program that has the property Selflooping.” that loops on some (in fact. say. COMPUTABILITY Figure 1. If not. we compile it into the same object code as the program “main () {}.-o x x. which starts on page 12. as we will do a little later. y. One convenient extension to this language is to regard Unix shell constructs like grep Boulder | sort as programs.10 void main( void ) { while (TRUE). and the string y in a ﬁle named. x. we compile it with our trusty old C or C++ compiler. say. let’s call it C−−. maybe. and then we use our new C−− compiler and do CC-.3. If the string is a C or C++ program. Strings vs. In fact. and will print "No" otherwise.cmm x < y So let’s assume from now on that we are programming in “C−−. It means that we store the string x in a ﬁle named.cpp. they are about fundamental aspects of any form of computation. Surprising. Our results are not about peculiarities of somebody’s favorite programming language.” Then every string is a program and “running x on y ” now makes sense for any strings x and y . } CHAPTER 1.cmm.” and thus avoid having to mention “syntactically correct” all the time. in which every string is a program.3: Another program. in Section 1. We could do all our programming in a language. “useless.” The choice of programming language does not matter for any of our discussions. to mention syntax at all is a red herring. but true and not even very hard to prove.
  1. contradicting the fact that p and q were relatively prime. Therefore p itself must contain a factor of 2. The conclusion then is that the assumption must have been false.” Squaring both sides of (1. New York. pp. The “something” that we know is not true often takes the form of “A and not A. 1972.3) This shows that p2 contains a factor of 2.2 A Proof By Contradiction Before we prove our ﬁrst undecidability result here is an example that illustrates the technique of proving something “by contradiction.5) (1. (A rational number is one that is equal to the quotient of two integers.) 3 Proof (Pythagoreans. Then √ p 2= (1.3) and (1. the theorem true.4) yields 2 × q2 = 2 × 2 × r which simpliﬁes to q2 = 2 × r (1.4) for some integer r.2) 2 × q 2 = p2 (1.” √ Theorem 3 2 is irrational. assume that 2 is rational.2.” Proving a theorem by contradiction starts with the assumption that the theorem is not true.6) which shows that q also contains a factor of 2. An irrational number is one that is not rational. which gives p2 at least two factors of 2: p2 = 2 × 2 × r (1. A PROOF BY CONTRADICTION 11 1. 3 according to Kline. √5th century B. Next we apply this proof technique to get our ﬁrst undecidability result. Mathematical Thought from Ancient to Modern Times. Combining (1. From this assumption we derive something that we know is not true.1) q for two integers p and q which have no factors in common: they are “relatively prime.1) yields 2= and therefore p2 q2 (1.32-33 . But p2 cannot have a factor of 2 unless p did.) For the sake of deriving a contradiction.C.
  Then. else while (TRUE).4: A program “diagonal. sending it into an inﬁnite loop and hence have property Selflooping. assume that there is a programs that decides if its input has property Selflooping. when run with its own source code as input. COMPUTABILITY // stop. Does the program in Figure 1.4.4 itself have property Selflooping? Claim 1 It does not. If such a program existed we could re-write it as a function boolean selflooping() that decides whether or not its input has property Selflooping. the function selflooping would return FALSE to the main program. Claim 2 It does. Proof of Claim 2 Assume it did not.cpp” that would exist if Selflooping were decidable 1. Proof of Claim 1 Assume it did. proving our initial assumption wrong. a contradiction. } CHAPTER 1.. 4 Why diagonal is a ﬁtting name for this program will become clear in the next section. when running the program with its own source code as input. Then. } boolean selflooping( void ) { . a contradiction. when running the program with its own source code as input. These two claims form a contradiction. We could then write the program diagonal4 shown in Figure 1. does not terminate. making it terminate and hence not have property Selflooping.3 A Proof of Undecidability Proof of Theorem 2 For the sake of deriving a contradiction.12 void main( void ) { if (selflooping()) . the function selflooping would return TRUE to the main program. . don’t loop // do loop forever // // // // // returns TRUE if the input is a program which. there cannot be a program that decides if its input has property Selflooping.. FALSE otherwise Figure 1.
  In other words. that contains all reals from the interval (0. p.4. Then there is a sequence that contains all reals and a subsequence r1 . . The number d thus deﬁned diﬀers from every number in the sequence r1 . . is a subtlety in this proof stemming from the fact that two diﬀerent decimal expansions can represent the same number: 0. r2 . . .1. = = = = = = = = = 0 0 0 0 0 0 0 0 0 . . 1]. Why is this of concern? And why is the proof ok anyway? 6 There 5 Cantor . .) Proof (Cantor. .” To understand that aspect of the proof. (See Figure 1. . 1890/1891) 5 Assume the real numbers are countable. according to Kline. . r2 . . . Consider the inﬁnite matrix that contains in row i a decimal expansion of ri .5: The diagonal in the proof of Theorem 4 1. . there is a sequence r1 . r3 .5 for an illustration. Theorem 4 (Cantor. and whose ith digit is 1 otherwise. r4 . r3 . 1) were in the sequence. . . (A set is countable if there is a function from the natural numbers onto the set.. . . .) Consider the number d whose ith digit is 9 if the matrix entry in row i and column i is 1.09999 . let’s look at a classical theorem with a diagonalization proof. . r3 . . 1873) The real numbers are not countable.6 gave a diﬀerent ﬁrst proof in 1873.. . one digit per column. = 0. DIAGONALIZATION digits → numbers ↓ d r1 r2 r3 r4 r5 r6 r7 r8 .997. .10000 . 13 1 10 1 100 1 103 1 104 1 105 1 106 1 107 1 108 ··· ··· ··· ··· ··· ··· ··· ··· ··· ··· 1 6 0 8 1 0 0 1 4 1 2 8 4 0 9 4 1 4 9 8 8 1 0 9 6 0 9 1 4 4 1 0 9 2 0 4 1 1 1 0 0 9 6 2 3 9 1 1 8 1 0 1 1 4 9 0 3 4 0 9 4 1 4 9 4 8 1 0 9 6 0 1 Figure 1. . r5 . contradicting the assumption that all reals from [0. .4 Diagonalization The proof of Theorem 2 was a special kind of proof by contradiction called a “diagonalization proof. r2 . that contains all members of the set.
  . be a sequence of all programs. . nobody would know how to prove Theorem 2 nor any of the other undecidability results in this chapter. This monopoly of one proof technique will cause problems in the study of computational complexity in Chapter 4. which we assumed to exist at the start of the proof of Theorem 2.14 CHAPTER 1. COMPUTABILITY programs (used as inputs) → programs ↓ p1 p2 p3 p4 p5 p6 diagonal : p1 p2 p3 p4 p5 p6 p7 p8 . . whether unsolvable in theory or not. is open to a heuristic approach. p3 . It does suggest that heuristics would be the way to go if one wanted to create software tools to solve in some practically useful sense any of the problems which this theory shows “unsolvable.6: The diagonal in the proof of Theorem 2 How does this proof of Cantor’s Theorem relate to our proof of Theorem 2? Programs are ﬁnite strings and hence countable. Diagonalization does not work nearly as well in that context.” . Heuristics Every computational problem.cpp does not terminate on input pi . does not terminate. and a 1 if it does. (See Figure 1.) The program diagonal.4. then diﬀers from each program in the sequence. Let p1 . The reason is that if program pi does terminate on input pi . when run with the source code of program j as input. Other proof techniques? It is remarkable that without diagonalization. and vice versa.cpp of Figure 1. we will not be able to prove some very basic conjectures. which injects a concern about eﬃciency into the discussion. The theory presented here does not address the feasibility of heuristic methods. : : : : : : : : 1 0 0 0 1 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 1 0 1 0 1 1 0 p7 0 0 0 0 0 0 0 1 0 p8 0 0 0 1 0 0 0 0 1 ··· ··· ··· ··· ··· ··· ··· ··· ··· ··· Figure 1.6 for an illustration. (Unlike a sequence of all reals. such a sequence does exist since programs are ﬁnite strings.) Consider the inﬁnite boolean matrix which has a 0 in row i and column j if program i. . contradicting the fact that all programs were in the sequence. p2 . . Without any other technique available. then diagonal.
  let’s look at some examples of transformations in other settings.” Note the similarity to considering a program that halts on all those programs that are “selﬂooping. Consider the Unix command sort It can be used like sort < phonebook to sort the lines of a text ﬁle. An example known to Greek philosophers over 2000 years ago is the selfreferencing statement that says “This statement is a lie.” 1. called Sorting. we will exploit the observation that they have strong connections to Selflooping.” which is not a problem yet. But what if the problem you want to solve is not Sorting. Before we apply the concept of transformations to our study of (un)decidability. TRANSFORMATIONS 15 Digressing for a moment from our focus on what programs can and cannot do.” the trouble stems from trying to compile a “catalog of all non-self-promoting catalogs. A (seemingly) non-mathematical example arises when you try to compile a “Catalog of all catalogs. we will show that there are “transformations” of Selflooping into other problems.” This may look like a pretty silly statement to consider.1. stltctwb for short? You could write a program “stltctwb” to solve stltctwb and use it like stltctwb < phonebook . but it is true that the variant that says “This is a theorem for which there is no proof. “sort” solves a certain problem.5. Here are a few others. but the problem of Sorting-Those-Lines-That-Contain-the-Word-“Boulder”. connections such that the undecidability of Selflooping “rubs oﬀ” on them. Self-reference in other programs that process programs are not the only example of self-referencing domains causing trouble. To prove other properties undecidable. G¨ odel’s Incompleteness Theorem. and far from the oldest one. but what about a “Catalog of all those catalogs that do not contain a reference to themselves?” Does it contain a reference to itself? If we call catalogs that contain a reference to themselves “self-promoting.5 Transformations Selflooping is not the only undecidable property of programs.8 on page 25. Speciﬁcally. a problem that is so common and important that whole chapters of books have been written about it (and a solution comes with every computer system you might want to buy). More on that in Section 1.” underlies what many regard as the most important theorem of 20th -century mathematics.
  (The “cat phonebook | ” merely provides the input. then there is a program that solves problem A. then problem B is unsolvable as well. (Let’s make it an “Observation.) The general pattern is that if “b” is a program that solves problem B.” Fol“decidable?” lowing this usage. we speak of “solvable and unsolvable problems” and “decidable and undecidable questions. given a string like satisﬁability (Sat) 10 + 11 = 011 (1.) Observation 1 If there is a program that transforms problem A into problem B and there is a program that solves problem B. “problems” get “solved” and “questions” get “decided. COMPUTABILITY cat phonebook | grep Boulder | sort This is an example of a transformation.16 But probably you would rather do CHAPTER 1.” That way we don’t have to spell out a proof.7) . the preﬁx “un-” means that there is no program to do the job. If we had to formulate a theorem about the matter it would read as follows. A problem is solvable if there exists a program for solving it. Observation 2 If there is a program that transforms problem A into problem B and problem A is unsolvable. solving the Selflooping problem for a program x is the same as deciding whether or programs have the property Selflooping. For example. In either case. The reason for doing such transformations is that they are a way to solve problems. addition into Boolean Let Binary Addition be the problem of deciding. Transforming binary For another example of a transformation. consider the following two problems. “Solvable” or In good English. The command “grep Boulder” transforms the stltctwb problem into the Sorting problem. The command “grep Boulder | sort” solves the stltctwb problem. then a to b is called a transformation of problem A into problem B. and a to b | b solves problem A.” For the substance of our discussion this distinction is of no consequence. the following rephrasing of the above observation will be more useful to us. Since at the moment we are more interested in proving that things cannot be done than in proving that things can be done.
  It adds up two 2-bit binary numbers. α represents a correct addition of two binary numbers ⇔ t(α) is a satisﬁable Boolean expression. but not all seven can be made true with one and the same assignment.7. c3 .) Let’s restrict the problem to strings of the form x2 x1 + y2 y1 = z3 z2 z1 (1. (This one isn’t. Let’s call this restricted version 2-Add. In terms of the circuit this means that if the inputs and outputs are ﬁxed to reﬂect the values given in (1.5. and c3 . or at least one wire has to have diﬀerent values at its two ends.1. z2 . but no choice of values for them will make the whole expression true. and zi being 0 or 1.7) to the expression (( (( (( (( (( (( (( 0 0 1 1 c2 c2 c3 ⊕ ∧ ⊕ ∧ ⊕ ∧ ∨ 1 1 1 1 z2 z2 c3 )⇔ 1 ) ⇔ c2 ) ⇔ z2 ) ⇔ c3 )⇔ 1 ) ⇔ c3 )⇔ 0 ) ) ) ) ) ) ) ∧ ∧ ∧ ∧ ∧ ∧ which. Let Boolean Satisﬁability (Sat) be the problem of deciding.8) with each xi . c3 . TRANSFORMATIONS 17 whether or not it is a correct equation between binary numbers. c2 . How can we construct such a transformation? Consider the Boolean circuit in Figure 1. and c3 . at least one gate in the circuit has to end up with inconsistent values on its input and output wires. Note that every single one of the seven “conjuncts” can be made true with some assignment of truth values to c2 . A transformation of 2-Add to Sat is a function t such that for all strings α. The circuit is “not satisﬁable. whether or not there is an assignment of truth values to the variables in the expression which makes the whole expression true. Instead of drawing the picture we can describe this circuit equally precisely by the Boolean expression (( (( (( (( (( (( (( x1 x1 x2 x2 c2 c2 c3 ⊕ ∧ ⊕ ∧ ⊕ ∧ ∨ y1 y1 y2 y2 z2 z2 c3 ) ⇔ z1 ) ⇔ c2 ) ⇔ z2 ) ⇔ c3 ) ⇔ z2 ) ⇔ c3 ) ⇔ z3 ) ) ) ) ) ) ) ∧ ∧ ∧ ∧ ∧ ∧ This suggests transformation of 2-Add to Sat that maps the string (1. is not satisﬁable. z2 .” . yi . appropriately. There are four variables to choose values for. given a Boolean expression.7).
  COMPUTABILITY x1 y1 x2 y2 + ^ c2 + z’ 2 ^ c’ 3 z1 + ^ c’’ 3 v z2 z3 Figure 1.18 CHAPTER 1.7: A circuit for adding two 2-bit binary numbers .
  5. TRANSFORMATIONS a 19 b f c e d Figure 1. e. because there is a person who needs to attend both. aBLUE ) true if and only if the vertex a gets colored green (red. blue).” The question then becomes. Are three slots enough? Instead of activities being scheduled into time slots. The statement that a and b get diﬀerent colors can then be made in the “language of Boolean expressions” as (aGREEN ∧ bGREEN ) ∧ (aRED ∧ bRED ) ∧ (aBLUE ∧ bBLUE ) (1. . Each node represents an activity. For example. a meeting of some group.7 In fact. Graphs like the one shown in Figure 1.1.g. but not Looping to Oddlength. e. Programming eﬀort is not addressed by this theory. Since activities a.8: A graph of activities and conﬂicts Note that transformations do not always go from a harder problem to an easier one. But the destination problem can be much harder than the source problem.” or “3Color” for short. the opposite is closer to the truth. without adjacent nodes getting the same color? This problem is known as “Graph 3-Colorability.” Most often the resource we care about is computing time. The idea behind the transformation is to make the variable aGREEN (aRED .8 can be used to represent scheduling Transforming Graph problems.g. and e are all conﬂicting with each other. “Harder” and “easier” do not mean harder or easier to program. and 3-Colorability into Sat an edge between two nodes means that the two activities must not be scheduled at the same time. b. graph theorists talk about “nodes” being “colored. The destination problem can be easier than the source problem only to the extent of the computational eﬀort expended in carrying out the transformation. you can transform Oddlength to Looping. The 7 “Harder” and “easier” mean “needing more and less of a computational resource.9) We need to make one such statement for each of the nine edges in the graph. are three diﬀerent colors enough to color all the nodes. there is no way to schedule all the activities into fewer than three time slots.
  on any input (which it ignores — see previous item). a choice that corresponds to avoiding color conﬂicts by not coloring the nodes at all. do “( x < x. to carry out the transformation for any graph G = G(V.10.10) In general. For node a this can be done by the expression (aGREEN ∧ aRED ∧ aBLUE ) ∨ (aGREEN ∧ aRED ∧ aBLUE ) ∨ (aGREEN ∧ aRED ∧ aBLUE ) (1..y )∈E This expression EG is satisﬁable if and only if the graph G is 3-colorable. we just have to make a statement like (1.cpp ) < input. Therefore we could take a copy of the editor and hard-code those commands into it (or.2 we could create the program in Figure 1. we can simplify this editing task greatly by making y the one-line program “( x < x. and 2.” which has all the right properties.e. Editing programs To make Observation 2 useful for proving undecidability results. E ).9) for every edge. Proof We can achieve this transformation “by hand” in a session with our favorite text editor. Lemma 1 There is a program edit which.10) for every node of the graph and a statement like (1.20 CHAPTER 1. put the commands into a “script ﬁle” or a “macro” for our editor8 ).cpp ).cpp” from Figure 1. y never reads any input. To see that it ignores its input. creates as output a program y with the following two properties. if we start with x being the program “x. COMPUTABILITY conjunction of these nine statements does not yet complete the transformation because it could always be satisﬁed by making all the variables false. yet. we need to ﬁnd transformations from one property of programs into another. As an example. preferably and equivalently. given as input a program x. runs like x would on input x. We can even do this in a way where the editing commands do not depend on the program x that is being editing. we have to write this expression EG : ((xGREEN ∧xRED ∧xBLUE )∨(xGREEN ∧xRED ∧xBLUE )∨(xGREEN ∧xRED ∧xBLUE )) x∈V ((xGREEN ∧ yGREEN ) ∧ (xRED ∧ yRED ) ∧ (xBLUE ∧ yBLUE )) (x. y .” 8 Better . i. We can complete the transformation by insisting that each node get exactly one color assigned to it. 1.
  . By Observation 2 this proves Looping undecidable. 10. . 3. THE UNDECIDABILITY OF ALL I/0-PROPERTIES void main( void ) { x_on_x_no_input(). .11 illustrates a transformation of Selflooping into the negation of property One to Ten. the entry in the right-hand column is an inﬁnite string. Theorem 6 One to Ten is undecidable. . If the program loops forever on some input and keeps producing output. This is necessary because otherwise the transformation would not work in the case of programs x that print the numbers 1. 3.6 The Undecidability of All I/0-Properties Recall that Looping is the property that there exists an input on which the program will not terminate. the function x on x no input no output suppresses whatever output the program x might have generated when run on its own source code as input.. . Let One to Ten be the problem of deciding whether or not a program prints the numbers 1. 3. . If you were working on an early homework assignment in Programming 101. but instead of reading input it ‘‘reads’’ from a string constant 21 Figure 1.1. Theorem 5 Looping is undecidable. 10. Proof The program edit of Lemma 1 transforms Selflooping into Looping. . the table shows the output that the program produces. so is the property One to Ten. Since the negation of One to Ten is undecidable. 2. For every input. . } // // // // runs like x.13 shows an example of an “input-output table” IOx of a program x.. . 10. 2. How many more undecidability results can we derive this way? Figure 1. “working correctly” may have meant printing the numbers 1. . Note that unlike the function x on x no input in the previous proof. .6. .9: The result of editing program x 1. What other properties of programs are often of interest? The prime consideration usually is whether or not a program “works correctly. Proof Figure 1.” The exact meaning of this depends on what you wanted the program to do. } void x_on_x_no_input( void ) { . 2. .
  10: The result of editing the program “x.\n\ ". } char mygetchar( void ) { static int i = 0. c = mygetchar().2 . } void x_on_x_no_input( void ) { char c.22 CHAPTER 1. void main( void ) { x_on_x_no_input(). COMPUTABILITY #define EOF NULL // NULL is the proper end marker since // we are ‘‘reading’’ from a string.\n\ \n\ c = getchar(). // loop forever if an ’X’ is found c = mygetchar(). // end of input string //////////////////////////////// return( inputstring[i++] ). static char* inputstring = // start of input string ////////////////////////////// "\ \n\ void main( void )\n\ {\n\ char c.\n\ }\n\ }. // loop forever if an ’X’ is found\n\ c = getchar().\n\ while (c != EOF)\n\ {\n\ while (c == ’X’). } } Figure 1.cpp” of Figure 1. while (c != EOF) { while (c == ’X’).
  // input it ‘‘reads’’ // from a string constant. one_to_ten(). for (i = 1.11: Illustrating the proof of Theorem 6 void main( void ) { x_on_x_no_input_no_output().. but // instead of reading ...1... // and it does not } // generate any output void one_to_ten( void ) { int i. } // runs like a program // that does not have // property PI Figure 1. i++) cout << i << endl.6. } void x_on_x_no_input_no_output( void ) { .. i <= 10. notPI(). THE UNDECIDABILITY OF ALL I/0-PROPERTIES 23 void main( void ) { x_on_x_no_input_no_output(). } void x_on_x_no_input_no_output( void ) { // runs like x. } void notPI( void ) { . } Figure 1.12: The result of the editing step rice on input x .
  let’s call it Recursive. Yes. No. a property of programs is an inputoutput property if the information that is necessary to determine whether or not a program x has the property is always present in the program’s input-output table IOx . ··· Figure 1. Thus a property of programs is nontrivial if there is at least one program that has the property and one that doesn’t. . a property Π of programs is an input-output property if the following is true: For any two programs a and b. For example. The property of running in linear time is not an input-output property. More formally. or neither does. • the property of printing the words Core dumped on some input. .. A property of programs is trivial if either all programs have it. • the property of not looping on any inputs of length less that 1000.13. you would still not know whether the program was recursive. is not an input-output property. Core dumped. Yes. Yes. or if none do. CHAPTER 1. Even if you knew the whole input-output table such as the one in Figure 1. . Yes. .24 Inputs λ 0 1 00 01 10 11 000 001 010 . Informally speaking. Segmentation fault. COMPUTABILITY Outputs Empty input file. No. No. nor the property of being a syntactically correct C program. Yes. Examples of input-output properties include • the property of generating the same output for all inputs. the property of making a recursive function call on some input. if IOa = IOb then either both a and b have property Π.9 9 There are only two trivial properties of programs. Yes. They merely serve to deﬁne what an “inputoutput property” is. No. nor is the property of always producing some output within execution of the ﬁrst one million statements.13: The “input-output function” of a lousy program These tables are not anything we would want to store nor do we want to “compute” them in any sense. Yes.. One is the property of being a program . Okay. . .
  Then Selflooping can be solved by running halting with the two arguments being the same. it is worth pointing out that our proof techniques easily let us prove what is certainly the most famous theorem in mathematical logic and one of the most famous in — all programs have that property. i. as arguments and prints Yes if x. } (1. The function rice transforms Selflooping into Π.1. The other is the property of not being a program — no program has that property. Let’s ﬁrst prove the theorem for properties Π which this program (1. changing Yes to No and vice versa. call it rice. which proves Π undecidable.11) does have. Consider the program that is outlined in Figure 1. and print “No” otherwise.11) has the property Π or it doesn’t. x and y. assume there is a program halting which takes two ﬁle names. and then reversing the answer. when run on y.9 was created from x by the program edit of Lemma 1. Proof Assume Halting is decidable. halts. a program x and a string y . much like the program in Figure 1. It can be created from x by a program.7. these are the set of all programs and the empty set. when run on y .7 The Unsolvability of the Halting Problem The following result is probably the best-known undecidability result. In terms of sets. Let notPI be a function that runs like a program that does not have property Π. Theorem 8 (Unsolvability of the Halting Problem) There is no program that meets the following speciﬁcation.. There are two possibilities. and prints No otherwise.11) does not have property Π? In that case we prove Rice’s Theorem for the negation of Π — by the argument in the ﬁrst part of this proof — and then observe that if the negation of a property is undecidable. so is the property itself. 1. Proof Let Π be a nontrivial input-output property of programs.12. I. THE UNSOLVABILITY OF THE HALTING PROBLEM 25 Theorem 7 (Rice’s Theorem) No nontrivial input-output property of programs is decidable. . When given two inputs. Either the program main () { while (TRUE). print “Yes” if x. terminates. 1.e.e.8 G¨ odel’s Incompleteness Theorem Once more digressing a bit from our focus on programs and their power. What if the inﬁnite-loop program in (1.
  Now we prove Selflooping “unprovable. Monatshefte f¨ ur Mathematik und Physik. What if we enlarged the system. This is correct but note that the program had the axiom system “hard coded” in it.14 is a modiﬁcation of the program we used to prove Selflooping undecidable. done by a program. 10 Kurt G¨ ¨ odel. Proof If every true statement had a proof. Thus the existence of such a proof leads to a contradiction.26 CHAPTER 1. the program in Figure 1. Since no such proof exists the program keeps looking forever and thus does in fact have property Selflooping. The following lemma is an example. Does this mean that once we choose a set of axioms and proof rules. 38.. What is impossible is the design of an axiom system that is strong enough to provide proofs for all true conjectures. by the following algorithm. for example by adding one new axiom that says that this program has property Selflooping? Then we could prove that fact (with a one-line proof no less). Put another way. For any given program x the algorithm generates all strings in some systematic order until it ﬁnds either a proof that x has property Selflooping or a proof that x does not have property Selflooping. It is not the veriﬁcation of formal proofs that is impossible.e. i. Proof The program in Figure 1. Uber formal unentscheidbare S¨ atze der Principia Mathematica und verwandter Systeme I.14 does have property Selflooping but there is no formal proof of that fact within the system. So if we change the system we get a new program and the lemma is again true — for the new program. . Selflooping would be decidable. It is just that our (informal) proof cannot be formalized within the given system. Lemma 2 For any system of formal proofs. the program searches systematically for a proof that it does loop and if it ﬁnds one it terminates. 1931. 173–98. It is crucial for the above argument that veriﬁcation of formal proofs can be automated. it put an end to the notion that mathematics could somehow be perfectly “automated. 1931) For any system of formal proofs that includes at least the usual axioms about natural numbers.” When given its own source as input. Note that there is no contradiction between the fact that there is no proof of this program having property Selflooping within the given axiom system and the fact that we just proved that the program has property Selflooping. there are theorems that are true but do not have a proof in that system. The result was published in 1931 by Kurt G¨ odel and put an end to the notion that if only we could ﬁnd the right axioms and proof rules. COMPUTABILITY all of mathematics. we could prove in principle all true conjectures. our mathematical results will forever be limited to what can be proven from those axioms and rules? Not at all.” 10 Theorem 9 (G¨ odel’s Incompleteness Theorem.
  14: Illustrating the proof of Lemma 2 Exercises Ex. Give three such properties. 1.2. is there a set of edges so that each node is an endpoint of exactly one of the edges in that set.3. The function edit that is used in the proof of Theorem 5 transforms Selflooping not only to Looping but also to other properties. 1. That person hands you a ﬂoppy with the source code of a program that they claim does in fact carry out that analysis. 1. // in the proof system under consideration // and stops if and when it finds a proof // that its input has property SELFLOOPING } 27 Figure 1. Consider the following problem. (Such a set of edges is called a ”complete matching”..) Describe a transformation of this problem to the satisﬁability problem for Boolean expressions. whether or not p when run on input x goes into an inﬁnite loop. 1. Why does Diagonalization the diagonalization proof that shows that the real numbers are not countable not work for the rational numbers? Ex.1. Given a graph.¨ 1. Ex.8. . Ex. The rational numbers are countable (as argued in class).7. You may do this by showing (and explaining. given a program p and input x.8.6.5.. if it’s complicated) the expression for one of the above graphs. 1. 1. Is it decidable if a program will try to “de-reference a null pointer?” Ex. Let Selfrecursive be the property of programs that they make a recursive function call when running on their own source code as input. Ex. Describe a transformation from Selflooping to the property of Transformations not reading any input. 1. Somebody who has not taken this course cannot believe that there is no way to write a program that ﬁnds out. GODEL’S INCOMPLETENESS THEOREM void main( void ) { // systematically generates all proofs .4. Describe a diagonalization proof that Selfrecursive is not decidable. 1. Can you produce a program p and input x on which their program fails? Ex. Is it decidable if a program has a “memory leak?” Ex.
  15. but were not mentioned in class. Ex. 1. . For example. (a) Transform Oddlength into Selflooping. and.10. 10 If having a choice is not acceptable in a real “algorithm. (a) Would this still be a transformation of 3-Color into Sat? Explain why. Ex. (b) What does this prove? Ex. COMPUTABILITY Ex.9.16. hence undecidable. if t(G) is the Boolean expression the graph gets translated into by our transformation of 3-Color into Sat. Consider trying to transform 3-Color into Sat but using.14. Ex. 1.13. 1. “Autograders” are programs that check other programs for correctness. describe how you can translate that assignment of truth values back to a 3-coloring of the graph. 1. Ex. 1. ﬁnally.” we can always program the algorithm to prefer red over green over blue. Give three others that are nontrivial I/O properties. Is Selflooping an input-output property? More Undecidability Ex. (a) Transform Selflooping into the property of halting on those inputs that contain the word Halt. 1. (b) What does this prove? I/0-Properties Ex.28 CHAPTER 1.17. in place of (aGREEN ∧ aRED ∧ aBLUE ) ∨ (aGREEN ∧ aRED ∧ aBLUE ) ∨ (aGREEN ∧ aRED ∧ aBLUE ) the expression xGREEN ∨ xRED ∨ xBLUE . or why not. 1. an instructor in a programming course might give a programming assignment and announce that your programs will be graded by a program she wrote. Give three undecidable properties of programs that are not inputoutput properties and that were not mentioned in the notes. 1. If G is a graph that you are trying to ﬁnd a 3-coloring for. if someone handed you an assignment of truth values to the variables of t(G) that satisﬁed the expression. 1. Which line of the proof could otherwise be wrong? Ex. We needed to suppress the output in the x on x no input no output function used in the proof of Rice’s Theorem.11. (b) Could we still translate a satisfying assignment of truth values back into a 3-coloring of the graph? Explain how. or why not.12. Formulate and prove a theorem about the impossibility of program Results optimization. and looping on all others.
  a program which. 2. “Autograding” is possible.e. “Autograding” is not possible.. Containing only correct comments. . Ex. The output not depending on the input. 5. It depends on the programming assignment. when the input is a C program.e. and ”No” otherwise). a program which outputs ”Yes” if the input is a syntactically correct C program..8. 4. Containing comments. 1.). 2. 3. 1.18. ”Equivalent” means having the same IO table.¨ 1. Being a correct parser for C (i. Being a correct compiler for C (i. outputs an equivalent machine-language version of the input. GODEL’S INCOMPLETENESS THEOREM 29 What does theory of computing have to say about this? Choose one of the three answers below and elaborate in no more than two or three short sentences. Which of the following ﬁve properties of programs are decidable? Which are IO-properties? 1. 3.
  30 CHAPTER 1. COMPUTABILITY .
  the single circle. we can examine the values of its lone variable. We consider only programs whose storage requirements do not grow with their inputs. bugger on it and set a breakpoint just before the getchar() statement. Let’s give the program another input character. 2. from state B Yes.1 summarizes all of this. What does the program do when an EOF happens to come around? From state A the program will output No. in this chapter we restrict the programs under consideration very severely.1 on page 8. The double circle means that an EOF produces a Yes. This rules out dynamically growing arrays or linked structures. Applying mathematical induction will provide us with a second complete characterization. as well as recursion beyond a limited depth. where we did not place any restrictions on the kinds of programs we were considering. This is another “state” the program can be in. If we resume execution and provide an input character. The central piece of mathematics for this chapter is the notion of an “equivalence relation. When once more.The program oddlength. Σ is the set of all possible input 31 . The next time our debugger lets us look at things the program is back in state A. The value of toggle is FALSE at that time and the “instruction pointer” is of course just in front of that getchar() statement. in a picture the program stops for the ﬁrst time at that breakpoint. then at the next break the value of toggle will be TRUE and the instruction pointer will of course again be just in front of that getchar(). it bounces back and forth between state A and state B . As long as we keep feeding it input characters. in a picture. Let’s use a de. a No.Chapter 2 Finite-State Machines In contrast to the preceding chapter. Let’s refer to this situation as state A.1 State Transition Diagrams Consider the program oddlength from Figure 1. Figure 2. Let’s call it state B .” It will allow us to arrive at a complete characterization of what these restricted programs can do.
  (2. is indicated by an unlabeled arrow. a function from states and input symbols to states. 1 Purists would call the picture the “transition diagram” of the machine.1 An input string that makes the machine end up in an accepting (rejecting) state is said to be accepted (rejected) by the machine. FINITE-STATE MACHINES Σ A Σ B Figure 2. 11. single circles are rejecting states. 1} . . a start state. If M is a ﬁnite-state machine M then LM = {x : x is accepted by M } is the language accepted by M. 010. 101. Thus LMATCH = {0. 000. Labeling a transition with Σ means that any input character will cause that transition to be made. 111. and deﬁne the machine itself to be the mathematical structure that the picture represents. it is necessary to remember what the ﬁrst character and the last character were. How can a ﬁnite-state machine “remember” or “know” this information? The only thing a ﬁnite-state machine “knows” at any one time is which state it is in. . cxc : x ∈ {0. Terminology A picture such as the one in Figure 2. States drawn as double circles are accepting states. For every state and every input character there must be an arrow starting at that state and labeled by the input character. and a set of accepting states. c ∈ {0.1 describes a ﬁnite-state machine or a ﬁnite-state automaton. 1. . 0000. The initial state. Arrows that go from state to state and are labeled with input characters indicate which input characters cause what transitions. more formally. 0010.} or.1: The program oddlength as a ﬁnite-state machine characters. the input alphabet. Another Example Let LMATCH be the language of all binary strings that start and end with a 0 or start and end with a 1.32 CHAPTER 2. that we are dealing only with inputs consisting entirely of 0s and 1s). or start state. Input characters cause transitions between states.1) Can we construct a ﬁnite-state machine for this language? To have a chance to make the right decision between accepting and rejecting a string when its last character has been read. A ﬁfth state will serve as start state. This structure consists of a set of states. 1}}. when no characters have been read yet. LMATCH = {c. We need four states for the four combinations of values for those two characters (assuming. 00. . None of those four states would be appropriate for the machine to be in at the start. again.
  What does a ﬁnite-state machine need to remember about its input to handle this language? We could make it remember exactly what the input was. It works for any ﬁnite language. rejecting the input string. See Figure 2. up to length 4. 0011. The preceding example proves Theorem 10 Every ﬁnite language is accepted by a ﬁnite-state machine. Such a binary tree structure corresponds to a “table lookup” in a more conventional language. putting in the right transitions is straightforward. . 10. and FF — just draw a line surrounding them Machines and regard the enclosed area as a new single state —.2: A ﬁnite-state machine that accepts Lmatch Now that we know what each state means. Thus LEQ-4 = {01. STATE TRANSITION DIAGRAMS 0 0 1 0 1 33 1 1 0 1 0 Figure 2.2. 0110. Being in any one of these four states. Minimizing Finite-State if we “merged” the states H. no matter what the rest of the input. Why was this merger possible? Because from any of these four states.) Consider the language LEQ-4 of all nonempty bitstrings of length 4 or less which A third example contain the same number of 0s as 1s. 1010.4.2 shows the ﬁnished product. 0101. For example. This results in a ﬁnite-state machine with the structure of a binary tree of depth 4. 1001.1. only rejecting states can be reached. 1100}. Figure 2. we would have reduced the size of this machine by three states (without changing the language it accepts).3 and translating it into a ﬁnite-state machine by the same process that we used earlier with the program oddlength.4 is not as small as it could be. P. the outcome of the computation is always the same. Q. (Note that we could have arrived at the same machine by starting with the program match in Figure 2. The ﬁnite-state machine of Figure 2. viz.
  the string 011 is a reason why states B and C in Figure 2. (Note that s could be the empty string.4 cannot be merged. else cout << "No". Figure 2. Observation 3 If c is a character and s a string. the other accepting. next. which is the basis for an algorithm for ﬁnding all the states that can be merged. FINITE-STATE MACHINES first = last = next = getchar().) This condition can be rephrased to talk about when a merger is not feasible. One of E and Q is rejecting. and cs is a shortest reason why p and q cannot be merged. while (next != EOF) { last = next. } boolean match( void ) { char first. if there is an input string s such that of the two states δ (p. What if the subsequent input is 01? Out of A the string 01 drives the machine into state E — a fact usually written as δ (A. s) and δ (q. the two states δ (p. then s is a shortest reason why δ (p. though. s) one is rejecting and one is accepting. s) and δ (q.5 is an illustration for the following observation. which prevents a merger of an accepting with a rejecting state. Let’s call such a string s “a reason why p and q cannot be merged. s) are either both accepting or both rejecting. viz. c) . last. This is not a suﬃcient reason to allow a merger. match Can J and K be merged? No.3: Another “ﬁnite-state” program. CHAPTER 2. c) and δ (q. 01) = Q.” For example. This prevents us from merging A and D. next = getchar(). A reason why states D and E cannot be merged is the string 11. When is a merger of two states p and q feasible? When it is true that for all input strings s. The shortest reason for this is the empty string.34 void main( void ) { if (match()) cout << "Yes". The shortest reasons for this are the strings 0 and 1. because a subsequent input 1 leads to an accepting state from J but a rejecting state from K . 01) = E — and out of D the string 01 drives the machine into state δ (D. } return ((first == last) && (first != EOF)): } Figure 2. What about A and D? They both have only rejecting states as their immediate successors.
  FF 0.1 0.4: A ﬁnite-state machine for LEq-4 .2.1 0..1.1 ..1 0. STATE TRANSITION DIAGRAMS 35 A 0 1 B 0 1 0 C 1 D 0 1 0 E 1 0 F 1 0 G 1 H 0 1 0 I 1 0 J 1 0 K 1 0 L 1 0 M 1 0 N 1 0 O 1 P Q R S T U V W X Y Z AA BB CC DD EE 0.1 Figure 2.
  As we ﬁnd reasons for not merging states. for example “[0]. LINE1[W3]. 2 Note: This works because the binary relation MergablePairs is always an equivalence relation. which is consistent with the fact that the shortest strings that are reasons not to merge a and b are of length 3.9 the algorithm ends after one pass through the loop.2 Two states p and q are in the same cluster if (p. then p and q will remain together through the ﬁrst k − 1 passes through the repeat-loop and get separated in the k th pass. consider the ﬁnite-state machine in Figure 2. If p and q are two states that cannot be merged and the shortest reason for that is a string of length k > 0. as we ﬁnd the condition of the if-statement to be true. resulting in the machine shown in Figure 2.. .4. States a and b get separated in the third pass. The binary-tree-shaped machine of Figure 2.8 shows how the clusters of states develop during execution of the minimization algorithm. It accepts “identiﬁers with one subscript” such as A[2].11. TEMP[I].7. Figure 2. q ) ∈ MergablePairs. resulting in the machine shown in Figure 2. On the ﬁnite-state machine of Figure 2.e.5: Illustrating Observation 3 cannot be merged.6 is correct.” Other examples are the ﬁnite-state machines in Figures 2.9 and 2. X034[1001]. Observation 3 implies that the algorithm is Figure 2. it is convenient to keep track of clusters of states.36 CHAPTER 2. When carrying out this algorithms by hand.4 takes three passes. i.10. FINITE-STATE MACHINES p c c q S S Figure 2. we break up the clusters. For example.
  c).1.9 [ ] h Α. q ) ∈ MergablePairs for all input characters c if ( (δ (p.9 0.7: A ﬁnite-state machine for accepting identiﬁers with one subscript . c)) ∈ OldMergablePairs ) MergablePairs = MergablePairs − (p.9 g [ ] [ Α...Ζ 0..9 Α....Ζ [ d Α.Ζ 0.9 b ] c [ ] Α. δ (q.6: An algorithm for ﬁnding all mergable states in a ﬁnite-state machine e Α. q ) until MergablePairs = OldMergablePairs Figure 2.Ζ [ Α..9 ] a 0.9 ][ 0...Ζ 0.. STATE TRANSITION DIAGRAMS 37 MergablePairs = ( States × States ) − ( AcceptingStates × RejectingStates ) repeat OldMergablePairs = MergablePairs for all (p.9 [ ] Figure 2.Ζ 0.......Ζ 0.2.Ζ f ] Α.
  8: Minimizing the ﬁnite-state machine of Figure 2.38 CHAPTER 2.7 . FINITE-STATE MACHINES a b c d e f h g ] Pass 1 0 a 0 ] ] ] ] ] ] 0 b c 0 e 0 h A d A f g Pass 2 [ a Pass 3 A a Pass 4 a Pass 5 (no change) a e h b c d f g e h b c d f g A e A h b c d f g [ b [ e [ h c d f g Figure 2.
  STATE TRANSITION DIAGRAMS 39 0 0 1 1 1 0 0 1 1 0 0 1 Figure 2.9: A ﬁnite-state machine for checking parity 0 1 0 1 Figure 2.1.10: A minimal ﬁnite-state machine for checking parity .2.
  CC.M.Z.R.N U 1 S.W.F 1 0 G 1 K.L 1 U 0 E. FINITE-STATE MACHINES A 0 1 B 0 1 0 C 1 D 1 0 I.X.DD.1 H.BB 0.T.EE.11: A minimal ﬁnite-state machine for LEq-4 .40 CHAPTER 2.Q.V.U.O.FF 0.J.P.AA.1 Figure 2.Y.
  12: Equivalence classes of coins 2.” into disjoint subsets. But the diﬀerence between a dime and a quarter is of interest. Relations And given an equivalence relation.3 3 Note: This is easier to visualize when the set of nodes is ﬁnite. Given a partition.” The set of all dimes is another such class. as is the set of all pennies. there is a unique partition that goes with it.” Collectively. as illustrated in Figure 2. they form a “partition” of the set of all coins: every coin belongs to one and only one subset. one quarter looks as good as the next. THE MYHILL-NERODE THEOREM All coins 41 dollars dimes quarters pennies halfdollars nickels Figure 2.2.12. but works also for inﬁnite . Draw the graph of the relation. Partitions We start with a brief introduction of equivalence relations and partitions because these are central to the discussion of ﬁnite-state machines. when you are getting 25 cents back. or “partitioned. It consists of disconnected complete subgraphs whose node sets form the sets of the partition. Informally. The set of all coins gets carved up. There are ﬁnitely many such subsets in this example (6 to be precise).2 The Myhill-Nerode Theorem Equivalence Relations. The set of all quarters is what mathematicians call an “equivalence class. and so on. Each subset is called an “equivalence class. At the checkout counter of a supermarket.2. two objects are equivalent if distinguishing between them is of no interest. there is a unique equivalence relation that goes with it: the Partitions ≡ Equivalence relation of belonging to the same subset.
  one quarter is not always going to look as good as the next. are not going to be equivalent to us even if they are indistinguishable at run time. So there is a sequence of half a dozen or so successive reﬁnements of notions of and even uncountably inﬁnite sets.42 CHAPTER 2. we won’t care about further reﬁnements. programs with equally many pages of source code are equivalent. capturing the ﬁner distinctions that are of interest to a coin collector. A further reﬁnement could take running time into account: IOa = IOb and ta = tb (“t” for time). If all we want to do with the program is run it.12 gets more boundary lines added to it. FINITE-STATE MACHINES All coins 1891 "D" quarters in mint condition dollars dimes quarters pennies halfdollars nickels Figure 2. . If all we want to do is put a printout of their source under the short leg of a wobbly table. Our original equivalence class of all quarters gets subdivided. The picture of Figure 2. We might insist that the two programs compile to the same object code.) The reﬁnements need not stop here. (Both t and s are functions of the input. Then two programs. What else would we want to do with a program but run it? We might want to modify it. A further reﬁnement could in addition include memory usage: IOa = IOb and ta = tb and sa = sb (“s” for space). A very natural notion of equivalence between programs came up in Chapter 1: IOa = IOb . but one quarter that was minted in 1891 and has a D on it and is in mint condition looks as good as the next such quarter. To someone who is collecting rare coins.13: A reﬁnement of a partition Reﬁnements Just exactly what distinctions are of interest may vary. Equivalence Relations What diﬀerences between program do we normally ignore? It depends on what Among Programs we are interested in doing with the programs.13. See Figure 2. one with comments and one without.
  0000. Next we will apply the concept of partitions and equivalence relations to ﬁnite-state machines. (Alice still has not seen x. Alice asks a question Q about the string x. Bob gives a truthful answer AQ (x) and writes down a string y for Alice to see. if her question was “What is the last bit of x?” she might lose the game because an answer “1” and a subsequent choice of y = 1 would not allow her to know if xy ∈ Parity since x might have been 1 or 11. THE MYHILL-NERODE THEOREM 43 program equivalence. For example. Proof Whatever ﬁnite-answer question Q Alice asks. and concedes defeat. If Alice at this point has suﬃcient information to know whether or not xy ∈ Parity. .2. 100 ∈ Parity. Theorem 11 Alice can’t win the More-0s-Than-1s-game. the set of bitstrings that contain more 0s than 1s. . 000. What does this game have to do with ﬁnite-state machines? It provides a complete characterization of the languages that are accepted by ﬁnite-state machines . If not. each of which is the right one for some circumstances. 1.”. 00. She is sure to win. she loses. Alice cannot know whether or not x1i ∈ More-0s-Than-1s — she loses. 00. 3. Let Parity be the set consisting of all binary strings with an even number of A Game 1s. she wins.. 000. “What is x?” is out. . }. . . Can Alice be sure to win this game? For example. Alice realizes that x could be any one of 0.2. . and then write y = 1i . What would be a better question? Alice asks “Is the number of 1s in x even?”. answer Alice’s question. . .g. the answer will be the same for some diﬀerent strings among the inﬁnite collection {0. Let 0i 4 and 0j be two strings such that 0 ≤ i < j and AQ (0i ) = AQ (0j ). 2. Alice asks “How many more 0s than 1s does x have?”. Since 0i 1i ∈ More-0s-Than-1s and 0j 1i ∈ More-0s-Than-1s. but “What are the last three digits of x?” is allowed. Bob points out that the question is against the rules. Alice instead asks “Does x have more 0s than 1s?”. Bob writes down a string x without showing it to Alice. The question must be such that there are only ﬁnitely many diﬀerent possible answers. The Bob can write down x = 0i or x = 0j . 10100011 ∈ Parity. played by Bob and Alice.) 4. . 0000. Bob answers “Yes. E. Tired of losing Bob suggests they play the same game with a diﬀerent language: More-0s-Than-1s. Consider the following game. 4 Note: 0i means a string consisting of i 0s.
  Alice runs M to see whether or not it ends up in an accepting state.44 CHAPTER 2. 1. (xy ∈ L) ⇔ (x y ∈ L).14. L is accepted by some ﬁnite-state machine. FINITE-STATE MACHINES Theorem 12 Consider Bob and Alice playing their game with a language L. two strings x1 and x2 are equivalent.5 The following two statements are equivalent. In this sense the two strings x1 = 1 and x2 = 11 were “not equivalent” and Alice’s question made her lose because it failed to distinguish between two nonequivalent strings. if x has fewer than three digits. 2. Deﬁnition 1 Let L be any language. In fact. This partition “induced by ≡Q ” is illustrated in Figure 2. or.e. The Relation ≡Q Consider the question Q that made Alice win the Divisible-by-8-game: “What are the last three digits of x. There is a ﬁnite-state machine with k states that accepts L. if they draw the same answer. The following two statements are equivalent. . if for all strings y . what is x?”. The Relation ≡L Why was it that in the Parity game asking whether the last bit of x was a 1 did not ensure a win? Because there were two diﬀerent choices for x. 2. but for which there exists a string y with (x1 y ∈ Parity) ⇔ (x2 y ∈ Parity). the connection between the game and ﬁnite-state machines is even stronger. Alice draws a picture of a k -state machine for L and asks which state the machine is in after reading x. if for all strings y . x1 ≡parity x2 . Two strings x and x are equivalent with respect to L. Deﬁnition 2 Let Q be any question about strings. 1. To see that (2) implies (1) is the easier of the two parts of the proof.. AQ (x) = AQ (x ). The relation ≡Q “partitions” all strings into disjoint subset. x ≡Q x . x1 = 1 and x2 = 11. Alice can be sure to win. (x1 y ∈ Parity) ⇔ (x2 y ∈ Parity). When Bob writes down y . Two strings x and x are equivalent with respect to Q. i. Such nonequivalence of two strings is usually written as x1 ≡parity x2 . Conversely. which draw the same answer. Theorem 13 Consider Bob and Alice playing their game with the language L. x ≡L x . There is a question that lets Alice win and that can always be answered with one of k diﬀerent answers. The key to proving that (1) implies (2) is to understand why some questions that Alice might ask will make her win the game while others won’t.
  111.15 is not minimal. but 01 and 001 don’t.. The problem is that the two strings 0 and 00 draw the same answer. 0111. .. . what are the last three digits of x. THE MYHILL-NERODE THEOREM 45 11 1 011. Figure 2. say so. 0011. 1100. .. 1111. 0 100. 10 010. if x is nonempty but contains only 0s. . or. . Kind of messy.2... 00 000.. 1001. . Could it happen that this construction results in more than one transition with the same label out of one state? In this example. “If x is empty. 0110. As an example. 0101. if x has fewer than three digits. and this won’t happen. though.16 shows where this causes a problem in the construction of a ﬁnite-state machine. 0000.. 1101. 1011. 1000. The start state is the state containing the empty string.. In general it might. we got lucky — Alice asked a very nice question —. 110.14: The partition induced by Alice’s question The subsets of the partition induced by ≡Q then become the state of a ﬁnite-state machine. This is illustrated in Figure 2. The resolution is easy. Figure 2. otherwise. but let’s instead try to work on “mini5 Note: A “language” in this theory means a set of strings. .. what is x?”. 1010.. 0001. . 01 λ 101..15. say so. Why is this safe? Because ≡Q is a “reﬁnement” of ≡L and it is safe to ignore distinctions between strings that are equivalent under ≡L .. Generally. There is a transition labeled 1 from the state that contains 0 to the state that contains 01. We could The best question run our minimization algorithm on it.. It seems that the ﬁnite-state machine in Figure 2..2. 0100.. States containing strings in the language are accepting states. 1110. 001. Alice might have asked the question.. . there is a transition labeled c from the state containing s to the state containing sc. We arbitrarily choose one of the conﬂicting transitions and throw away the other(s). but better than the original question in the sense that it takes fewer diﬀerent answers. 0010.
  Figure 2. ... 1001. 1000. 0100. Figure 2. 1 001. 0001. 1110... 0010.. ... 0011.. FINITE-STATE MACHINES 1 1 0 11 0 1 1 1 01 011.... . . .. 0111. 1 1 001. 0000. . 0 1 1 λ 0 1 0 0 101. 1111.. 0001. 1 110. 0 0 0 00 0 1 0 010.. 1011. . 0101. 1101.. 1 10 0 1 111. 1 0 100. 1001. 1010. 1100.. 00. 000.16: A conﬂict in the construction of a ﬁnite-state machine ... . 0000..46 CHAPTER 2. . 0110.15: The ﬁnite-state machine constructed from Alice’s question 01 1 0. . 0 0 0 000..
  2.2. THE MYHILL-NERODE THEOREM
  47
  mizing” the question Q from which the machine was constructed. Minimizing a question for us means minimizing the number of diﬀerent answers. The reason why this construction worked was that ≡Q was a reﬁnement of ≡L . This feature of the question Q we need to keep. But what is the reﬁnement of ≡L that has the fewest subsets? ≡L itself. So the best question is “Which subset of the partition that is induced by ≡L does x belong to?”. This is easier to phrase in terms not of the subsets but in terms of “representative” strings from the subsets: “What’s a shortest string y with x ≡L y ?”. Let’s apply this idea to the Divisible-by-8 problem. There are four equivalence classes: 0, 00, 000, 0000, 1000, . . . , λ, 100, 0100, 1100, . . . , 10, 010, 110, 0010, 0110, 1010, 1110, . . . , and 1, 01, 11, 001, 011, 101, 111, 0001, 0011, 0101, 0111, 1001, 1010, 1101, 1111, . . . . The four answers to our best question are λ, 0, 10, and 1. The resulting ﬁnitestate machine has four states and is minimal. Let’s take stock of the equivalence relations (or partitions) that play a role in ﬁnite-state machines and the languages accepted by such machines. As mentioned already, every language, i.e., set of strings, L deﬁnes (“induces”) The Relations ≡L , ≡Q , an equivalence relation ≡L between strings: ≡M (x ≡L y ) ⇐⇒ (for all strings z, (xz ∈ L) ⇔ (yz ∈ L)).
  def
  (2.2)
  Every question Q about strings deﬁnes an equivalence relation ≡Q between strings: (x ≡Q y ) ⇐⇒ (AQ (x) = AQ (y )).
  def
  (2.3)
  And every ﬁnite-state machine M deﬁnes an equivalence relation ≡M between strings: (x ≡M y ) ⇐⇒ (δM (s0 , x) = δM (s0 , y )).
  def
  (2.4)
  s0 denotes the start state of M . Note that the number of states of M is the number of subsets into which ≡M partitions the set of all strings. As mentioned earlier, for Alice to win the game, her question Q must distinguish between strings x and y for which there exists a string z with (xz ∈ L) ⇔ (yz ∈ L). In other words, she must make sure that (x ≡L y ) ⇒ (x ≡Q y ) (2.5)
  i.e., ≡Q must be a reﬁnement of ≡L . If we regard binary relations as sets of pairs, then (2.5) can be written as ((x, y ) ∈ ≡L ) ⇒ ((x, y ) ∈ ≡Q ) or, for the notationally daring, (2.6)
  48
  CHAPTER 2. FINITE-STATE MACHINES
  1 1, 01, 11, 101, ... 1 0 1 10, 110, 1010, ... 1 0, 00, 000, ... 0 λ , 0100, 1100, ... 0
  0
  Figure 2.17: Illustrating the uniqueness of a minimal machine
  ≡ Q ⊆ ≡L
  (2.7)
  Instead of talking about Alice and her question, we can talk about a ﬁnitestate machine and its states.6 For the machine M to accept all strings in L and reject all strings that are not in L — a state of aﬀairs usually written as L(M ) = L —, M must “distinguish” between strings x and y for which there exists a strings z with (xz ∈ L) ⇔ (yz ∈ L). In other words, (x ≡L y ) ⇒ (x ≡M y ) (2.8)
  i.e., ≡M must be a reﬁnement of ≡L . Again, if we regard binary relations as sets of pairs, then (2.8) can be written as ≡M ⊆ ≡L (2.9)
  Every machine M with L(M ) = L must satisfy (2.9); a smallest machine M0 will satisfy ≡M0 = ≡L (2.10)
  Uniqueness How much freedom do we have in constructing a smallest machine for L, given
  6 The analogy becomes clear if you think of a ﬁnite-state machine as asking at every step the question “What state does the input read so far put me in?”.
  2.2. THE MYHILL-NERODE THEOREM
  49
  that we have to observe (2.10)? None of any substance. The number of states is given, of course. Sure, we can choose arbitrary names for the states. But (2.10) says that the states have to partition the inputs in exactly one way. Each state must correspond to one set from that partition. Figure 2.17 illustrates this for the language Divisible-by-8. This then determines all the transitions, it determines which states are accepting, which are rejecting, and which is the start state. Thus, other than choosing diﬀerent names for the states and laying out the picture diﬀerently on the page, there is no choice. This is expressed in the following theorem. Theorem 14 All minimal ﬁnite-state machines for the same language L are isomorphic. Unlike our earlier construction of a ﬁnite-state machine from a (non-minimal) winning question of Alice’s, this construction will never result in two transitions with the same label from the same state. If that were to happen, it would mean that there are two strings x and y and a character c with x ≡L y and xc ≡L yc. Since xc ≡L yc there is a string s with (xcs ∈ L) ⇔ (ycs ∈ L) and therefore a string z (= cs) with (xz ∈ L) ⇔ (yz ∈ L), contradicting x ≡L y . Figure 2.17 could have been obtained from Figure 2.14 by erasing those lines that are induced by ≡Q but not by ≡L . This is what our minimization algorithm would have done when run on the machine of Figure 2.15. Summarizing, we have the following theorem. Theorem 15 (Myhill-Nerode) Let L be a language. Then the following two statements are equivalent. 1. ≡L partitions all strings into ﬁnitely many classes. 2. L is accepted by a ﬁnite-state machine. Furthermore, every minimal ﬁnite-state machine for L is isomorphic to the machine constructed as follows. The states are the subsets of the partition induced by ≡L . The start state is the set that contains λ. A state A is accepting if A ⊆ L. There is a transition from state A to state B labeled c if and only if there is a string x with x ∈ A and xc ∈ B .
  Regular Expressions (Kleene’s Theorem)
  The above Myhill-Nerode Theorem provides a complete characterization of properties that can be determined with ﬁnite-state machines. A totally different characterization is based on “regular expressions,” which you may know as search patterns in text editors. Regular languages and expressions are deﬁned inductively as follows. Deﬁnition 3 (Regular languages) 1. The empty set ∅ is regular.
  2. Any set containing one character is regular.
  q . L is accepted by some ﬁnite-state machine. with the induction being over this numbering of the states. 1. ﬁrst number the states of the machine 1. and A . Observation 5 A language L is regular if and only if there is a regular expression E with L = L(E ). Theorem 16 (Kleene’s Theorem) The following two statements are equivalent. concatenation and Kleene star operations are deﬁned as usual: A ∪ B = {x : x ∈ A or x ∈ B }. This is a nice example of the fact that induction. Any single character c is a regular expression and L(c) = {c}. [The union. Here are the details. (2) ⇒ (1) To construct a regular expression from a given ﬁnite-state machine M .50 CHAPTER 2. and A = {x1 x2 · · · xk : k ≥ 0 and xi ∈ A for all i} . . can be thought of as extending to other sets as well. . 2. FINITE-STATE MACHINES 3. and L(α ) = (L(α)) . 3. and α . The symbol ∅ is a regular expres- 2. This is a nontrivial example of a proof by induction and a nice illustration of the fact that inductive proofs often become easier once you decide to prove a stronger result than you really wanted. L is regular. Deﬁnition 4 (Regular expressions) sion and L(∅) = ∅. αβ . 1. AB .] Observation 4 Finite languages are regular. To construct ﬁnite-state machines from regular expressions we do an “induction over the structure” of the expression. with state 1 being the start . L(αβ ) = L(α)L(β ). To construct regular expressions from ﬁnite-state machines we number the states and carry out an inductive construction. . Proof Both parts of this proof are inductive constructions. If A and B are regular then so are A ∪ B . If α and β are regular expressions then so are α ∪ β . while strictly speaking a technique that is tied to natural numbers. . AB = {xy : x ∈ A and y ∈ B }. and L(α ∪ β ) = L(α) ∪ L(β ). The important result in this section is the following.
  THE MYHILL-NERODE THEOREM 51 3 a b a b 1 2 a b Figure 2. For example. 3 In this example. k−1 k −1 k −1 k −1 Lk (2. This we show by induction on k . This equation amounts to saying that there can be two ways to get from state i to state j without going through states numbered higher than k . In this sequence 1 → 2 → 3 we say that the machine “went through” state 2. We simplify the subexpressions as we go. if the machine of Figure 2. which are only the starting point and the endpoint in the sequence. What’s left to show is that these languages Lk ij are always regular.19 illustrates this. corresponding to the subexpression before the union and the subexpression after the union. It contains those characters that label a transition ij from state i to state j . L0 is ﬁnite. hence L(λ) = {λ}.” Some parentheses are omitted.11) where the union is taken over all accepting states of M .12) ij = (Lik (Lkk ) Lkj ) ∪ Lij . Deﬁne Lk ij to be the set of all strings which make the machine go from state i into state j without ever “passing through” any state numbered higher than k . Figure 2. Either we go through k . For k > 0. Other examples are babb ∈ L32 and a ∈ L23 . λ is short for ∅ . or we don’t. If i = j the empty string gets added to the set.20 shows some of the steps in the construction of a regular expression from the ﬁnite-state machine of Figure 2. the language accepted by a ﬁnite-state machine M is L(M ) = f Lq 1f (2. It did not “go through” states 1 or 3.18.18 happens to be in state 1 then the string ba will make it ﬁrst go to state 2 and then to state 3. (We wouldn’t want to miss a chance to be obscure.18: Illustrating the notion of a string making a machine “pass through” states state. by the deﬁnition of the -operation. For k = 0. Since it did not go through any state numbered higher than 2 we 0 0 2 have ba ∈ L2 13 .2. k k k Expression Eij describes language Lk ij : “L(Eij ) = Lij .2. would we?) . Figure 2. In general. in fact {a} = L23 . the language accepted by the machine is L13 .
  21.” This means that ﬁnite-state machines are ﬁrst constructed for small subexpressions. This seems easy enough. FINITE-STATE MACHINES L k L k-1 ik k-1 kk L k-1 kj i L k-1 ij j Figure 2. For example.. presented with such an ambiguity? Can we deﬁne the “semantics. For simplicity let’s adopt the convention that if a transition goes to a rejecting state from which the machine cannot escape anymore then we won’t draw the transition. can we deﬁne how such a machine is to run. but what if the expression had been just slightly diﬀerent. see Figure 2. see Figure 2. shown in Figure 2.52 CHAPTER 2.19: The pattern of induction for constructing the regular expressions Lk ij in Kleene’s Theorem (1) ⇒ (2) To construct ﬁnite-state machines from regular expressions we use what is called “induction on the structure of the expression” or “structural induction.” i. with an extra b in front: E = (babb) ∪ (b(aa) bab) Then our construction would have gotten us this machine ME in Figure 2.24. nor that “dead” state. Adding parts that correspond to subexpressions abb and bab completes the construction. This machine ME has a ﬂaw: there are two transitions labeled b out of the start state.e. We can combine Mb and M(aa) to get Mb(aa) . in such a way that the machine ME accepts the very same strings that the expression E describes? . see Figure 2.23. if we want to convert the expression E = (abb) ∪ (b(aa) bab) a machine M(aa) for the subexpression (aa) is shown in ﬁgure 2.25. When we run such a machine what will it do? What should it do. then combined to form machines for larger subexpressions and ultimately for the whole expression.22. A machine Mb is easier yet.26. This simpliﬁes the picture for M(aa) .
  2.2.18 into a regular expression . THE MYHILL-NERODE THEOREM 53 0 E11 0 E21 0 E31 = = = λ∪a ∅ b 0 E12 0 E22 0 E32 = b = λ∪b = a 0 E13 0 E23 0 E33 = = = ∅ a λ 1 0 0 0 0 E32 = (E31 (E11 ) E12 ) ∪ E32 = (b(λ ∪ a) b) ∪ a = ba b ∪ a 1 E11 1 E21 1 E31 = = = a ∅ ba 1 E12 1 E22 1 E32 = a b = λ∪b = ba b ∪ a 1 E13 1 E23 1 E33 = = = ∅ a λ 1 1 1 1 2 = (a b(b ) a) ∪ ∅ = a bb a ) ∪ E13 ) E23 (E22 = (E12 E13 2 E11 2 E21 2 E31 = = = a ∅ ba 2 E12 2 E22 2 E32 = a b+ = b = ba b+ 2 E13 2 E23 2 E33 = a bb a = b a = (ba b ∪ a)b a 2 2 2 3 2 ) E31 ) ∪ E11 = (a bb a((ba b ∪ a)b a) ba ) ∪ a (E33 E11 = (E13 Figure 2.20: Converting the ﬁnite-state machine from Figure 2.
  23: The machine Mb a b a Figure 2.22: The machine M(aa) . FINITE-STATE MACHINES a a b b a. b Figure 2.21: The machine M(aa) a a Figure 2.24: The machine Mb(aa) .54 CHAPTER 2. a simpliﬁed drawing b Figure 2.
  25: The machine ME b a 7 b 8 b 9 10 1 a b 2 a b 4 a 5 b 6 3 Figure 2. THE MYHILL-NERODE THEOREM 55 b b a a b a b a b Figure 2.2.2.26: The machine ME .
  Our construction will result in the machine ME of Figure 2.27. which states would it be in after reading an a as the second character? In states 3 and 8. it should be accepted by ME . FINITE-STATE MACHINES a b 7 b 8 9 1 a b 2 a b 4 a 5 b 6 3 Figure 2. Our conventions about running these machines and knowing when a string is accepted are the “semantics of nondeterministic ﬁnite-state machines. Since bba is a string described by E . which is accepting. With this way of running the machine. It is more elegant — although ultimately of no important consequence — to say that the machine is only in state 9. 7 After reading another b — for a total input string of babb — ME is in state 10 and accepts the input. . it can only be in state 9. say 11. the machine is in state 5.27: The machine ME After reading an initial b. What if some such machine ends up in a set of states some of which are accepting and some are rejecting? The expression E = (bba) ∪ (b(aa) bab) provides an example. If we take the view that the state is there — we just don’t draw it — then at this point we should say that the machine would be in state 9 and that invisible state. After reading a b as the third character. After reading bba. a rejecting state from which the machine cannot escape..” These 7 This could be up to some discussion depending on how we view our convention of not drawing a “dead” state. Being in state 7 after the ﬁrst b give us a chance to accept babb. and in state 9.56 CHAPTER 2.e. and being in state 2 gives us a chance to accept baabab. i. There is one more issue that this example does not address. should ME be in state 2 or in state 7? We want to accept both babb and baabab. This suggests that the right way to run this machine is to have it be in both states 2 and 7 after reading the ﬁrst b. That settles the issue: a string is accepted if at least one of the states it puts the machine in is an accepting state. This is clumsy. which is rejecting.
  converting the machine ME into a deterministic one. . An example that uses all the parts of this algorithm is the machine shown in Figure 2. The states of the deterministic machine are the sets of states of ME .2.e. δ (A. THE MYHILL-NERODE THEOREM b 3.8 a b a 9 b 10 57 a 2 3 a 1 2. where the concept of nondeterminism will play a crucial role in deﬁning a class of computational problems). Tracing the execution results in a tree and a program is said to accept its input if at least one of the paths leads to acceptance.28: Converting ME into a deterministic machine conventions can be applied to more powerful programs as well (and we will do so in a later chapter on computational complexity. (Many of these sets of states may not be reachable from the start state and can be discarded.” The inductive algorithm for constructing a (nondeterministic) ﬁnite-state machine ME from an arbitrary regular expression E then works as shown in Figure 2. “Nondeterministic C. ) The way to think about the execution of such nondeterministic programs is that there is more than one path of execution.) In the deterministic machine.2. is the set of those states b of ME to which there is a c-transition (in ME ) from some state a ∈ A. the state reached from a state A via a transition c.30. i.” which we do in a breadth-ﬁrst approach. Can we convert this new kind of machine into a standard (deterministic) ﬁnite-state machine? This task amounts to a search of the “execution tree.28 gives an example. Figure 2.29.. The standard term is a “λ-transition. x = 0. a transition that does not consume an input character.7 b 4 b a 5 b 6 Figure 2.” for example. The algorithm for constructing a nondeterministic machine from a regular expression is easier to describe if we allow yet another form of nondeterministic behavior: making a “spontaneous” transition. c). could be C with the extra feature of a nondeterministic branch: ( nondet-branch. x = 1.
  For any single character c. Mα λ λ Mβ Constructing Mα∪β from Mα and Mβ λ Mα λ M β Constructing Mαβ from Mα and Mβ λ λ Mα λ Constructing Mα from Mα Figure 2. If α and β are regular expressions then Mα∪β . M∅ is the following machine: 2. Mc is the following machine: c 3.58 CHAPTER 2. Mαβ . and Mα are constructed from Mα and Mβ as shown below. FINITE-STATE MACHINES 1.29: An algorithm for converting regular expressions to ﬁnite-state machines .
  Given that ﬁnite-state machines and regular expressions are equivalent concepts. Is the set of binary numbers that are divisible by 2 regular? . Here are some examples.31. all of them variations of the question of which languages are regular: Question 1. Some Regular Expressions? questions are easier to deal with in terms of ﬁnite-state machines. Finite-State Machines or we can choose to approach some problem in terms of one or the other.2.31: A deterministic version of M(aa) ∪(a(bb) ) Nondeterministic machine with λ-transitions can be turned into deterministic ones with the same set-of-states approach used earlier. For example. others in terms of regular expressions. THE MYHILL-NERODE THEOREM 59 λ λ λ a λ a λ a λ λ b λ λ b Figure 2. the machine in Figure 2.30 turns into the machine in Figure 2.2. This concludes the proof of Kleene’s Theorem.30: The machine M(aa) ∪(a(bb) ) a a a a b b b Figure 2.
  110.. Question 3. what is the common pattern of 11. Is the set of binary numbers that are divisible by 3 regular? The answer is yes. yield a remainder of 0. In terms of ﬁnite-state machines. this requires an argument involving nondeterministic machine (as used in the proof of Kleene’s Theorem). The Myhill-Nerode Theorem tells us that we would have to ﬁgure out exactly what it is that we want to keep track of as the input gets processed. The complete machine is shown in Figure 2. Question 4. After all. It follows from the deﬁnition of regular sets and expressions. 1. The four equivalence classes of ≡L (and hence the states of the minimal machine for L) are {λ} and the three sets of binary numbers which.33. when divided by 3. when divided by three. There is no equally simple argument in terms of regular expressions. Question 2. and vice versa. Are regular languages closed under complement? I.60 0 CHAPTER 2.e.) What if a 1 had been read instead? This would have amounted to multiplying the previous number by 2 and adding 1. if A is regular. would yield a remainder of 1. with the right approach anyway. is their union A ∪ B regular as well? The answer is yes. if A and B are regular. . If we know that the input so far was a number that was divisible by 3. (This is so because reading that 0 amounted to multiplying the previous number by 2. is A = {x : x ∈ / A} regular as well? This is most easily dealt with in terms of ﬁnite-state machines. The resulting number. If M is a ﬁnite-state machine for A then a ﬁnite-state machine for A can be obtained by turning accepting states into rejecting ones. 10000000010.. These remainders are used as the labels of the three states. and 2. the number is again divisible by 3.32: A piece of a ﬁnite-state machine This is easy in either domain. 1001? Yet it is not all that hard to build a ﬁnite-state machine for this language. The right regular expression is (0 ∪ 1) 0. Are regular languages closed under union? I.32.e. then we know that after reading one more 0. This translates into a piece of a ﬁnite-state machine shown in Figure 2. but it seems nearly impossible to build a suitable regular expression. FINITE-STATE MACHINES 0 1 Figure 2.
  e. x. x.2. cannot be done with an amount of memory that does not depend on the input that is being parsed. it takes a “stack” to parse. is their diﬀerence A − B = {x : x ∈ A and x ∈ / B } regular as well? There is a construction based on ﬁnite-state machine that proves this closure result and is interesting in its own right. Is the set of syntactically correct C programs regular? It is not. THE MYHILL-NERODE THEOREM 0 61 0 0 λ 1 1 1 0 2 1 1 0 Figure 2. x x x x x = = = = = ( (( ((( (((( ((((( This shows that parsing programs.. if A and B are regular... x.2. Question 6. x. and the only proof method presented so far is to argue that there are inﬁnitely many equivalence classes induced by ≡L . . () () () () () { { { { { int int int int int x. As we will see in the next chapter.33: A ﬁnite-state machine for binary numbers divisible by 3 Question 5. Here is an inﬁnite list of nonequivalent strings: main main main main main . Are regular languages closed under set diﬀerence? I. which is a necessary part of compiling them.
  w = uvx. The challenge is to cover all ways in which the strings could get divided up into three pieces. Assuming it was. implying that it is possible to split w into three parts. Let’s use this Lemma to prove that the language L = {an bn : n ≥ 1} is not regular. uv i x ∈ L. There are three strings. x with w = uvx and |v | > 0 such that for all i ≥ 0. Let k be the constant said to exist in the Lemma. Pumping is important if for no other reason then because unlike the analysis of ≡L in Myhill-Nerode. No matter what three-way split of w one tries. v .) This contradiction shows that L is not regular. Consider the string w = ak bk . for all i ≥ 0. although no more powerful than the analysis of ≡L that we have discussed already in Sections ??. Lemma 3 Pumping Lemma for Regular Languages. x. hence statements (1) — (3) from the Lemma apply. v . and 2. It is interesting in its own right. pumping arguments also work for the more powerful programs we will consider in the next chapter. uv i x ∈ L. (One can come up with a more detailed argument for this. Here is an attempt: The substring v cannot contain more than one type of character because otherwise v 2 contains characters in an order not permitted in L. An added challenge is to come up with a concise argument. w = uvx. u. |v | > 0. such that 1. and 3. 2k ). The length of w is k or more (in fact. Then the Lemma applies. which therefore does not contain equal numbers of both characters anymore. But this is not true. If L is a regular language then there is a constant k such that for every string w in L of length k or more the following is true. uv 2 x is never of the form of equal numbers of a’s and b’s. such that |v | > 0 and uv 2 x ∈ L. Why is this so messy? The Pumping Lemma can be stated as follows. FINITE-STATE MACHINES 2. . L regular =⇒ There is a constant k such that for all strings w ∈ L with |w| ≥ k there are strings u.62 CHAPTER 2. Hence at most one type of character take part in the “pumping” from uvx to uv 2 x.3 The Pumping Lemma for Regular Languages “Pumping” is a second technique to prove nonregularity.
  This is what we did in the previous example. we can prove that some language L is not regular by proving the above four-line statement ¬B : “For all .) What’s the next step? We have to show that for all strings u. Here is such an exhaustive set of cases for this example.” where B is that complicated statement with four alternating quantiﬁers. To show this. all we need to do is exhibit one such string. This is the hard part of the proof because we somehow have to provide an argument that covers all possible such three-way breakups of w. x with w = uvx and |v | > 0 there is an i ≥ 0 with uv i x ∈ L.” Spelled out in detail. THE PUMPING LEMMA FOR REGULAR LANGUAGES 63 The above statement is of the form “L regular ⇒ B . Thus. there is a constant i with uv i x ∈ L. How do we prove something “for all constants k ”? By starting out the proof by saying “Let k be an arbitrary constant” and carrying the k along as a parameter of the argument. Then uv 0 x ∈ L if for no other reason then because it does not have an = in it. .2. Here is another one. This language L is not regular. v .3. a more useful version of the statement is “L not regular ⇐ ¬B . v contains the =. It can be done. let’s pick w to be the string 1···1+2···2 = 3···3 k k k Another Example (Note how the k plays its role. x which represent a three-way breakup of w (“w = uvx” ) and which don’t have the second piece empty ( “|v | > 0” ). a fact we prove by proving the statement “¬B ” above. v . . by dealing with an exhaustive set of cases. Since we are trying to prove that some languages are not regular. What’s next? We need to show that there exists a string w with certain properties. In this. Case 1. ∈ L”. . this version reads L not regular ⇐= For all constants k there is a string w ∈ L with |w| ≥ k such that for all strings u.
  Let L be the language of all bitstrings that contain a run of three Theorem consecutive 0s. Is it true that 01 ≡L 001? Explain. 2.7.1. Ex. Same for the language of bitstrings whose ﬁrst two bits are equal to its last two bits.3. Let L be the language of all bitstrings x such that x contains at least one 0 and at least one 1. 1.64 CHAPTER 2. . 00 ∈ L. Let L be the language of binary strings whose third bit from the end is a 1. then 0001 ≡L 0011. Let L be the language of all bitstrings x such that x contains at least one 0.) Minimizing ﬁnite-state Ex. 01 ∈ L. then 01 ≡L 00.) How many equivalence classes does ≡L induce? Ex. Draw the smallest ﬁnite-state machine M for the language of all bitstrings that contain the substring 011. 2. 2. If L is the language of all bitstrings of length 8. (Show the development of the “clusters” of states just like Figure 2. thus creating a string outside of L. 1111 ∈ L. The Myhill-Nerode Ex.) How many equivalence classes does ≡L induce? 4. Then v lies entirely on one side of the = and pumping (with any i = 1) changes that side of the equation but not the other.) Ex. 2.4. (For example. 2. (For example. 1111 ∈ L. Carry out the minimization algorithm of Figure 2. Draw a ﬁnite-state machine on which the minimization algorithm runs through more than 10 passes. If L is the language of all bitstrings that start with a 1 and end with a 0.6. 2. Draw the smallest ﬁnite-state machine M for the language of all bitstrings that contain the substring 011. v does not contain the =. Give one string from each of the equivalence classes induced by ≡L . 3.5. Exercises Ex.6 for the ﬁnitemachines state machine shown below. 2.8. (Make sure you provide a 0 and a 1 transition out of every state. Ex.2. 01 ∈ L. 2. FINITE-STATE MACHINES Case 2. 2.8 did. Ex.
  one can always choose the accepting states of the cross-product machine such that it accepts A ∩ B. Strings containing all their opening parentheses before all their closing ones.12.g. Are the two relations ≡L and ≡L ¯ (that’s L L) always the same. 2. $}: L = {x$y : x. Strings containing more opening parentheses than closing ones. b. Do not give explanations.13. Consider the following language L over the alphabet Σ = {a. 2. Which of the following languages is regular? (Just circle “Yes” or “No” for each language. As discussed in class.11. e. Ex. ¯ .14.10. Can one always choose the accepting states of the cross-product machine such that it accepts . e. Strings containing the word while 3. or show an example were they aren’t? Ex. give a counterexample.g. “abcabaeeee” Ex.) Ex. “())(((” 5. 2. Strings of even length whose ﬁrst half and second half both start with the same letter.3. 2. give one shortest string from the class. y ∈ {a. e. Ex. 2. for any given L? Either argue that they are. or show an example were they don’t? (xrev is x written backwards. “(a)b)c(d(e(f” 4. “((x(y-/)” 6. 2. Give one representative of each equivalence class of ≡L for L = {x : x contains the substring 000}. . the complement of Ex. (b) Do the partitions that go with ≡L and with ≡Lrev (that’s Lrev = {xrev : x ∈ L}) always have the same equivalence classes? Either argue that they always do.g. Is it true that x ≡L y implies xz ≡L yz ? Is it true that xz ≡L yz implies x ≡L y ? When your answer is no. Strings containing two opening parentheses with no closing parenthesis in between the two. THE PUMPING LEMMA FOR REGULAR LANGUAGES 65 Ex.) 1. 2. Consider two ﬁnite-state machines that accept languages A and B and consider their cross-product machine. The language described by (ab(a ) (b ∪ (aab) )) 2.2. .g.9. e. b}+ and x does not start with the same letter as y } For every equivalence class of ≡L .15. .
  2. (a) Give an example where the cross-product machine is minimal. (b) If the cross-product machine is minimal. For any k ≥ 0 let Lk be the language of bitstrings whose ﬁrst k bits equal their last k bits. Nondeterministic Ex. A ? Regular Expressions Ex. A ∪ B ? CHAPTER 2.22. Is it decidable whether or not a program p has a regular language Lp ? Brieﬂy justify your answer. 2. 2. FINITE-STATE MACHINES 2. does that imply that the two “factors” in the product had to be minimal? Give an argument or a counterexample. Ex. Strings in A ∪ B of length ﬁve or more? 5. Let Fsm-Equivalence be the problem of deciding. given two ﬁnitestate machines P and Q. Does the following approach work: Make all rejecting states of the nondeterministic machine into accepting states and all accepting states into rejecting states? Brieﬂy explain your answer. 2.16. 2. For any program p deﬁne Lp to be the set of input strings on which the program prints the word “Yes” and terminates. Give three properties that are decidable for ﬁnite-state machines but undecidable for arbitrary programs. 2. AB ? (That’s A concatenated with B : {xy : x ∈ A and y ∈ B }.66 1.23. 2. How many states does a deterministic ﬁnite-state machine for Lk need? Same for a nondeterministic machine. all the strings that are not in A). Consider the problem of taking an arbitrary nondeterministic Machines ﬁnite-state machine that accepts some language A and modifying the machine to accept the complement of A instead (i.) 4.) 3.. How long is a regular expression for Lk ? Ex. Ex.18. Which of the languages in Figure ?? are regular? Cross-Product Ex. Ex.e.21. 2. whether or not L(P ) = L(Q). Miscellaneous Ex.19.17. Let Fsm-Emptiness . Assume we are dealing with bitstrings only.20. Derive an asymptotic bound on the length of the expression constructed from a ﬁnite-state machine in the proof of Kleene’s Theorem. Consider creating the cross-product machine of two given maConstruction chines and choosing the accepting states such that the new machine accepts the intersection of the languages of the two old machines. A − B ? (That’s {x : x ∈ A and x ∈ / B }.
  then minimization.27. 8 To do: This is not in the sample solutions yet. 2. Prove that every inﬁnite regular language contains a string whose length is not a prime number.2.3.· · · is not regular. Construct a ﬁnite-state machine for the language {x : x ∈ {0. Use this stronger version to prove two languages from 8 Figure ?? not regular which cannot be proven not regular with the original weaker version. Ex. . Ex. 2. 001} }. whether or not L(M ) = ∅. 2. Ex.25.26. Ex. 1} and x ∈ / {000. one that says not only that pumping occurs somewhere in a long enough string but that it occurs within every long enough substring. then apply the subset construction. 2. Describe how a cross-product construction can be used to transform Fsm-Equivalence into Fsm-Emptiness.24. Ex. First construct a nondeterministic machine that is as small as possible. 2. Formulate a stronger version of the pumping lemma. For two of the nonregular languages in Figure ?? use the pumping lemma to prove that they are not regular. THE PUMPING LEMMA FOR REGULAR LANGUAGES 67 be the problem of deciding.28. Prove that the set of strings that are ﬁnite preﬁxes of this inﬁnite string -0-1-10-11-100-101-110-111-1000-1001-1010-1011. given a ﬁnite-state machine M .
  68 CHAPTER 2. FINITE-STATE MACHINES .
  one feature of programming languages that makes them nonregular is the arbitrary nesting of parenthesis in expressions. Regular expressions cannot be used to deﬁne what syntactically correct expressions look like. The best way to keep track is a tree: 69 .Chapter 3 Context-Free Languages 3. An example of a context-free grammar is this: E E E E E → → → → → E + E E × E (E) x y The way this grammar — call it G1 — is used is to start with the symbol on left of the arrow in the ﬁrst line. This is the “start symbol” — the string of length 1 we start out with. We then grow that string by repeatedly replacing what’s on the left-hand side of an arrow with what’s on the right-hand side.1 Context-Free Grammars As we saw at the end of the previous chapter. E . “Context-free grammars” can.
  only the ﬁrst one matches our conventions about the precedence of arithmetic operations. CONTEXT-FREE LANGUAGES E E E E E E E x + y X ( x + y ) There are choices of which replacements to make.” The problem is. into machine language. and in interpreters to know how to evaluate an expression. Derivation trees are also called “parse trees.70 E CHAPTER 3. The ﬁrst one is “right. This is because it allows a second derivation tree for the same string: E E E E E E E E x + y X ( x + y ) Such a grammar is said to be “ambiguous. or a whole program. while it does perfectly well describe the language in question. The following . The choices made above resulted in a tree whose leaves. The tree is a “derivation tree” for the string. read from left to right. or interpret a whole program. does not help us deﬁne the one and only “correct” way to parse each string in the language. though. the ﬁrst tree leads to a value of 36 for the expression. read x + y × (x + y ).” They are used in compilers to know how to translate a given expression. Of the two trees.” the second one is “wrong. The grammar G1 is said to “derive” that string: x + y × (x + y ) ∈ L(G1 ). whereas the second one leads to a value of 31. this grammar G1 . If we plug in x = 1 and y = 5.” The two parse trees for the string x + y × (x + y ) lead to two diﬀerent evaluations of the expression. Our sample grammar does not serve that purpose well.
  parentheses.2 Parsing Recursive-Descent Parsing A grammar like G2 above is easily turned into a parser. and the variables x and y . The program is nondeterministic. though — certainly a ﬂaw we do not want to tolerate for very long. See Figure 3. the “right” one. a program that determines whether or not a given input string is in the language and does so (necessarily) by building a parse tree. implicitly or explicitly.1.2. i.3. PARSING grammar G2 does do that: E E T T F F F → E + T → T → T × F → F → (E) → x → y 71 This grammar only allows one derivation tree. In what sense does this program “parse” expressions? . The tree for the expression x + y × (x + y ) is: E T F E E E T T T T F F F F x + y X ( x + y ) 3.e. ×. for any arithmetic expressions involving +..
  } } F ( ) { nondet-branch: {terminal(’(’). E().72 CHAPTER 3. T().} {F().} {terminal(’y’).} {T(). else REJECT. terminal(’*’).} } terminal( char c ) { if (getchar() != c) REJECT. } Figure 3.} {terminal(’x’).} } T ( ) { nondet-branch: {T(). } E ( ) { nondet-branch: {E().1: A nondeterministic recursive-descent parser . F(). CONTEXT-FREE LANGUAGES main ( ) { E(). terminal(’)’). if (getchar() == EOF) ACCEPT. terminal(’+’).
  ) Deterministic Parsing Intrinsically Nondeterministic Context-Free Languages With regular languages and ﬁnite-state machines it was always possible to remove the nondeterminism from the machines. here is an example of a grammar that derives an intrinsically nondeterministic language. Without giving any proof of this. or even just two stacks. PARSING 73 Being a nondeterministic program. providing a complete characterization of context-free languages in terms of a type of program. S B → BSB | 1 → 0|1 To parse this language with a pushdown machine. Thus the above example generalizes to the following theorem. This approach does not work for pushdown machines because a set of stacks. Consider again the string x + y × (x + y ). but the intuition is that the parser must keep building up a stack of recursive calls for the ﬁrst half of the input string and climb out of the recursion during the second half. Such programs are often called “pushdown machines” or “pushdown automata (pda’s). The one path of execution that results in acceptance is the one where the ﬁrst execution of E in turn calls E. all bitstrings of odd length whose middle character is a 1. The accepting computation looks like a preorder traversal of the parse tree of the input string. The converse is also true. it is not possible to know when the second half starts. the program as a whole is said to accept an input string if there is at least one path of execution that accepts it. nondeterminism is needed.” They are equivalent to ﬁnite-state machines augmented by a stack to implement recursion. The language is very simple. If we chart the calling pattern of the accepting execution we end up drawing the parse tree. Theorem 18 For every pushdown machine M there is a context-free grammar G with L(M ) = L(G). Theorem 17 For every context-free grammar G there is a pushdown machine M with L(M ) = L(G). (We don’t prove this here. The above construction shows how to turn any context-free grammar into a program that is like the ﬁnite programs of Chapter 2 except that it uses two additional features: recursion and nondeterminism. These calls correspond to the children of the root of the derivation tree. The problem is. cannot be implemented with a single stack. . and T. with information limited by a constant amount. This worked because the power set (set of subsets) of a ﬁnite set was again a ﬁnite set. The trick was to replace states by sets of states.3. We do not prove this here.2. terminal(’+’).
  176ﬀ.) While a deterministic recursive-descent parser does not exist for all contextfree languages. First. the righthand sides diﬀer in their ﬁrst character. which is impractical even when implemented by a deterministic tree-traversal.74 CHAPTER 3. we change the grammar into “Chomsky Normal Form. Principles of Compiler Design. The new grammar leads to a parser in which the decision which of the nondeterministic paths has a chance of leading to acceptance of the input can be made on the basis of looking at the next character of the input. which was to generate intermediate strings of the form T + T + ··· + T is done by the ﬁrst two productions of the new version. 1978. This is so because whenever the grammar has several right-hand sides for the same left-hand symbol. into the following: E E T T F F F → → → → → → → TE +T E | λ FT ×F T | λ (E) x y The work done by the ﬁrst two productions version. A General Parsing Algorithm Instead of using nondeterminism. depending on the implementation) builds up information about subproblems. Dynamic programming is a type of algorithm somewhat resembling “divide-and-conquer. without changing the language. The resulting deterministic parser in shown in Figure 3.” which inductively (or recursively. CONTEXT-FREE LANGUAGES A “Predictive” Recursive-Descent Parser (The example is after Aho and Ullman. the grammar for arithmetic expressions used earlier can be transformed. some languages are simple enough to yield to such an approach. one can use a “dynamic programming” approach to parse any context-free language.” then we apply the actual dynamic programming algorithm. There are two steps to using this technique. For example. p. Chomsky Normal Form Every context-free grammar can be converted into an equivalent one in which each production is of one of the two forms . It typically has O(n3 ) time complexity.2.
  just return } T ( ) { F(). F(). } E’ ( ) { if (NEXTCHAR == ’+’) {terminal(’+’). E’(). case ’y’: terminal(’y’). E(). } T’ ( ) { if (NEXTCHAR == ’*’) {terminal(’*’). case ’x’: terminal(’x’)..} else // do nothing. E’(). T().} else // do nothing. } } . break. default: REJECT.. T’(). just return } F ( ) { switch (NEXTCHAR) { case ’(’: {terminal(’(’).2. } E ( ) { T(). else REJECT. PARSING 75 main ( ) { E(). if (getchar() == EOF) ACCEPT.3.2: A deterministic recursive-descent parser .} break. terminal(’)’). break. T’(). Figure 3.
  in a pattern indicated by the two arrows in the Figure. B . pp. Languages.) CYK Parsing: An Figure 3. starting position = 3” ﬁeld Example of “Dynamic contains all those variables of the grammar from which one can derive the subProgramming” string of length 2 that starts at position 3. or even sets of parse trees. The whole string is in the language if the start symbol of the grammar shows up in the bottom ﬁeld.76 CHAPTER 3. The “length = 2.1 can equivalently be replaced by the set of productions A → BCDEF G D → d F → f and the production 3.2) 3. The variables are S and C . Introduction to Automata Theory.) The rows of the table get ﬁlled in from top (length=1) to bottom (length=5).2 can then be replaced by the set of productions A → BC C → CD D → DE E → EF F → FG (The example below is from Hopcroft and Ullman. not necessarily diﬀerent ones. 1979. (The string is ab.3 shows an example.3 The Pumping Lemma for Context-Free Grammars The grammar .1) The production 3. (3. 140-1. Filling in a ﬁeld makes use of the contents of ﬁelds ﬁlled in earlier. This form of a context-free grammar is called “Chomsky Normal Form. CONTEXT-FREE LANGUAGES A A → → BC a where A. The algorithm is easily modiﬁed to yield a parse tree.” To see how this conversion can be done consider this production: A → BCdEf G (3. and a is a terminal symbol. and Computation. C are syntactic variables.
  C B B S. A φ φ S. A.3. C B B b 4 a 5 A. THE PUMPING LEMMA FOR CONTEXT-FREE GRAMMARS 77 S A B C → → → → AB BA CC AB | | | | BC a b a starting position length 1 2 3 4 5 B b 1 a 2 A. C S. C Figure 3. A.3. C a 3 A. A S. C S.3: An example of “CYK” parsing .
  m ≥ 1}. m ≥ 1}. The languages L8 and L9 are of no practical interest. as for example in L9 = {an bm cn dm : n. L6 = {an bm cn |n. It is the most common argument for proving languages not to be context-free. a. then these language all are examples of balanced parentheses. If you try to write context-free grammars for L8 and L9 .e.) Similar grammars generate L5 = {0n 1n 0m 1m |n. Then L7 becomes {(n [m ]m )n |n. m ≥ 1}. In fact. the numbers of repetitions of characters) in the deﬁnitions of these languages.. The proof of Theorem 19 uses a “cutting-and-pasting” argument on the parse trees of context-free grammars. If we think of 0. and if we match numbers of characters beyond a nesting pattern. you sooner or later get frustrated trying to achieve the speciﬁed equalities between the exponents (i. A 1 L is an example of nesting of diﬀerent types of parentheses. m ≥ 1}. and d as closing parentheses. as for example in L8 = {an bn cn : n ≥ 1}. and b as opening parentheses and 1. Theorem 19 L8 and L9 are not context-free. (0n means a string of n 0s. c.78 CHAPTER 3. L7 = {an bm cm dn |n. But after we develop a proof technique for showing that these languages are not context-free we will be able to use that technique to show that some aspects of programming languages are not context-free either. m ≥ 1}.1 More complicated languages arise if we try to have equal numbers of not just two but three or more diﬀerent characters. CONTEXT-FREE LANGUAGES S generates the language → 0S1 | 01 L4 = {0n 1n |n ≥ 1}. The heart of the argument is that every context-free language has a certain “closure” property: The fact that certain strings are in the language forces other strings to be in the language as well. Replace a and d by opening 7 and closing parentheses and b and c by opening and closing square brackets. .
  z in Figure 3.3.4. If a language L is context-free there is. THE PUMPING LEMMA FOR CONTEXT-FREE GRAMMARS 79 >= log bn height A repetition A u v x >= n leaves y z Figure 3.5 is also a parse tree in this grammar. by deﬁnition. call it A. the string uv i xy i z becomes uxz and uvxyz . where height is deﬁned to be 0 for a tree consisting of a single node. x. resulting in a tree that derives the string uvvxyyz . v . In other words. respectively.4: Illustrating the proof of Theorem 19 language that does not have the closure property cannot be context-free. It derives the string uxz . Figure 3. Let b be the maximum length of right-hand sides of productions of G. We can build trees that derive uvvvxyyyz . L = L(G). the substring x from the second occurrence. the second occurrence of A can be made to derive vxy . a context-free grammar G that generates L.) Knowing that we can derive x from A. 1. and pick two occurrences of A along one path. Then a parse tree with n leaves must be of height at least logb n.”) Then some syntactic variable must show up at least twice along that path. No need to stop there. There are other trees we can construct. The details of that closure property are developed next. in fact any string uv i xy i z for any i ≥ 2. y .4 illustrates this.3. labeled u. What if n is so large that logb n − 1 exceeds the number of diﬀerent left-hand sides of productions in G? (These are also called “syntactic variables. This means that a tree with n leaves must have a path from the root to a leaf with at least logb n − 1 interior nodes.5. Pick such a variable. (The substring vxy is derived from the ﬁrst of the two occurrences of the variable A. For example. as illustrated in the bottom half of Figure 3. This lets us summarize the eﬀect of all this cutting-and-pasting of trees as the fol- . For i = 0. These two occurrences of A split the string w that the tree derives into ﬁve parts. uvvvvxyyyyz . the tree shown in the top half of Figure 3. b is a bound on the maximum number of children for nodes in parse trees of G.
  80 CHAPTER 3.5: Deriving uxz and uvvxyyz . CONTEXT-FREE LANGUAGES A x u z A A u v A y z v x y Figure 3.
  3. But this is not true. x. y .3 This contradiction shows that L8 is not context-free. The proof is left as an exercise.3. The Pumping Lemma can be stated as follows. implying that it is possible to split w into ﬁve parts. L context-free =⇒ There is a constant k such that for all strings w ∈ L with |w| ≥ k there are strings u. and c’s following each other. Assuming it was. w = uvxyz . Why is this so messy? . z with w = uvxyz and |vy | > 0 such that for all i ≥ 0. 3 One can come up with a more detailed argument for this. w = uvxyz . The length of w is k or more (in fact.2 This lemma is strong enough to let us prove that the language L8 = {an bn cn : n ≥ 1} from Theorem 19 is not context-free. Version 1) If L is an context-free language then there is a constant k such that for every string w in L of length k or more the following is true. 2 This last part of the statement we did not prove. x. b’s. v . for all i ≥ 0. An added challenge is to come up with a concise argument. z . hence statements (1) — (3) from the Lemma apply. Then Lemma 4 applies. such that 1. 3k ). y . such that |vy | > 0 and uv 2 xy 2 z ∈ L8 . uv 2 xy 2 z is never of the form of equal numbers of a’s. which therefore does not contain equal numbers of all three characters anymore. Here is an attempt: The substring v cannot contain more than one type of character because otherwise v 2 contains characters in an order not permitted in L8 . The challenge is to cover all ways in which the strings could get divided up into ﬁve pieces. |vy | > 0. Hence at most two types of characters take part in the “pumping” from uvxyz to uv 2 xy 2 z .) Proof that L8 is not context-free. Consider the string w = ak bk ck . uv i xy i z ∈ L. (We will need to add one more reﬁnement before we can handle L9 . and 2. The same is true for y . v . Let k be the constant said to exist in Lemma 4. THE PUMPING LEMMA FOR CONTEXT-FREE GRAMMARS lowing 81 Lemma 4 (Pumping Lemma for Context-Free Languages. There are ﬁve strings. u. The statement is saying that not both of v and y are empty. No matter what ﬁve-way split of w one tries. uv i xy i z ∈ L. and 3.
  ” There are formal results that say that adding a level of alternation makes statements harder in the sense that even if we had a program that could determine the truth of statements with q alternations of quantiﬁers.” Spelled out in detail. (The concept of transformations is rearing its ugly head. This is what we did in the previous example. y . this version reads L not context-free ⇐= For all constants k there is a string w ∈ L with |w| ≥ k such that for all strings u. How do we prove something “for all constants k ”? By starting out the proof by saying “Let k be an arbitrary constant” and carrying the k along as a parameter of the argument. In this example. . this would not enable us to build a program that determines truth of statements with q + 1 quantiﬁers. Thus. CONTEXT-FREE LANGUAGES What makes such a statement hard to digest are the alternations of “for all” with “there is/are.” where B is that complicated statement with four alternating quantiﬁers. Another. What’s next? We need to show that there exists a string w with certain properties. . To show this. we can prove that some language L is not context-free by proving the above four-line statement ¬B : “For all . x.4 The above statement is of the form “L context-free ⇒ B .” akin to the nesting of loops in a program. a more useful version of the statement is “L not context-free ⇐ ¬B .) Look for the subject of “the arithmetic hierarchy” in textbooks on recursive function theory if you want to read about this. v . This language L is not context-free. let’s pick w to be the string 1···1+2···2 = 3···3 k 4 This k k this called an “alternation of quantiﬁers. a fact we prove by proving the statement “¬B ” above.82 CHAPTER 3. ∈ L”. . all we need to do is exhibit one such string. Since we are trying to prove that some languages are not context-free. z with w = uvxyz and |vy | > 0 there is an i ≥ 0 with uv i xy i z ∈ L. Here is another one.
  m > 0}. It proves nothing.2. Case 2. we would sooner or later realize that there is a case (or two) that does not work.1.2. This is the case where v consists of a number of a’s and y consists of the same number of c’s.2. Case 2. Case 2.2. Case 2. x. thus creating a string outside of L.3. Case 2. One of v and y is empty. Either v or y or both contain the + and/or the =. It is tedious. creating a string outside L. This case is analog to Case 2. Neither v nor y is empty.3. v lies within the second number of the equation.2.5 It takes a stronger version of the Pumping Lemma to prove that this language is not context-free. Then pumping (with any i = 1) changes that side of the equation but not the other.) The proof attempt fails.1. Here is such an exhaustive set of cases for this example. Case 2.1. Neither v nor y contain the + or the =. 5 This is nothing but bad news. THE PUMPING LEMMA FOR CONTEXT-FREE GRAMMARS 83 (Note how the k plays its role.2.2.) What’s the next step? We have to show that for all strings u. Failing to prove that L is not context-free does not prove that it is context-free either. z which represent a ﬁve-way breakup of w (“w = uvxyz ” ) and which don’t have the second and fourth piece both empty ( “|vy | > 0” ).2. Case 2.2. there is a constant i with uv i xy i z ∈ L. Going back to the language L9 = {an bm cn dm : n. if we tried to con. Then uv 2 xy 2 z is of the form 1···1+2···2 = 3···3 >k k >k which is not a correct equation. Case 1. (The other case is the analogous situation with b’s and d’s. but it can be done.2. Then pumping changes one side of the equation but not the other.1.2. v lies within the ﬁrst number of the equation.An example that does struct a proof that it is not context-free by constructing an exhaustive set of not work cases just like in the previous example. by dealing with an exhaustive set of cases. uv 2 xy 2 z is of the form 1···1+2···2 = 3···3 k >k >k which is not a correct equation. v . . This is the hard part of the proof because we somehow have to provide an argument that covers all possible such ﬁve-way breakups of w. Then uv 0 xy 0 z ∈ L if for no other reason then because it does not have a + and an = in it. y . Both v and y lie on the same side of the =. v lies entirely on the left side of the equation and y lies entirely on the right side.
  ” This is very helpful. CONTEXT-FREE LANGUAGES A Stronger Version of In the proof of the Pumping Lemma. and all of this is then followed by opening cs matched by closing ds. We only have to show that For all constants k there is a string w ∈ L with |w| ≥ k such that for all strings u. v .) Now it works The only diﬀerence to the old version is the additional condition that “|vxy | ≤ k . because for v to consist of a’s only and y of c’s only (or b’s and d’s. Here are the details. because it means that cases in which “|vxy | > k ” need not be considered. (There is a little subtlety in this statement in that the old constant k and the new constant k were combined into a single constant. respectively) the string vxy would have had to stretch over all the b’s (or the c’s) and hence be longer than k .” When writing grammars for languages like {ai bi cj dj : i. This number is a constant that depends only on the grammar. we could instead argue that such a repetition can always be found “close to” a leaf. y .4 A Characterization of Context-Free Languages As alluded to in the title of this current chapter. A complete grammar for the above language is . The type of production that is at the core of a grammar for such matching characters is T →aT b which results in a path of T s in a derivation tree with matching as and bs being generated on the left and right of the path. context-free grammars “place parentheses. This then implies a limit on how long the string vxy can be.) This improvement in the proof of the Pumping Lemma makes it easier to prove that a language L is context-free. though. x. then a repetition of a syntactic variable occurs within s + 1 steps along the path from a leaf to the root and the subtree under the second occurrence of the variable (second meaning the one further away from the leaf) has at most k = bs+1 leaves. which is the maximum of the two and called again k . 3. This takes care of the failed cases in the unsuccessful proof attempt above. If a grammar has s diﬀerent syntactic variables and no right-side of a production is longer than b. instead of merely arguing that there is the Pumping Lemma a repetition of some syntactic variable along a path from the root to a leaf. (Draw a picture.84 CHAPTER 3. z with w = uvxyz and |vy | > 0 and |vxy | ≤ k there is an i ≥ 0 with uv i xy i z ∈ L. it helps to recognize the pattern of “parentheses” in the language: Every “opening” a is matched by a “closing” b. j ≥ 0}.
  The grammar S S S S S → SS | λ → (S) → [S] → {S} → a | b generates all properly nested strings of three types of parentheses with letters a and b interspersed. and let T = {a1 . Theorem 20 Every context-free language is equal to EraseA (R ∩ P ARn (T )) for some sets A and T of characters. let EraseA (L) be the set of all strings in L from with all occurrences of characters in A have been deleted. . am interspersed. for every set A of characters and every language L. . then the grammar S S → → . am } be any ﬁnite set of characters. . . If we use (i and )i to denote parentheses of type i. What this theorem says is that every context-free language can be built by starting with a context-free grammar that produces essentially nothing but . . A CHARACTERIZATION OF CONTEXT-FREE LANGUAGES 85 S T T → T T → aT b | λ → cT d | λ On the other hand. . of course. . . a non-nested pattern of matching characters as in {ai bj ci dj : i. Let’s call this language P ARn (T ).4. → → SS | λ (1 S )1 S S (n S )n a1 | · · · | am generates all properly nested strings of these n types of parentheses with letters a1 . j ≥ 0} is beyond the power of context-free grammars (as can be shown with a pumping argument). Furthermore. It is easy to see that any depth of nesting with any set of parentheses-like characters is within the reach of context-free grammars. and some integer n ≥ 0. Three types of parentheses is not the limit. nor are two characters. 1 ≤ i ≤ n. some regular language R.3.
  1. does this imply that the string is not in the language? What if one of them is ∅? What if all the entries in the second row are ∅? The third row? Fourth? Ex.3. Pumping Lemma Ex. 010000#111111. CONTEXT-FREE LANGUAGES nested parentheses.2. 1}∗ . grammar Is it decidable if a context-free language. . 3. 3. and the bitwise and between α and β is not all zeroes }. Prove that L is not context-free. . if all the entries in the top row are ∅. Examples of strings in L are 01#11. 3. and intersecting this language with a regular language and ﬁnally throwing away all occurrences of certain characters. given by a context-free 1. contains a string of length less than 100? 2. L = {an bm cq : n. any placement of matching characters that does not follow the nesting pattern of parentheses is bound to be beyond the power of context-free grammars. . Is it decidable if the set of all outputs of a program forms a contextfree language? Explain brieﬂy. β ∈ {0. Which of the following languages are context-free? . 3. . L = {an bm cq : n. .6.4. contains a string of length more than 100? 3. . The only contextfree step in this construction is the initial generation of P ARn (T ).] Exercises Ex. .5. Consider the language L = {α#β : α. m. [A proof of the above theorem can be found in Davis and Weyuker. L = {an bm : n. Which of the following languages are context-free? 1. In the dynamic programming algorithm for parsing. . q ≥ 1 and at least two of the three numbers are equal } 3. 3. α and β have the same length. 3. m ≥ 1 and both even } 2. This supports the contention that the only aspect of a context-free language that could possibly require a context-free grammar is some form of placement of parentheses and conversely. 010101#101011. q ≥ 1 and at least two of the three numbers are diﬀerent } Ex. Ex. m. is ﬁnite? Ex. .86 CHAPTER 3.
  The latter approach entails a redeﬁned notion of “height” of a parse tree. What is that? (b) There are two ways to correct this deﬁciency in the proof. Any ﬁnite-state machine for a language L can be turned into a context-free grammar for L. then the grammar has a production A → cB . and T ) has unit productions but does not have derivation trees with v and y both empty. j ≥ 0} 3. One is to show that unit productions can be removed can be removed from any grammar without changing the language — this involves suitable changes to the other productions. This might happen when the grammar has what is known as “unit productions” — productions that merely replace one variable by another.3. · · · . · · · . A ∩ B may not be. For example. our grammar for arithmetic expressions (the version with variables E . F . L10 = {ai bj aj bi : i. Usually. What would be a suitable modiﬁcation of this deﬁnition? Ex. L11 = {ai bj ai bj : i. The other is to rephrase the proof of the Pumping Lemma to work around the fact that some nodes of a parse tree might only have one child. 1} . Even if A and B are both context-free. If the machine has a transition labeled c from state A to state B . 3. q ≥ 0.8. Something else must be true about the unit productions. the height of a tree is deﬁned to be 1 plus the maximum height of its subtrees. 3. Ex. (An example of a unit production is E → T in one of our grammars for arithmetic expressions.7.) Prove your answer either by giving a grammar or by providing an argument that the language is not context-free. α2 . βi ∈ {0. L12 = {α1 #α2 # · · · #αp $β1 #β2 # · · · #βq : p. Prove this by giving an example of two context-free languages whose intersection is not context-free. j ≥ 0} 2. αi .) . A CHARACTERIZATION OF CONTEXT-FREE LANGUAGES 1. αp } ⊇ {β1 . 3. βq }} 87 (L12 captures the properties of some programming languages that variables need to be declared before they are used.4. (b) What other productions need to be added to the grammar to make sure it derives the same strings that the machine accepts? (Give an answer for the speciﬁc example and also for the general case. and {α1 . (a) Carry out this translation from machine to grammar for the machine that accepted binary numbers divisible by 3. The proof of the Pumping Lemma did not show that the pumped strings v and y could not both be empty. Ex. β2 .9.) (a) The mere presence of unit productions does not yet make it possible for both v and y to be empty.
  88 CHAPTER 3. The special form of the grammars obtained from ﬁnitestate machines suggest a simpliﬁcation to the statement of such a Pumping Lemma. then the Pumping Lemma applies to every regular language. (a) Spell out such a simpliﬁed version. . (b) Use it to prove that L13 = {an bn |n > 0} is not regular. CONTEXT-FREE LANGUAGES Ex. if every regular language is context-free (as proven by that exercise). 3.10. Following up on the previous exercise.
  into three time slots. In terms of Boolean expressions. some of which (3-Color) cannot take place at the same time. let’s say that any algorithm whose running time is bounded by a polynomial in the input length is “reasonably fast. Satisfiability. given an expression E . whether or not there is an assignment of Boolean values to the wires in the circuit that “satisﬁes” all the gates. Then the problem becomes. false to the other.” Below that. trying to ﬁt a set of activities. assigning true to both inputs of an and-gate and false to its output would not be consistent. and true to the output wire.1 The State of Knowledge Before NP-Completeness Up to and through the 1960s. Here are four examples of such “intractable” problems. local optimization. the problem is to ﬁnd out. But there were also plenty of problems which deﬁed all attempts at solving them with reasonably fast algorithms1 . Graph 3-Colorability It is really a scheduling problem. introduced in Chapter 1. “Satisfying a gate” means assigning consistent values to its input and output wires. given a circuit C . Boolean expressions or circuits — one can easily translate circuits into expressions and vice versa — turn out to be a rather ﬂexible “language” for expressing constraints on solutions in many situations.” there is general agreement that the exponential running of exhaustive search is not “reasonably fast. nor would be assigning true to one input wire. This situation is captured by a graph with the activities being the nodes and with conﬂicting activities being connected by an edge.” 89 . given a 1 On the question of what is “reasonably fast. whether or not there is an assignment of Boolean values to the variables of E that makes E true. can be phrased in terms of Boolean Satisfiability expressions or in terms of circuits. algorithm designs such as greedy algorithms. the problem is to ﬁnd out. and dynamic programming were applied successfully to many problems. For example. In terms of circuits.Chapter 4 Time Complexity 4. divide-and-conquer. Graph 3-Colorability (3-Color) problem also was introduced in Chapter 1.
  such as how to handle operations other than and. It would not work very fast. and the Yes/No question to decide is whether or not the graph G can be covered with n nodes. we only need to keep track of the assignments to those variables that also occur in the other “half” of the expression. yes.3 provides an illustration. . let us try to solve Sat with a divide-and-conquer approach. but in order to satisfy the right subtree it is necessary to make the same variable x false. true. Then to compute the set of assignments that satisfy the whole expression. . it would work. The challenge is to “cover” the graph in the following sense. It could be that in order to satisfy the left subtree it is necessary to make a variable. . but. let’s ﬁnd the set of assignments of true and false to its variables that satisfy it. The problem is to cover all the edges of the graph choosing the smallest number of nodes. the graph in Figure 4. Hamiltonian Path Think of a graph as a map of cities that are connected by roads. Vertex Cover Vertex Cover is another problem deﬁned on graphs. Figure 4. How Algorithm Designs It is instructive to try to apply some of our algorithm design ideas to some of Can Fail: One Example these intractable problems. As one example. but not with 3. In the Hamiltonian Path problem the challenge is to ﬁnd a tour from a given start node to a given end node that visits each city exactly once. . because the sets of assignments can get large. Instead of just ﬁnding out whether or not a subtree is satisﬁable. how . For example.90 CHAPTER 4. to ﬁnd out if it is possible2 to color each node with one of the three colors without ever giving the same color to the two endpoints of an edge. Consider the parse tree of the expression and divide it into the left and right subtree. Dividing a Boolean expression is easy. there is the “decision version” — “Is it possible .1 can be covered with 4 nodes. if both subtrees are satisﬁable and are connected with an and operation. Have we succeeded in devising a divide-and-conquer solution to Boolean satisﬁability? We need to ﬁll in some details. . in fact exponentially large in terms of the number of variables. ?”. This is a special case of the so-called Traveling Salesman Problem (tsp) where each edge has a length and the challenge is to ﬁnd a tour of minimal length that visits each city. In the decision version of this problem you are given a graph G and a number n. though. This suggests a more sophisticated divide-and-conquer. the whole expression may still not be satisﬁable. A simple-minded notion of “conquering” a subtree would be to ﬁnd out whether or not it is satisﬁable. You can choose a node and thereby “cover” all the edges connected to that node. we can intersect (in the case of an and) the sets of assignments that satisfy the two parts. So we have a exponential-time divide-and-conquer 2 With all these problems. say x. ?” — and the full version — “Is it possible and if so. In fact. For example. But this information about a subtrees is not enough to ﬁnd out whether or not the whole expression is satisﬁable. TIME COMPLEXITY graph.
  4.2: Illustrating the Hamiltonian Circuit problem . THE STATE OF KNOWLEDGE BEFORE NP-COMPLETENESS 91 Figure 4.1.1: Illustrating the Vertex Cover problem Figure 4.
  . y=F x = T. TIME COMPLEXITY (x.. y=F x=F. x=F.z) (..... x = T.. y=F Figure 4.w) x = T..3: Trying to solve Sat with a divide-and-conquer approach .x. y=F ..x..y. y=T .92 CHAPTER 4..y..
  To make a long story short. A nondeterministic program. Each node at that level in the execution tree corresponds to one assignment of colors to the nodes of the graph. After all.4. there is successive branching that builds up a tree with 3n nodes at the end of this loop. as described earlier. resulting in an execution tree. but so do Sat. there is no polynomial-time algorithm for Sat. it prints “No. may contain a statement like nondet {x[i] = BLUE. This leads us to developments of the early 1970s. With this notion of programs and time. Hamiltonian Circuit) and dozens or hundreds3 of similar “combinatorial optimization” problems. If this statement is inside a loop that runs i from 1 to n. x[i] = GREEN. Second. If so. The same is true for the other three problems (3-Color. we deﬁne the output of the whole tree to be “Yes” if there exists a path at the end of which the program prints “Yes” (and “No” if the program prints “No” at the end of all paths). nobody has succeeded in trying to do that. What to do? We could try to compute and store those sets more eﬃciently. Figure 4.” For an example. This is the notion of nondeterministic time. 4.” The technical term for this form of cheating is “nondeterministic time.2 NP Sat. this nondeterministic time allows us to conduct what amounts to an exhaustive search without paying for it. not only 3-Color becomes easy to solve in polynomial time. we could have done a simple exhaustive search in the ﬁrst place. 3-Color and all the other problems are in fact easy to solve. Vertex Cover. Vertex Cover. we deﬁne the time spent on this computation to be the height of the tree. it prints “Yes. in poly. NP 93 algorithm for Sat.” if not. consider 3-Color. Deﬁnition 5 The class of problems that can be solved in nondeterministic polynomial time is called NP.” Now is the time to “cheat.4 shows the shape of such a tree. Hamiltonian Circuit and in fact just about any combinatorial optimization problem that you can think of.” in two ways. x[i] = RED. 3 depending on whether one counts related problems as separate ones . But if we were willing to tolerate exponential running times (which we are not). we can have a deterministic function that tests if the assignment of colors is a correct 3-coloring.Nondeterminism nomial time — if we “cheat.2.} Execution of such a statement causes a three-way branch in the execution sequence. and in fact there is evidence that none might exist. Following this loop. Up to this date. First.
  Testing Phase No No .. . .. . No No No Figure 4. . . . i=1 x[1] = BLUE x[1] = GREEN NonDeterministic i++ x[2] = GREEN Phase . . . . TIME COMPLEXITY .. .4: The shape of an execution tree for 3-Color . Yes No .94 CHAPTER 4..
  and the operation of a gate in the circuit at time Boolean Expressions t becomes a boolean operation that deﬁnes the correct operation of the gate. we did transformations of two diﬀerent problems into Sat. more precisely.3 The NP-completeness of Satisfiability The big step forward in the early 1970s was to realize that many of these intractable problems were related to each other. 3-Color. deterministically. no less ambitious than 3-Color)..e. One was Graph 3-Coloring. Transforming speciﬁc problems into Sat is usually not that hard. Hamiltonian Circuit ∈ NP. The value of a wire w of a circuit at time t becomes the value of a and Satisfiability of variable wt in the expression. 5 honestly.3.4 4. Moreover. i. 4 Here we are talking about the “decision versions” of the problems. just the right use of familiar concepts like compilation and circuits. For example. Vertex Cover. Since we are now concerned about running time. What happens when a Boolean circuit executes can be captured by a boolean Circuit Satisfiability expression. but it is not diﬃcult to verify that Observation 7 There is a polynomial-time5 transformation of 3-Color into Sat. we would also have a polynomial-time algorithm for 3-Color. it matters how fast that transformation can be carried out. In Chapter 1. The proof of Cook’s Theorem needs to be an argument that for any problem in NP there is a polynomial-time transformation into Sat. All we know is that the problem is in NP. polynomial-time . there aren’t a lot of messy technical details to this proof.4. [Coo71]) For every decision problem L ∈NP. 3-Color can be transformed into Sat. the other a problem of addition of binary integers. as we have seen in Chapter 1. THE NP-COMPLETENESS OF SATISFIABILITY 95 Observation 6 Sat. The challenge with Cook’s Theorem is that we need to argue that there exists a transformation of a problem into Sat without even knowing exactly what that problem is. Observation 8 If we could ﬁnd a polynomial-time algorithm for 3-Sat. As a consequence. there is a polynomial-time transformation of L into Sat. what the Observation says about 3-Color is also true for Vertex Cover and Hamiltonian Circuit and in fact for all problems in NP: Theorem 21 (Cook. As it turns out. the versions that ask for a “Yes/No” answer. This last Observation makes 3-Sat the more ambitious of the two problems to solve (or.
  Circuit Satisfiability is the problem. The program is stored as some pattern of 0s and 1s on . Satisfiability of Boolean expressions is the problem. of deciding whether or not it is possible to assign values to all the wires at all times such that all gates operate correctly at all times.96 CHAPTER 4.5. at the risk of being repetitive.out < infile then the situation just before the execution of the compiled program is illustrated in Figure 4.5: A simple computer. given a circuit with some constant values given to some wires at some points in time. of deciding whether or not it is possible to assign values to the variables such that the value of the whole expression becomes true.c a. If you issue a sequence of two commands like cc my. given a Boolean expression which may contain constants. TIME COMPLEXITY P X N M bits set to represent machine-level version of program bits to represent input file Figure 4. just before executing a program with an input ﬁle Thus.
  What question? With certain wires initially set to 0s and 1s to represent the program and the input ﬁle. THE NP-COMPLETENESS OF SATISFIABILITY 97 some of the wires.” i. In fact.e..c is such that it always produces a 0 or a 1 as an answer (or a “Yes” or “No”). This is a wire which in the circuit satisﬁability problem that is being constructed is simply not assigned a value. is that whole circuit satisﬁable? This transformation of a question about a program and input ﬁle into a satisﬁability question — a transformation of software and data into hardware — is most of the proof of Cook’s Theorem.. can be transformed (by a polynomial-time transformation) into Circuit Satisfiability. what we have proven so far is that every problem in “P.c.c a. deterministically) be solved in polynomial time. Let’s say the program my.6 What happens when you issue the statement cc my. every problem that can really (i. else x = FALSE. What happens to the preceding “compilation” process if the program my. “Compilation” into a circuit . What are we accomplishing with all of this? A transformation.c on input infile produce a “Yes”? is equivalent to a satisﬁability question about our circuit.} How would we compile this into our circuit? Much like an if-statement if . and with the “output wire” set to 1 when the computation terminates. Then the question Does my.out < infile to this circuit? The “compiling” process creates the machine-level version of the program my.c had a statement nondet {x = TRUE.. but our proof works equally well if we now allow nondeterminism in the program my.c and stores it by setting some of the wires of the circuit to speciﬁc values.3. x = TRUE.c is Enter Nondeterminism nondeterministic? What if my.4. All we need is 6 Otherwise we would need to draw a disk and turn our neat “circuit” satisﬁability problem into a “circuit–and–disk” satisﬁability problem.e. x = FALSE. and we might as well assume that the input ﬁle has been copied from disk and is also stored on some set of wires. except that we let the “choice” between the two options depend on a wire that is not the output of any gate.. That result is worthless.
  4. Proof It is easy to see that Vertex Cover can be solved by the exhaustive search approach that costs only “nondeterministic polynomial time” — Vertex Cover is in NP. (nor-gates would have been the other possibility. That proof showed that Satisfiability of circuits was NP-complete. For such a transformation of 3-Sat to Vertex Cover to prove that Vertex Cover is NP-complete we have to show that 3-Sat is NP-complete. To simplify matters we could build those circuits out of just one type of gate. In 1972. If we could turn each clause into 3-conjunctive normal form. with all those clauses anded together. TIME COMPLEXITY one such wire. We do this by slightly modifying our original proof showing that Satisfiability was NP-complete.) Each 3-literal disjunction is called a “clause” of the expression.4 More “NP-Complete” Problems Deﬁnition 6 A problem in NP which has the property that every problem in NP can be transformed into it by a polynomial-time transformation. Then compiling a two-way nondeterministic branch is just like an if-statement with the choice being determined by the value of that wire at the time of the execution of the statement. Turning circuits into boolean expressions results in one “clause” per gate. Since there is a polynomial-time transformation from any problem in NP to Satisfiability this then implies that there is (a two-step) polynomial-time transformation from any problem in NP to Vertex Cover. is called NP-complete. (There is no requirement that literals in a clause have to be diﬀerent. Richard Karp ([Kar72]) showed that other problems were NP-complete as well. To show that Vertex Cover is NP-complete we describe a polynomial transformation from Satisfiability into Vertex Cover. we would have shown 3-Sat to be NP-complete.” or 3-Sat for short. Theorem 22 Vertex Cover is NP-complete. Cook’s Theorem states that Sat is NP-complete.) Each gate then turns into a clause of the form .98 CHAPTER 4. This transformation is made easier if we transform only a special form of Satisfiability known as “Satisfiability of boolean expressions in 3-conjunctive normal form. Expressions of this form look like this: (x ∨ ¬y ∨ z ) ∧ (w ∨ z ∨ w) ∧ (¬x ∨ ¬y ∨ ¬z ) ∧ (z ∨ ¬w ∨ ¬z ) They are conjunctions (ands) of disjunctions (ors) of three “literals” — variables or negated variables. Vertex Cover was one of them. Let’s choose nand-gates.
  4.6: Translating 3-Sat into Vertex Cover (¬(xt ∧ yt ) ⇔ zt+1 ) which we could split into (¬(xt ∧ yt ) ⇒ zt+1 ) ∧ (zt+1 ⇒ ¬(xt ∧ yt )) and further rewrite into ((xt ∧ yt ) ∨ zt+1 ) ∧ (¬zt+1 ∨ ¬(xt ∧ yt )) One more rewriting get us close to the right form: (xt ∨ zt+1 ) ∧ (yt ∨ zt+1 ) ∧ (¬zt+1 ∨ ¬xt ∨ ¬yt ) The rewriting into 3-Sat is complete if we repeat a literal in those clauses that are short of having three literals: (xt ∨ zt+1 ∨ zt+1 ) ∧ (yt ∨ zt+1 ∨ zt+1 ) ∧ (¬zt+1 ∨ ¬xt ∨ ¬yt ) This proves Lemma 5 3-Sat is NP-complete. consider the expression (x ∨ ¬y ∨ z ) ∧ (x ∨ z ∨ w) ∧ (¬x ∨ ¬y ∨ ¬z ) ∧ (x ∨ w ∨ ¬z ) . MORE “NP-COMPLETE” PROBLEMS (x ∨ ¬y ∨ z ) ∧ (x ∨ z ∨ w) ∧ (¬x ∨ ¬y ∨ ¬z ) ∧ (x ∨ w ∨ ¬z ) x ~x y ~y z ~z w ~w 99 12 ~y z ~y w x z x w ~x ~z x ~z Figure 4.4. As an example. What’s left to do in order to show that Vertex Cover is NP-complete is to transform 3-Sat into Vertex Cover.
  ¬x) and the (z. ¬z ) edges? They are not covered yet. The same applies to the second triangle from the left. We are doing well so far. So just “eyeball” the given example. w covers the only edge that ¬w covers and then some.6. importantly. No matter how we choose nodes within these set-aside allocations. there needs to be a number — the number of nodes about which we ask whether or not there is a vertex cover of that size. we have to choose values for the variables such that for each clause at least one of its literals is true. w) edge down to the rightmost triangle is now covered and there is no disadvantage in choosing the other two corners of that triangle for covering it. all the edges between those two triangles and variable edges are covered as well.6 for an illustration. But we have spent . That’s not a good idea. It is probably small enough to do that with success. ¬w) variable edge.) Choosing the ¬y endpoint of the (y. Two triangles are covered (which as said before is automatic) and. Here is the secret for success: We have to choose endpoints of the variable edges such that for each triangle at least one of three edges from the triangle corners up to the variable edges is covered from above. This number is equal to the number of diﬀerent variables in the expression plus twice the number of clauses in the expression. to cover a triangle. the variable edges and the clause triangles will all be covered. The (w. What happened to the (x. Thus there is no disadvantage in choosing w. For example. In the example. (One can already smell the connection to the 3-Sat problem: To satisfy the expression. (These “clause triangles” are at the bottom of the graph in Figure 4. but a single corner does not. any two of its three corners suﬃce. Finally. either one of its endpoints suﬃces. This has further ramiﬁcations. (These “variable edges” are at the top of the graph in Figure 4. let’s build up some intuition about the matter by trying ﬁrst to ﬁnd a vertex cover for the graph. So we might as well set aside two of our 12 choices of nodes for each of the triangles. ¬y ) edge then lets us take care of the remaining two triangles.) For each variable in the expression there is an edge in the graph with one of its endpoints labeled by that variable and the other by the negation of the variable. How would we go about that? One might be tempted to think about some algorithm for this. So we set aside one of our 12 choices for each variable edge.100 CHAPTER 4. but only between nodes that have the same label. (See Figure 4. Heuristics may help. this number is 12. And to cover a variable edge. TIME COMPLEXITY The translation algorithm is as follows. Consider the (w. The challenge is to make the remaining choices such that the edges than run between variable edges and clause triangles get covered as well. for this to be a “Yes/No” Vertex Cover problem.6. To explore the connection between a cover of this size for the graph and a satisfying assignment of truth values for the expression.) For each clause in the expression there is a “triangle” in the graph with its edges labeled by the three literals in the clause. and they are the only edges that are not covered yet. since no sub-exponential algorithms are known — and it is conjectured that none exist.) Now edges get drawn between the variable edges and the clause triangles.
  That method was diagonalization.e. In both chapters we deal with proving problems to be diﬃcult. on the expression side. 4. Satisfiability) or solvable in polynomial time (Oddlength.4. In both settings there are transformations that turn one problem into another. The rest of the cover then falls into place without any diﬃcult choices. . the transformations take over and prove all sorts of other problems to be diﬃcult as well. But there is one huge diﬀerence. ¬x) and the (z.e. for “diﬃculty. so similar.5. the transformations merely had to be computable7 .5 The “Holy Grail”: P = NP? In order not to lose sight of the big picture over all these technical details. 7 All the examples were actually computable in polynomial time. there are problems that are known to be “easy” — solvable (Oddlength. the transformations had to be computable in polynomial time. Being free to choose either endpoint of the (x. In Chapter 1.) Choosing the ¬y over the y node in the graph corresponds to making the variable y in the expression false and satisﬁes the remaining clauses. Making a variable x true (false) in the expression translates into choosing the x (¬x) endpoint of the (x. In Chapter 1. we did have a method to prove that one of the problems (Selflooping) was indeed diﬃcult. unsolvability.” i. In the current chapter. let’s summarize the results of this chapter and contrast them with the results of Chapter 1. Back to the question of how such a solution to the vertex cover problem implies that the expression is satisﬁable. into the observation that ¬w does not occur in any clause.7 and 4. In both settings these transformations “propagate diﬃculty” — if A can easily be transformed into B and A is diﬃcult. THE “HOLY GRAIL”: P = NP? 101 only 10 of our 12 node choices so far and can choose either endpoint of those two edges.. Choosing w over ¬w initially in the graph corresponds to making the variable w in the expression true. In the current chapter. the notion of diﬃculty of a problem is that there is no program that solves the problem in polynomial time. to propagate.8 illustrate the results in both of these two settings. In both settings. Binary Addition).” i.. So far. With this starting point of one problem provably diﬃcult. (The reason why this was a no-loss choice in the graph translates. Figures 4. the lack of a polynomial-time solution. then B is diﬃcult as well. In Chapter 1. we ignored the importance of time and considered as “diﬃcult” problems that could not be solved by any program running in a ﬁnite amount of time. Binary Addition. for “diﬃculty. ¬z ) edges to complete the cover of the graph corresponds to the fact that the expression is already satisﬁed no matter what we assign to x and z . A similar discussion illustrates how a satisfying assignment to the variables of the expression translates into a cover of the right size for the graph. ¬x) variable edge of the graph. to propagate.
  x) SELFLOOPING undecidable decidable SAT ODDLENGTH BINARY ADDITION PROBLEM unsolvable problem transformation propagating unsolvability Figure 4.102 CHAPTER 4.7: Results from Chapter 1 . TIME COMPLEXITY LOOPING ON SOME INPUTS NEVER_HALTING HALTING (P.
  not solvable in polynomial time) transformation propagating probable intractibility Figure 4.4.e.5. THE “HOLY GRAIL”: P = NP? 103 NP 3-COLOR VERTEX COVER HAMILTONIAN CIRCUIT SAT P BINARY ADDITION ODDLENGTH PROBLEM probably "intractible" problem (i.8: Results from the current chapter .
  104
  CHAPTER 4. TIME COMPLEXITY
  In the current chapter, there is no such starting point. We do not have a proof that Sat, or Vertex Cover, or 3-Color, or Hamiltonian Circuit, do indeed require more than polynomial running time to be solved. So what’s missing in the current chapter is a proof technique — some way to show that, say, Sat cannot be solved in polynomial time. Such a result would show that P = NP. Conversely, ﬁnding a polynomial-time algorithm for Sat because of the NP-completeness of Sat would imply that P = NP.8 Why is P = NP So Hard The question of whether or not there are polynomial-time algorithms for Sat, to Resolve? Vertex Cover, or — equivalently — any other NP-complete problem has been the main unresolved problem in theory of computation for twenty years. While most people conjecture that there are no such polynomial-time algorithms, in other words P = NP, there is no proof. What then makes this question so hard? The sensitivity of the One reason for the diﬃculty is that there are models of computation that are question very close to the standard model — very close to the kind of programs you write all the time — in which P = NP and, in a diﬀerent model, P = NP. In other words, a slight change in our concept of what a computer is (or, because the distinction between hardware and software is of no consequence here, in the ? deﬁnition of what a program is) changes the answer to the P = NP question. Why is this a reason that resolving the P = NP question might be hard? ? Because it implies that any proof technique that can resolve the P = NP question must be sensitive to those slight changes. It must work in the standard model of computation but must not work in a closely related model. Worse yet, of the ? two obvious approaches for trying to resolve P = NP question — one approach for showing equality, the other for showing inequality — neither is sensitive to those slight changes in the model of computation. Co-processors What are these “slight changes” in the model? In terms of hardware, the change is to plug in a “co-processor” — a circuit, or chip, which takes some input and produces some output. The most common co-processor in practice may be a “numerical co-processor” — a chip that takes as input two ﬂoating-point numbers and produces, say, their product. The co-processors we are interested in here take as input one bitstring and produce one bit as output. Each such co-processor can be thought of as implementing a set of strings. For each string it responds with a “Yes” or a “No.” Figure 4.9 illustrates this. In terms of software, this amounts to stipulating that there is a new boolean function built into our programming language which takes a bitstring as argument and that we do not count the execution time of that function as part of our programs’ execution time. What happens to our computer (or programming language) once we add this new feature? Programs that do not access that co-processor (do not call that new function) run like they always did. What about programs that do
  8 Current
  ?
  ?
  betting is overwhelmingly against this resolution of the question.
  4.5. THE “HOLY GRAIL”: P = NP?
  105
  Yes / No
  A
  input string
  Figure 4.9: A co-processor
  106
  CHAPTER 4. TIME COMPLEXITY
  access the new feature? What they do depends on exactly which model of a co-processor we plugged in, i.e., which set it implements. As long as that new co-processor only does things we could have done by programming our own boolean function it is not very interesting. But what if somehow, mysteriously, we had come across a co-processor that does in one single step something that we might otherwise not be able to do at all with a program or at least not as fast? Co-processors that are We can consider co-processors of any kind of power, e.g. even a co-processor that “out of this world” solves Halting in a single leaping bound: a co-processor which, once you have fed it a bitstring that describes a program and input, responds with a “Yes,” if that program on that input would terminate, and with a “No” otherwise. Does such a co-processor exist? Deﬁnitely not on this planet, but that need not keep us from considering what such a new — very powerful — machine could do.9 Another co-processor of interest might be one that solves Satisfiability in a single step. PA and NPA For any co-processor A, let PA be the class of problems that a machine with co-processor A can solve in polynomial time. For any co-processor A that is worth talking about, PA is bigger than P. This is certainly true if A = Halting and probably true if A = Sat. Similarly, let NPA be the class of problems that a machine with co-processor A can solve in nondeterministic polynomial time. NPA can be bigger than NP. This is certainly true if A = Halting and probably true if A = Sat. Theorem 23 (Baker, Gill, Solovay 1976) such that PA = NPA . 2. There is a co-processor B such that PB = NPB . (A proof can be found in Hopcroft and Ullman. A co-processor A for which PA = NPA is the one that answers questions of the form, “Is it true that there is a u such that for all v . . . there is a z such that E (u, v, . . . y, z ) ?” where E is a boolean expression. This kind of problem — “quantiﬁed Boolean expressions” — is a generalization of Sat, which has only one “quantiﬁer.” (“There is a set of values that makes the expression true.”)) Proof techniques can be Think of the result of Baker et al as saying that in one set of circumstances “too powerful” 9 This is a form of transformation. We are asking what problems we could solve if we had
  a way to solve Halting. 10 A set of bitstrings, really. The commonly used term is actually “oracle.”
  1. There is a co-processor10 A
  itself) will do (loop or not) and based on that decision do the opposite.4. especially co-processor B from the theorem. Are we talking about some far-out or nonexisting proof techniques? Not at all. decided what its input (that is.11 Are we likely to succeed? Not if this approach would also work “on other planets. If we want to prove that Sat cannot be solved in polynomial time. does it? Does an argument like “if it was possible to decide in substantially less than 2n steps that a program was going to halt within that many steps then we could write a program that took itself as input.5. what would be a natural ﬁrst technique to try? What technique do we have to show that something cannot be done? The one and only method we have is diagonalization. but that should not worry us as we have yet to come up with any transformation that was not polynomial-time . Given a pro. will not work — once we plug in co-processor A.” i. with time playing a role. decided what its input (that is. exponential time to solve? Consider the following “exponential-time” version of Halting. we have no chance unless we come up with a proof technique that will collapse – will not be applicable. if space travel does not seem the right subject to drag into this discussion. the argument is that if it was possible to decide in substantially less than 2n steps that a program was going to halt within that many steps then we could write a program that took itself as input. Now. itself) will do (run for more than 2n steps or not) and based on that decision do the opposite. decided what its input (that is. Or. THE “HOLY GRAIL”: P = NP? 107 (“on planet A”) P = NP and in a diﬀerent set of circumstances (“on planet B ”) P = NP. in the presence of co-processors. Why do we on planet Earth care? Because if we want to prove that P = NP.Time-limited gram P and an input x. is “too powerful” to prove P = NP. we have no chance unless we come up with a proof technique that does not work on planet A. by a diagonalization proof that is practically identical to the one we used to prove that the (unlimited) halting problem was undecidable.. How could we try to use it for proving that something takes. The central argument in such a proof is just like our argument in Chapter 1 that Selflooping was not decidable.e. Similarly. itself) will do (run for more than 2n steps or not) and based on that decision do the opposite” 11 with a polynomial-time transformation. This diagonalization is clearly a decidable problem — just run the program for that long and see. any technique that is so robust that it even works in the presence of any co-processor. This is of course very exciting. say. There the argument was that if Selflooping was decidable then we could write a program that took itself as input. Any technique that is so robust that it even works in the presence of any co-processor is “too powerful” to prove P = NP. Well. Is there a faster way to decide it? No. We have a way to prove that something takes exponential time! All that’s left to do in order to prove that P = NP is to transform this Exponential-Time Halting problem to Sat. decide whether or not P halts within 2|x| steps.
  itself) will do (run for more than 2n steps or not) and based on that decision do the opposite” Thus this diagonalization technique is doomed.” If it does. actually impossible. But if we succeeded in such an eﬃcient deterministic “simulation” of nondeterminism. Whatever the reason for entertaining the conjecture that P = NP. We are not able to settle the question of whether or not P = NP and variations of existing proof techniques are not likely to help. There is nothing wrong with the same argument on machines that have a co-processor: “if it was possible on a machine with co-processor A to decide in substantially less than 2n steps that a program was going to halt within that many steps then we could write a program to run on a machine with co-processor A that took itself as input. the answer is yes. Ex. we should be aware that “extra-terrestrial” arguments similar to the ones presented above show that P = NP is not likely to be provable with “the obvious” kind of approach. just One reason why a conjecture can be hard. the technique is again “too good. of course.” Well.2. Is it true that if a circuit is satisﬁable then there is more than one way to satisfy it? Explain brieﬂy. then maybe. TIME COMPLEXITY still work when there is a co-processor present? Unfortunately. 4. It is “too good. Is it true that if a graph is 3-Colorable then there is more than one way to 3-Color it? Explain brieﬂy.108 CHAPTER 4. We have used this insight before. 4. decided what its input (that is.” Now what? That’s where the matter stands. Exercises Ex. How could we even approach this problem? By trying to devise some very clever way of searching nondeterministic execution trees of polynomial height but exponential size in (deterministic) polynomial time. What is the “obvious” kind of approach? What is the problem? The problem here would be to ﬁnd a way to solve Sat and its equivalent problems with a deterministic polynomial-time algorithm. P = NP ? the conjecture might not be true. a reasonable alternative was to consider the possibility that the language was not context-free and to try to prove that instead. If we had a hard time coming up with a context-free grammar for some language.1. We are able to build and program computers but we cannot claim to understand the nature of computation. to prove is that maybe. why should such a simulation not work just because a machine might have a co-processor? Why should this not work “on planet B . .
  6.3. If a polynomial-time algorithm exists for sat then polynomial-time algorithms exist for all problems in NP. variables or negated variables. and there is a polynomial-time transformation from A to B then B is NP-complete as well. Replace each clause (A ∨ B ∨ C ) where A. 4. If A and B are two problems in NP.5.4. A is NP-complete. 2. THE “HOLY GRAIL”: P = NP? 109 Ex. 3. 4. There is a polynomial-time transformation from 3-Sat to Vertex Cover. Consider the following attempt at transforming 3-Sat to 2-Sat.”) 1. . No polynomial-time algorithm for Vertex Cover is known. C are. Which of the following statements are true? (“Polynomial-time” is understood to mean “deterministic polynomial-time. by (A ∨ B ) ∧ (B ∨ C ) ∧ (A ∨ C ) What is wrong with this “transformation”? Ex. 5. If A is a problem that cannot be solved in polynomial time and there is a polynomial-time transformation from A to B then B cannot be solved in polynomial time either. 4. No polynomial-time algorithm for Graph 2-Coloring is known. For each of the ﬁve statements below. as usual. 3. Don’t explain your answer. If A and B are two NP-complete problems then there is a polynomial-time transformation from A to B and also a polynomial-time transformation from B to A. There is a polynomial-time transformation from Vertex Cover to 3Sat.5.4. 4. 4. B . If A and B are two problems in NP. 2. A is NP-complete. Ex. 1. 5. and there is a polynomial-time transformation from B to A then B is NP-complete as well. No polynomial-time algorithm for Graph 3-Coloring is known. Any problem A that can be transformed into Circuit Satisfiability by a polynomial-time transformation is NP-complete. state whether it is TRUE or FALSE.
  hamiltonian path is the problem of deciding. TIME COMPLEXITY 7. whether or not there exists a loop that touches each node in the graph exactly once. if there are 10 solutions to the vertex cover problem. how could you program a polynomial-time solution to the full version of the problem? (The full version of the problem means that if a satisfying assignment exists you have to ﬁnd one. (a) Apply the transformation shown in class from 3Sat to Vertex Cover to this expression: (x ∨ ¬y ∨ z ) ∧ (x ∨ y ∨ w) (b) If in a Boolean expression in 3-Cnf the same variable shows up unnegated in all clauses. Describe a transformation of 4-Sat into Vertex Cover. 1.7. Given a polynomial-time solution to the decision version of Satisfiability. not just say that one exists. 4..) Ex. if we know that B is NP-complete.6. given a Boolean function that tells you whether or not a circuit you pass to it is satisﬁable. and we know that there is a polynomial-time transformation of A into B . 4. Ex. in the transformation discussed in class)? (c) Still about the same transformation. To solve this problem. 4.9. Vertex Cover is known to be NP-complete. For reference and/or clariﬁcation note that .8. If a polynomial-time algorithm exists for Vertex Cover then polynomialtime algorithms exist for all problems in NP. whether or not there exists a path from a to b that touches each node in the graph exactly once. (You can do this by an example if you like. a new graph G such that there is a Hamiltonian Path in G from a to b if and only if G has a Hamiltonian Circuit.) . 8. If A and B are two problems in NP.. 4. Describe a transformation from hamiltonian path to hamiltonian circuit. 2. does this imply that there are 10 ways to satisfy the expression? Explain brieﬂy. Ex. 3.110 CHAPTER 4. given a graph G and two nodes a and b in G. i. hamiltonian circuit is the problem of deciding. Ex. you need to show how to construct. what property of the graph does that translate into (again. then we know that A is NP-complete as well. 9. given a graph G.e. given a graph G and two nodes a and b in G.
  New York. Dean. pages 151–158. Miller and J. Well written. Addison–Wesley. Graham. [Gur89] E. [Dea96] N.). The Essence of Discrete Mathematics. Quite formal. New York.Bibliography [BB88] G. Assoc. ACM Symposium on Theory of Computing. H. E. MIT Press / McGraw–Hill. Thatcher. W. Bratley. and O. Brassard and P.C662. Brookshear. G. Computer Science Press. Plenum Press. A. Leiserson. and R. QAA9. A standard reference. Prentice–Hall. divide and conquer. Benjamin/Cummings. for Computing Machinery. 1989. Reducibility among combinatorial problems. graph search. 1996. Karp. Gurari. The complexity of theorem-proving procedures. 1989. Eds. [GKP89] R.6. QAA 39.B73. Cormen. QAA 76. dynamic programming.G733. Rivest. [Coo71] S.6. Algorithmics: Theory and Practice. L.B76. QAA 267. Includes chapters on greedy algorithms. 1989. 1988. 1990. Knuth. 1972. Presents the standard hierarchy of languages and machines (“Chomsky’s hierarchy”) in terms of programs. pages 85–104. Concrete Mathematics: A Foundation for Computer Science. 3rd Ann. An Introduction to the Theory of Computation. Theory of Computation. Introduction to Algorithms. Patashnik. Cook. restricted in various ways. M. D.2. In Proc. C. [Bro89] [CLR90] T. 1971. M. Addision–Wesley. 111 [Kar72] . Comprehensive and readable. In Complexity of Computer Computations (R. E. QAA 76. R. J.G828.
  A Creative Approach. 1996. Discrete Algorithmic Mathematics. Wood. [Mar91] J. Harper & Row. [PS82] [Raw92] G. Papadimitriou.M394. QAA. formal. Introduction to Algorithms.D35M36. Manber. Chapter Zero: Fundamental Notions of Abstract Mathematics. 1998. Schumacher. Prentice Hall. 1987.A43R39. Compared to What? An Introduction to the Analysis of Algorithms.5. QA 267. [Sch96] C.9.L49. Englewood Cliﬀs. [MR91] S.W66. C. 2 edition. R. 1989.2. . C. Maurer and A. Papadimitriou and K. Steiglitz. 1992. 1991. Lewis and C. QAA 76. Addison–Wesley. H. Martin. 1991. QAA 267. Elements of the Theory of Computation.9.112 [LP98] BIBLIOGRAPHY H. [Man89] U. McGraw–Hill. 1982. E. Combinatorial Optimization: Algorithms and Complexity. Theory of computation. QAA 267. Addison–Wesley.S4M29. Prentice Hall. Rawlins. Standard textbook. [Woo87] D. comprehensive. QAA 39. Addison–Wesley. Introduction to Languages and the Theory of Computation. B. Ralston. H. Computer Science Press. J.