GB2398901A - Coverage measurement of a formal verification property in which covered states cause the property to fail if the value of an observed signal is changed - Google Patents

Abstract

Measuring the coverage of a formal verification property includes the steps of receiving a logic model of the design which has a plurality of states, receiving a verified property of the design and receiving observed signals for the property. A set of covered states is then provided for the signals, such that the states are those in which changing a value of the observed signal in the state causes the property to fail. The method may provide a set of uncovered states for the observed signals. The uncovered states are such that the value of the signal has no effect on the validity of the property. The method may identify a set a reachable states of the signals and calculate a coverage percentage. The method may display a list of the states in the sets, the lists being user modifiable. The method may generate a trace of the states leading to one of the covered states or uncovered states. The trace may be provided by a breadth first reach ability analysis from the initial state.

Description

PROPERTY COVERAGE IN FORMAL VERIFICATION

Field of the Invention

The present invention is related to computer-aided design (CAD), and more particuisrly lo property coverage in formal verification of integrated circuits.

Background Information

One of the problems with modern logic design is the verification that the "esi;l acua;y works in one way it was intended to work. Undetected errors in the logic design may cause costly redesigns, or even loss of consumer confidence in the product if the product has been released on the market.

Model checking is one method of verifying designs. Model checking is a formal verification (FV) technology for property verification. A property specifies the desired values of particular circuit signals at various points in time in relation to other signals. Given a model of a design and some desired properties, a model checker like Symbolic Model Verifier (SMV) verifies whether the model satisfies all the desired properties under all possible input sequences. 1 ne properties are specified in a property specification language such as Computation Tree Logic (CTL). Although model checking is an exhaustive FV technique, a bug can escape the model checking effort if the properties specified by the user do not check for the erroneous behavior caused by the bug.

Such erroneous behavior usually occurs in some obscure corner case that has been missed by the user. This is quite common when the specification has to be manually decomposed into a set of smaller, more tractable properties that are verifiable by the moxie! checker. To - ace g escapes, She user needs to continuously strengthen existing properties and specify new properties, without knowing if the additional verification is insufficient or redundant.

Logic simulation is another method of verifying designs. In existing simulation-based verification methodologies, coverage metrics are used to improve the quality of a test suite and estimate the progress of the verification task. For example, a common coverage metric for siinuiation is "code coverage", which measures the fraction of hardware description language ("HDL") statements exercised during simulation. An "observability based code coverage" enhances this metric by factoring potential error propagation to observability points. "Transition coverage" is another metric for control state machines. Such coverage metrics are effective in reducing bug escapes by pointing out coverage holes in the test suite.

However, the existing coverage metrics for simulation do not apply directly to model checking, e.g., a naive interpretation of the code coverage or transition coverage metric on a model checking task gives a meaningless coverage of 100% for every property. Logic simulation is dynamic and its cheese s i', en A.' ill'Ut still- - alloy' vwrb, whereas mooed checking is static -' without any notion of circuit execution. Unlike logic simulation, the likelihood of having a bug escape detection in a model checking effort depends solely on I 5 the quality of the properties verified. Therefore, what is needed is a coverage metric that estimates the "completeness" (i.e. Me quality) of a set of properties against which the design has been verified.

For example, consider the CI L formula for count, a modulo-5 counter, with stall and reset as external inputs: AG[((-stallA-reset/(count = Q/(C<S))-AX(count = C + 1)] TIES GIIIIUIa SPeCifieS that i f fit e snail and reset signals are deasserted and the counter value is less than 5, then the counter increments by I in the next step.

The model checker explores the entire reachable state space to verify the i.

properly. However, in reality, it ascertains the correctness of the condition on count (that it increments correctly) only in those states that are immediate successors of states satisfying the antecedent. The actual checking of the correctness condition on the model state space is thus constraint by the CTL formula. Thus. this property cannot he said to revue 0 /o c^''e"ge. Tells example illustrates the need to define a coverage measure for formally verified properties.

For this and other reasons, there is a need for the present invention.

Summary of the Invention

One aspect of the present invention is a method of measuring coverage of a formal verification property. The method includes receiving a model of a logic design w}e.reim the model has a plurality of states. 1 ne method also incindes receiving a property verified for She 1llode' of'le logic design and receiving one or more observed signals for the property. The method further includes providing a set of covered states in which checking a value of the one or more observed signals is sufficient to determine the validity of the verified property.

A further aspect of the present invention is an alternate method of measuring coverage of a formal verification property. The method includes receiving a model of a logic design wherein the model has a plurality of states.

he method also includes receiving a property verified for the model of the logic - design and receiving one or snore observed signals for the property. The method further includes providing a set of covered states for the observed signal of the property, wherein the set of covered states comprise each one of the states in which changing a value of the observed signal in the state causes the property to fail.

Brief Description of the Drawings

Figure 1 is a block of a computer system upon which one embodiment of the present invention is implemented.

Figure 2 is a block diagram of a system for measuring coverage of a formal verification property.

Figure 3 illustrates a fragment of a state transition graph of a circuit for which property coverage is being computed.

Figure 4 illustrates a second state transition graph of a circuit for which property coverage is being computed.

Figure 5 illustrates a third state transition,rapn of a circuit for which r'-^P" )' c^,e-ge.O being CV111IJU'L.

Description of the Preferred Embodiments

In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments In lA, hich the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing Dom the scope of the present invention.

The following detailed description describes a coverage metric to identify that part of a state space which i s covered by properties verified by model checking. In each proper y, a sigma! is identified 'or a proposition On several signals) as the observed signal in that property. The coverage metric measures the coverage of a set of properties with respect to the observed signal. The coverage metric identifies the reachable states in which the value of the observed signal determines the validity of the verified properties. Then a model checking algorithm can be used to check the correctness condition on the observed signal in these "covered" states to prove or disprove the property.

1 ne detailed description is organized as follows. The first section 3 describes a computer system upon which one embodiment of the present invention is implemented. The second section presents an overview of one embodiment of the present invention. Ike third section describes property coverage in formal verification The fourth section presents an example embodiment of a method of computing a coverage metric. The fifth section provides example implementations of property coverage in formal verification.

Finally, a conclusion is provided.

Fxamplle. (2omg System Referring to FIG. 1, a computer system upon which one embodiment of the present invention is implemented is shown as 100. Computer system 100 comprises a bus or other communication means 101 for communicating information, and a processor 102 coupled with bus 101 for processing information. Computer system 100 further comprises a random access memory (RAM) or other dynamic storage device 104 (referred to as main memory), coupled to bus IQ1 for storing inflation arid instructions to be executed by processor 102 /r'n memory; 1 nA also relay be -my Historming temporary variables or other intermediate information dunug execution of property coverage instnctions by processor 102. Computer system 100 also comprises a read only memory (ROM) and/or other static storage device 106 coupled to bus I 01 for storing static information and instructions for processor 109, and a dare storage device 107 such as a magnetic disk or optical disk and its corresponding disk drive. Data storage device i07 is coupled to bus 101 for storing information and instructions for measuring coverage in formal verification properties. Computer system 100 may further be coupled to a display device 1 7!, such as z cathode ray tube CRT) or liquid crystal display (LCD) coupled to fins 1QI for displaying.nfo;rnlat:on to a computer user. An aipilanumeric input device 122, including alphanuTneric and other keys, may also be coupled to -bus 101 for communicating information and command selections to processor 102.

An additional user input device may be cursor control device 123, such as a mouse, trackball, stylus, or cursor direction keys, coupled to bus 101 for I O communicating direction information and cormnand selections to processor 102, and for controlling cursor movement on display 121. Another device which may be coupled to bus 101 is hard copy device 124 which may be used for printing instructions, data, or other information on a medium such as paper, film, or similar types of media Note, also, that any or all of the components of computer system 100 and associated hardware may be used in one embodiment, however, it can be appreciated that any type of configuration of the system may be used for various purposes as the user requires in other embodiments.

Computer-readable instructions stored on a computer-readable medium (such as RAM, ROM, or other data storage devices) are executable by the processor 102 of computer system 100. The computer-readable ir.stn. Jctions cause the computer system 100 to perform a method of measuring coverage of a formal verification property as further described below.

Overview of one embodiment of the Present Invention An overview of one embodiment of the present invention is described by reference to Fig. 2. Fig. 2 is a block diagram of a system for measuring coverage of a formal verification property. The system shown in Fig. 2 comprises inputs, processing modules, and one or more outputs.

The Unpure case a p.-vpeiy, one or more observed signals, and a circuit model or other logic design model. The property is a formal verification property specifying desired values of particular circuit signals at various points in time in relation to other signals. In one embodiment, the property is written in CTL However, those skilled in the art wil! recognize That other property specification languages may be employed without diverting from the scope of the present invention.

The observed signal is a particular one of the circuit signals specified by The prope,ty. The observed signal is the signal being checked. In one cm'nodiment, tale obse. ved signal Is provided by a user. in an alternate embodiment, the observed signal is identified by the processing modules. The circuit model is a model of an integrated circuit design for use with a model checking verification tool and for use with the system of the present invention.

The processing modules comprise program modules that perform the function of measuring the coverage of properties by a model checking verification tool or in an alternate embodiment by a simulation checker.

Generally, program modules include routines, programs, objects, components, ^ beta structures, etc., that perform particular functions or implement particular abstract data types. The processing modules provide an indication of the completeness of the verification. The coverage metric executed by the processing modules is based on states rather than paths. Example embodiments of the coverage metric are described in detail below. The term "state" as used herein means both states and inputs. State coverage is static in the sense that a state may be reached via several paths.

In one embodiment, the outputs comprise a coverage percentage indicating the percentage of states in which the observed signal has been checked for the property. Additional outputs will be readily apparent from the following

detailed description.

Property Coverage in FonnLVenfication A"formula" (also referred to as a "property") specifies the desired values of particular circuit signals at various points in time in relation to other signals.

In other words, each formica specifies a correctness condition on certam circlet s;llat, anti also specines where in une circuit state space this condition should hold. One of the signals or propositions being checked in the correctness condition is identified as the "observed signal" and coverage is defined on this observed signal. In one embodiment, a sequential circuit is viewed as a Mealy finite state ac.ne. l

DefiruJinn 1: A pirate state machine (FSM) M is a Couple < S. TM P. S! >, where S is a finite set of states, TM 5 x 5 is a transition relation of M between pairs of states in S. and P = {p/. P2, Pn, q't is a set of signals, where q is the observed signal Each signa! is a Boolean, .netivn S _ T IZ;] representing a set of states, or equivalently, each signs! cow. espo-,s to atomic proposition.

S. c S is a set of initial states.

Given a property that has been verified by a model checking algorithm to be true of the circuit, the present invention defines coverage of that property for the specified observed signal in terms of a subset cuff circuit states reachable from the initial states. A state is reachable from the initial states if there exists an t.

input sequence which takes the FSM from an initial state to that state. In one e..uvdill,e,ui, "covered set of states for an observed signal is the set of reachable states in which the values of the observed signal must be checked to prove satisfaction of the property.

Thus, states in the covered set have two characteristics First, if the value of the observed signal is changed in any state of the model outside the covered set, the property should still be satisfied. Those states are not checked for the property. Second, if the value of the observed signal is changed in any covered state, the property should fail. In this embodunent, the set of covered states is a minimal set.

To determine whether a state belongs in the covered set, the value of the observed signal in that state is modified and its effect on the validity of the property is assessed. To facilitate this test, a dual FSM is defined for each state of the given FSM M as follows: Defini0Qn 2: Given an FSM M= <S. TM, P. S] >, where P = {P/, P2 p,,, qJ, and any state s S. the dual ESM with respect to state s is the 4-tuple < 5' TM, {P] P2 - Pa. As}7 5! >, where q^,(r) = l9(( I It(t s)) for every state t S. With this definition, On embodirn.ent of a covered set of states is deemed as follows.

Definition 3: Given a propertyf and an FSM M such that M satisfiesf with respect to its initial state set Sr' denoted by M, I = f. a set C s S is a covered set offon M for observed signal g if and only if for any state s E S. the dual FSM satisfies the condition (by, Sr! of) fits Q This definition is independent of the property specification!an.gl,=e Ad guarantees that changing the value of the observed signal in an uncovered state will not cause the property to fail while changing the value in a covered state m11 cause it to fail. This set of covered states is unique. Consequently, it constitutes a necessary and sufficient set to prove satisfaction of the property.

I O Covered states are defined in this manner so that the value of the observed signal on those states is guaranteed to satisfy the correctness conditions specified by the prope,.j. Claris, -veraDe gives a measure of now much of the state space oaths - model has been checked by the verified property for the observed signa!. T.his definition does not preclude multiple observable signals in the same property.

The covered states are then simply the union of the covered states for each individual signal.

To prove by contradiction that the set of covered states is unique, assume that there are two distinct covered sets C' and C2 for a propertyf and an observed signal q. As C, C2, there must exist a state s which belongs to one but not the other, say s C, and s C2 without loss of generality. By the definition of covered set C', if use value of q is changed in state s, the propertyf fails. This implies that s should belong to all covered sets and therefore s E C2, which is a contradiction. Therefore, Cal = C2 and the set of covered states is unique.

The following example illustrates the definition of a covered state by reference to Figure 3. This example computes coverage of the following simple CTL formula with q as the observed signal: AG Opt - AX4Xq). The fonnula specifies that wheneverp; is asserted, q will be asserted two steps In the fit-e.

Figure 3 illustrates a fragment of a state It ion Inch 300 of the circuit tic.

which property coverage is being computed. Consequently, must be asserted in the marked state 302 in Figure 3 for the formula to hold. This marked state 302 is a covered state. Inspection shows that the condition specified in Definition 3 above holds for this state. Note that there are other states 304, 306 with q asserted but are not marked as covered since they are not critical to the validity of the given formula.

In an alternate embodiment, a less strict definition of a covered set is a set of states im which it is sufficient, but not necessary, to check the value of tne observed sigma! A determine tale Alit of the CTr ro,lula. covered set in this case may be neither unique nor minimal.

Definition 4: Coverage of a formula for an "observed signal " on a given model with a given set of initial states is computed as the fraction of reachable states of the model feat a. e cove.-ed: number of covered states x 100% - number of reachable states Coverage for a set of properties is simply the coverage from the union of We covered sets from each individual property.

Full or 100% coverage for a particular observed signal thus means that the value of that signal has been checked! by the verified properties on all reachable states of the circuit. This serves as a very useful indicator of the completeness of the properties and the quality of the venfication. More importantly, the formulation of the coverage metric allows the identification of areas with low coverage in terms of uncovered states so that the user can write additional properties to Increase the coverage.

In an alternate embodiment, this coverage metric is also applicable to measurement of coverage for simulation tests. In which case, a covered state is defined as estate where the simulation checker checks the value of the observed signal during simulation.

The definition of coverage presented above is applicable to any property specification language. The rerr.air,der Of this section of the detailed description ? S precepts an exm.r!- e.b^d1.ent cf. an al. to co,.ll)u.e a covered set for a subset of ACTL, the universal subset of CTL.

The subset of ACTL acceptable to us is defined as follows: f::= b I b - f l AXf l AGf | Atfu g] I J'\ g where b is a proposition! fiorrnula andfand are temlpo.-al fonnulas fin the subset. Note that AFfcan be equivalently written as A[True Uf] and does not need to be treated separately. The Only ACTL construct missing from this subset is disjunction of temporal formulas.

Applying Definition 3 to this subset of ACTL, the set of states where the value of the observed signal is czcia, to the. z'idij of the formula can 'ce computed exactly Cheer, applying Edition 3 in such a mlal-lel 1es-ul. s in some unexpected coverage. The coverage for "eventuality properties" is extremely low. For instance, consider the property ALp' U q] and the state transition graph of a circuit as shown in Figure 4. The property specifies thatp' should be him on any path Tom an initial state until observed signal q is asserted. Intuitively, it appears that the first state encountered where q is asserted should be covered (as marked in Figure 4). However, changing the value of q in this states does not cause the property to fail becausep is high in) that state. In fact, none of the states on tlds path will be considered covered by the definition. Thus, the coverage for this property will be zero. This is contrary to the expectation from such a property. To obtain a more intuitive measure of coverage, the coverage effects of the nvo parts of the Until formula need to be isolated from each other and coverage for each part computed separately. To achieve this, a transformation on ACTL fonnulas is defined below. The transformation changes the syntactic structure of the formulas but maintains semantic equivalence.

Definition 5: For an FSM M, given a formulafin the acceptable ACTL subset and an observed signal q within the formula, a signal q' is introduced.

The signal q' is defined by the same function as the observed signal q. The "observability transformation," up, is defined by substituting occurrences of q inf with q' (denoted bye I q_q') (by- - b- Ace) = 19-q it" -v} ' 'V J p(AG/3 = AG(p(0 tp(AUUg]3 = A[<pUg]AAt(fA-g) U(pg)] Shag) = tpf)A<p(g3 I' and g' me wntt.e. as the shorthand for q3(vf) and Aid) respectively. The new signal q' is now the observed signal for the transformed fonnulaf Note that the formulas after the observability transformation are equivalent to the original formulas with respect to validity of the verification.

The only two cases in which the syntactic structure of the formula change are the inn.rlicatio" and Tonsil fo.:ulas. The motivation is to pinpoint the states which contribute cc,erage Mom Me consequent part Blithe imlpl. eatiori as well as the states which independently contribute coverage from each part of the Until formula. As a result of the observability transformation, two semantically equivalent formulas with different syntax can provide different coverage. Thus, the syntax of the formula beaer captures the verification intent of the user. The application of Definition 3 to the transformed formulas gives a more intuitive and pragmatic determination of the covered set. The observability transformation is consistent with the definition of a covered state where the set of covered states is identified as a set of states in which it is sufficient to check the value of the observed signal to determine the validity of the CTL formula (coverage (:omllutation This section of the detailed description presents an example embodiment of a method of computing the coverage metric. The example embodiment is a recursive algorithm to compute the set of covered states in the state space of an FSM for a given ACTL formula and a given observed signal. The algorithm operates on the original formula but gives the computed set of covered states with respect to the transformed formula as described above. Thus, in this embodiment, the computation of coverage does not require application ofthe observability transformation. Later, it is proved that the computed covered set is the same set of states as would have been obtained by direct application of Definition 3 to the transformed formula.

Prohlenl Statement: Given an FSM M with a set of initial states S/ and an acceptable ACTL fomu'a g such Flat M, S/ = it, compute the set of covered states Did Me con_. - ,e for GSCi it' Silm y.

Coverage for a nested formula g is computed in a recursive manner on the syntactic structure of g. This algorithm is summarized by Table I below.

Coverage for each sub-formula is computed with respect to a set of start states.

The covered state set for formula with respect to start state set is denoted by C(S g). Traversing down a parse tree, the set of start states used to compute coverage for a particular sub-forrnuia changes, as shown in the table. Coverage for the top-level formula g is computed with respect to the set of initial states SO of M(substituting SO = 51 in Table l), i.e., C(S/, A).

The algoritinn guarantees that the value of file observed signal in any covered state satisfies the correctness condition specified by the forn. :la. Lea sub-formula does not involve the observed signal, its covered set will be empty.

Definitions of the functions used in the algorithm are given below.

Table l. Recursive computation of covered set C(SO, g) | Covered Set of States to C(SO, b) SO n depend(b) C(SO, b -f) C(SO n lfb),f) C(S^. AXE -,, ,, C(SO, AGf) C(reacI2able(5O), f) C(So,AGf; Uf2]) C(traverse(so,};,f2), f6)u C(rstreached(So' f 2),J2) C(So,f /\J21 C(So,];) u C(SO,f2) Given a propositional formula b, let TV) represent the set of states which satisfy b. Note that the property is satisfied by the circuit if and only if b is true in all start states. The subset of these start states which are covered is identified as those start states where the satisfaction of predicate b actually depends on the value of observed signal q 'o may also specify conditions on other signals). In other words, changing the value of observed signal q on a covered state must falsify the formula b on Fiat state when otherwise it would be true. In the above table, this set of states is given by the function depend(b) = 1(b) n lIb l q _q) The computation for fonnulas of type -of, Aye and AGfis straightforward. The formula b -fspecifies that the formulafmust be true on those start states in 50 which satisfy the predicate A. The cowed set C(SO, b - I) ofthe subformula b -fwith respect t.o the start sees SO is - . .a'c-. to, -i computed as, covered set C(SO lab),]) forf with respect to the new set of start states.0 T(b).

The formula (Yf specif es thatJholds in all successor states Dom the start state set 50. The function,forward(S) gives states reachable in exactly one step from the start states in 50. This set becomes the new start states while computing coverage forJ: forward(SO) = {is" 1 As' SO, (s', sn) T.,} AGfspecifies that Beholds Owl all states reachable from 50. The function rcachabeSOi gives states reachable from SO In any number OI Iorward steps.

This set becomes the new set of start states for computing coverage of reachablefSO) = U forward '(SO) I =o The coverage estimation for the Until operator is a little more complicated. The computation of covered states for Atf; Uf2] is explained with the help of the state transition graph in Figure 5. Sub-formulaf' is verified to be true on states along paws fiord a start state (unique in this example) in SO, such thatf, is true untilf2 first becomes true. The function traverse(SO, f', f2) identifies states along paths starting from states in SO such thatf is true andf2 is not true until, but not including, states wheref2 becomes true. These states are marked and labeled byf, in Figure 5 and become the new set of start states while computing coverage for sub-formulaf'.

traverse(SO, f, f2) = S'O u traverse(forward(S'O), f, f2) where S'O = SO 1) n rift).

In addition, the states satisfyingf2 first encountered while traversing forward from SO are considered as start states for computing coverage for propositional sub-fonnulaf2. These are marked and labeled byf2 in Figure 5, and are computed by the functionfrsreached(SO,f2).

fi rstreached(SO, f2) = (SO n off)) u f rstreachedforward(SO n 7-f2)) f The covered set of the Until formula is the uruon of the coverage from C. raverse(SO, J', f2), Hi) and" C(f rstreached(SO, f,), f,).

flee -vee" Bed of a 1ol,-r-ula wiiicn is a conjunction or two subrormuias is simply the union of the covered sets of the sub-formulas, because both sum formulas must be satisfied by the FSM for the conjuncted formula to be satisfied.

(:nrrectness Theorem: Given an PSM M with initial states SO and anacceptable ACTS formula g with observed signal A, the above aigo.t.m computes correctly the set of covered states as specified by Definition 3 for the transformed formula g' = tp (g) and observed signal q', where up is the observability transformation.

Proof: The proof is by induction on the structure of the formula. Except for the three cases involving temporal operators, all other cases cat, be easily S obtained from the definitions of coverage the obserYabi!it, ^ans. fc....a.;on ally the algorithm. The cases of AX and AG operators follow directly from the following equations: M, fonvard(SO) l=J)-Elf, SO l= (X/) ) (M5 reachable(SO) l=f) (M, SO 1= (AGE)) The most complex case is the Until operator. The two terms in the definition of the observability transformation correspond to the two sets in the cv v Cl ago Igoltlll. Siil;c raverseSO,J',f2j represents the set of states on paths starting from So that satisfy,f; and do not satisfyf2 until and not including states that satisfyf2, the covered set computed by C(traverse(SO,f;,f2),f) is the correct l S covered set for A'U/2]. Likewise, the covered set computed by C(f rstreacked(SO, f2), f2) is the correct covered set for 4[(f /\ f2)U/2].

This algorithm is of Me same order of complexity as conventional symbolic model checking algorithms. Both are based on fix point computation using Binary Decision Diagrams (BDDs) which are exponential in the worst case. Results for sub-formulas computed during verification can be memoized and used dunug coverage estimation for a more efficient implementation. In practice, coverage estimation can be slightly more expensive than the verification in some cases because it may require computing the reachable states. This involves fix point computation which may not have been necessary for the actual verification of the CTL formulas.

In the example embodiment described above, after computing the set of reachable states and the set of covered states, the coverage estimator gives the coverage nerce.ntage and, in one em^A'met, r..,,+^ out a 'is. of unco-vel--ed states. This output aids the user in writing additional properties to cover the holes. The coverage estimator also prints out traces to uncovered states by performing a breadth first reachability analysis from the initial states to an uncovered state via the shortest paw And p enerating an input sequence corresponding to this path.

Fxarnple Tmplemeat,mr of Propped (:overage in Fo,nna1 Verification The embodiments of the coverage metric described above are useful in achieving a high degree of confidence in the completeness of the verification. A coverage estimator module for meas,'m.g cc v wage of Meshed properties should live used after the normal specification and.,e. ifica;o.l cycle has been completed and the user is reasonably confident of the completeness of the verification. In one embodiment, the verification is performed by a model checking module.

The coverage estimator module then measures the coverage of the desired properties.

I O Using the metric presented here, the verification engineer is able to identify behaviors exhibited by the circuit that have not been checked by any properly. line nrsl step in this process is to inspect uncovered states provided by the coverage estimator. If it is not immediately apparent from this inspection how to strengthen the verification to cover that hole, the second step is to instruct the system of the present invention to generate traces to specific uncovered states. These traces are evidence of circuit behavior leading to uncovered states and provide strong hints as to the nature of additional properties required to achieve higher coverage. The user can then strengthen the verification either by writing additional properties or improving existing ones by weakening the antecedent or strengthening the consequent. In one embodiment. the minimum coverage requirement recommended is 100% coverage for each primary output signal and each significant circuit signal.

A large fraction of the set of states not covered by the properties could be states on which the value of the observed signal is irrelevant to the correctness of the circuit. In one embodiment, these "don't care" states are supplied by the user as a set of propositions on state variables and excluded from the coverage space so as to give a more realistic coverage estimate.

Fairness conditions expressed in are -.oel chcc..s sys.el1 -nsllaiil ale system to only look at fair paths during the verification of a property, i.e., paths where the fairness constraints are true infinitely often. The presence of fairness constraints therefore requires the coverage estimation algorithm also to ignore states not falling on fair paths. to one embodiment. coverage is computed as the fraction of states reachable along fair paths.

A coverage estimator module according to the present invention has been implemented in conjunction with one version of SMV and applied to circuits from a microprocessor design. A signal from each circuit was selected as the observed signal and applied the estimator to deterTrine the coYera,e of properties which had been verified to check behavior of the signal. Table 2 below gives the names of the observed signals for which coverage was measured the number of properties verified for that signal, the coverage obtained for the given set of properties, the perform,ar.ce of model checking measured in terms of the number of BDO nodes and the run tune in seconds on a - 9000 workst=:on, and the runtime performance of the coverage estimator.

l auZe I. Coverage resuis | | Verification | Coverage Signal | # Prop | %COV | BDDs - time | BDD - time 15Circuit 1 (priority buffer) hi-pn 1 5 1 100.00 1 124k-59.28s 1 150k- 60. 41s lo-pri 1 5 1 99.98 1 155k - 61.37s 1 178k - 71.26s Circuit 2 (circular queue) wrap 5 60.08 26k- 8.3s 26k 7.46s full 2 100.00 21k- 1. 55s 21k- 1.52s empty 2 100.00 13k- 1.5's Ink1.55s Circuit 3 (pipeline) output 1 8 1 7436 1 lOk- 3.58s 1 lOk- 7.42s Circuit 1 is a priority buffer which schedules and stores incoming entries according to their priorities (high or low). The model had 24 variables. Given the number of entries already in the buffer and the number of incoming entries, the properties specify the correct nr.ber of ebonies;r. the buffer at the next Cock. For exmr'e, i,,th" 'fTer -awe i'j' - Yes r:..l,,Dlg e.-luies and + B is less than the size of buffer, then the buffer in the next clock should have + B entries. High and low priority entries are checked by different properties, and their counts are considered as the observed sisals. The set of verified properties should provide a complete analysis of all possible cases but a missing case was uncovered when the buffer is empty and low priority entries are incoming, the entries should be stored. A sunpie additional property was written to cover this case. Verification of this property failed and actually revealed a bug in the design of the buffer.

Circuit is a circular queue con"Glled by a read pointer, a write pointer and a wrap bit that toggles uh.e.,er e ther poin.e, -wraps around ule queue. it also has stall,- clear and reset signals as inputs. Properties were written to verify the correct operation of the wrap bit, the full and empty signals. The model had SO vanables. The coverage for the full and empty signals was 100%. But coverage for the wrap bit was 50%. Inspecting the uncovered states, three additional properties were written which still did not achieve 100% coverage.

The input/state sequences leading to these uncovered states was traced and found inane me value of wrap bit was not checked if the stall signal was asserted when the write pointer wraps around. Such a subtle corner case can easily be missed durung property specification. A property was added to specify that the wrap bit remains unchanged for this case and 100% coverage was achieved.

Circuit 3 is a pipeline in the instruction decode stage of the processor.

The width of the pipeline datapath was abstracted to a single bit Properties were verified on this signal to check the correct staging of data through the pipeline, rather than the actual data transformations. These properties generally took the form that an input to the pipeline will eventually appear at the output given certain fairness conditions on the stalls. The final model had 15 variables.

Coverage was increased to 100% by identifying uncovered states and Prancing the set of properties.

These examples demonstrate that coverage estimation can improve the quality of FV. Ike runtimes and memory requirements are similar to those required by the actual verification. Purtherrnore, the example are a good representation of ccr.'r.or. FV p.openies: the buyers involved syntacicaily simple properties, e.g., ACRID! TV ' VIJ2J "L.d tlIe ppelifle required evemuaiiry properties using the Until operator in a nested manner, e.g., AG(D' ALP7UALP3UP4]]) C:onclusion The detailed description has described a coverage metric for model- checking based verification methodologies. An example embodiment of a coverage metric for model checking that is applicable to a significant subset of CTL has also been described. The coverage metric identifies pronP-!tY coverage holes that can lead to the discovery of bugs that escaped the model checking process.

41ougn specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that arm.

arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiment shown. This application is intended to cover any a'1^ticn" or 'a.io.= oruhe present in-v-eilucn. 1nerelore, it is intended that this invention be limited only by the claims and the equivalents thereof.

Claims (12)

19 5411610 CLAIMS

1. A method of measuring coverage of a formal verification property, the method comprising: receiving a model of a logic design wherein the model has a plurality of states; receiving a property verified for the model of the logic design; receiving one or more observed signals for the property; and providing a set of covered states for the one or more observed signals of the property, wherein the set of covered states comprise each one of the states in which changing a value of the observed signal in the state causes the property to fail.

2. The method of claim 1, further comprising providing a set of uncovered states for the one or more observed signals of the property, wherein the set of uncovered states comprise each one of the states in which the value of the observed signal has no effect on the validity of the property.

3. The method of claim 1, further comprising: identifying a set of reachable states for the one 5411610 acre A^C' era TOG C:; - 1 1 if t he Caret T. And _ _ _ _ _ _ _ _,, calculating a coverage percentage wherein one coverage percentage is a fraction of the set of reachable states.

J

4. The method of claim 1, further comprising displaying a list of the states comprising the set of covered states.

5. The method of claim 4, wherein the list of the states comprising the set of covered states can be modified by a user as desired.

6. The method of claim 2, further comprising displaying a list of the states comprising the set of uncovered states.

7. The method of claim 6, wherein the list of the states comprising the set of uncovered states can be modified by a user as desired.

8. The method of claim 1, further comprising generating a trace of the states leading to one of the covered states.

21 5411610

9. The method of claim 2, further comprising generating a trace of the states leading to one of the uncovered states.

30. The method of claim 9, wherein providing the trace further comprises performing a breadth first reach ability analysis from one or more initial states to the uncovered state via a shortest path and generating an input sequence corresponding to the shortest path.

11. An article comprising: a computer-readable medium including instructions that when executed cause a computer to receive a model of a logic design wherein the model has a plurality of states; receive a property verified for the model of the logic design; receive one or more observed signals for the property; and provide a set of covered states for the one or more observed signals of the property, wherein the set of covered States comprise each one of the states in which changing a value of the one or more observed signals in the state causes the property to fail.

22 5411610

12. A.mcthod GO meas-urirlg covc--agc o, a formal verification property, the method comprising: receiving a model of a logic design wherein the model has a plurality of states; receiving a property verified for the model of the logic design; identifying one or more observed signals for the property; and providing a set of covered states for the one or more observed signals of the property, wherein the set of covered states comprise each one of the states in which changing the value of the one or more observed signals in the state causes the property to fail.