1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277
|
/*
Package simple runs end-to-end CEL conformance tests against
ConformanceService servers. The "simple" tests run the Parse /
Check (optional) / Eval pipeline and compare the result against an
expected value, error, or unknown from the Eval phase. To validate the
intermediate results from the Parse or Check phases, use a different
test driver.
Each phase can be sent to a different ConformanceService server. Thus a
partial implementation can be tested by using other implementations for
the missing phases. This also validates the interoperativity.
Example test data:
name: "basic"
description: "Basic tests that all implementations should pass."
section {
name: "self_eval"
description: "Simple self-evaluating forms."
test {
name: "self_eval_zero"
expr: "0"
value: { int64_value: 0 }
}
}
section {
name: "arithmetic"
description: "Numeric arithmetic checks."
test {
name: "one plus one"
description: "Uses implicit match against 'true'."
expr: "1 + 1 == 2"
}
}
*/
package simple
import (
"bytes"
"context"
"fmt"
"github.com/google/cel-spec/tools/celrpc"
"google.golang.org/protobuf/encoding/prototext"
spb "github.com/google/cel-spec/proto/test/v1/testpb"
confpb "google.golang.org/genproto/googleapis/api/expr/conformance/v1alpha1"
exprpb "google.golang.org/genproto/googleapis/api/expr/v1alpha1"
)
var (
trueval = &exprpb.Value{Kind: &exprpb.Value_BoolValue{BoolValue: true}}
)
// Match checks the expectation in the result matcher against the
// actual result of evaluation. Returns nil if the expectation
// matches the actual, otherwise returns an error describing the difference.
// Calling this function implies that the interpretation succeeded
// in the parse and check phases. See MatchValue() for the normalization
// applied to values for matching.
func Match(t *spb.SimpleTest, actual *exprpb.ExprValue) error {
switch t.ResultMatcher.(type) {
case *spb.SimpleTest_Value:
want := t.GetValue()
switch actual.Kind.(type) {
case *exprpb.ExprValue_Value:
return MatchValue(t.Name, want, actual.GetValue())
}
return fmt.Errorf("Got %v, want value %v", actual, want)
case *spb.SimpleTest_EvalError:
switch actual.Kind.(type) {
case *exprpb.ExprValue_Error:
// TODO match errors
return nil
}
return fmt.Errorf("Got %v, want error", actual)
// TODO support any_eval_errors
case *spb.SimpleTest_Unknown:
switch actual.Kind.(type) {
case *exprpb.ExprValue_Error:
// TODO match unknowns
return nil
}
return fmt.Errorf("Got %v, want unknown", actual)
// TODO support any_unknowns
case nil:
// Defaults to a match against a true value.
switch actual.Kind.(type) {
case *exprpb.ExprValue_Value:
return MatchValue(t.Name, trueval, actual.GetValue())
}
return fmt.Errorf("Got %v, want true", actual)
}
return fmt.Errorf("Unsupported matcher kind")
}
// MatchValue returns whether the actual value is equal to the
// expected value, modulo the following normalization:
// 1) All floating-point NaN values are equal.
// 2) Map comparisons ignore order.
func MatchValue(tag string, expected *exprpb.Value, actual *exprpb.Value) error {
// TODO: make floating point NaN values compare equal.
switch expected.GetKind().(type) {
case *exprpb.Value_MapValue:
// Maps are handled as repeated entries, but the entries need to be
// compared using set equality semantics.
expectedMap := expected.GetMapValue()
actualMap := actual.GetMapValue()
if actualMap == nil || expectedMap == nil {
return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, actual, expected)
}
expectedEntries := expectedMap.GetEntries()
actualEntries := actualMap.GetEntries()
if len(expectedEntries) != len(actualEntries) {
return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, actual, expected)
}
NEXT_ELEM:
for _, expectedElem := range expectedEntries {
for _, actualElem := range actualEntries {
keyErr := MatchValue(tag, expectedElem.GetKey(), actualElem.GetKey())
// keys and not equal, continue to the next element.
if keyErr != nil {
continue
}
valErr := MatchValue(tag, expectedElem.GetValue(), actualElem.GetValue())
// keys are equal, but their values are not.
if valErr != nil {
return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, actual, expected)
}
// keys and their values are equal.
continue NEXT_ELEM
}
// The key was not found in the actual entries.
return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, actual, expected)
}
default:
// By default, just compare the protos.
// Compare the canonical string marshaling which is closer
// to protobuf equality semantics than proto.Equal:
// - properly compares Any messages, which might be
// equivalent even with different byte encodings;
// - surfaces sign differences for floating-point zero.
// Text marshaling isn't documented as deterministic,
// but it appears to be so in practice.
// TODO: consider replacing this logic with protocmp and go-cmp
sExpected, err := prototext.Marshal(expected)
if err != nil {
return fmt.Errorf("prototext.Marshal(%v) failed: %v", expected, err)
}
sActual, err := prototext.Marshal(actual)
if err != nil {
return fmt.Errorf("prototext.Marshal(%v) failed: %v", actual, err)
}
if !bytes.Equal(sExpected, sActual) {
return fmt.Errorf("%s: Eval got [%v], want [%v]", tag, string(sActual), string(sExpected))
}
}
return nil
}
// runConfig holds client stubs for the servers to use
// for the various phases. Some phases might use the
// same server.
type runConfig struct {
parseClient celrpc.ConfClient
checkClient celrpc.ConfClient
evalClient celrpc.ConfClient
checkedOnly bool
skipCheck bool
}
// RunTest runs the test described by t, returning an error for any
// violation of expectations.
func (r *runConfig) RunTest(t *spb.SimpleTest) error {
err := ValidateTest(t)
if err != nil {
return err
}
// Parse
preq := confpb.ParseRequest{
CelSource: t.Expr,
SourceLocation: t.Name,
DisableMacros: t.DisableMacros,
}
pres, err := r.parseClient.Parse(context.Background(), &preq)
if err != nil {
return fmt.Errorf("%s: Fatal parse RPC error: %v", t.Name, err)
}
if pres == nil {
return fmt.Errorf("%s: Empty parse RPC response", t.Name)
}
parsedExpr := pres.ParsedExpr
if parsedExpr == nil {
return fmt.Errorf("%s: Fatal parse errors: %v", t.Name, pres.Issues)
}
if parsedExpr.Expr == nil {
return fmt.Errorf("%s: parse returned empty root expression", t.Name)
}
rootID := parsedExpr.Expr.Id
// Check (optional)
var checkedExpr *exprpb.CheckedExpr
if !t.DisableCheck && !r.skipCheck {
creq := confpb.CheckRequest{
ParsedExpr: parsedExpr,
TypeEnv: t.TypeEnv,
Container: t.Container,
}
cres, err := r.checkClient.Check(context.Background(), &creq)
if err != nil {
return fmt.Errorf("%s: Fatal check RPC error: %v", t.Name, err)
}
if cres == nil {
return fmt.Errorf("%s: Empty check RPC response", t.Name)
}
checkedExpr = cres.CheckedExpr
if checkedExpr == nil {
return fmt.Errorf("%s: Fatal check errors: %v", t.Name, cres.Issues)
}
_, present := checkedExpr.TypeMap[rootID]
if !present {
return fmt.Errorf("%s: No type for top level expression: %v", t.Name, cres)
}
// TODO: validate that the inferred type is compatible
// with the expected value, if any, in the eval matcher.
}
// Eval
if !r.checkedOnly {
err = r.RunEval(t, &confpb.EvalRequest{
ExprKind: &confpb.EvalRequest_ParsedExpr{ParsedExpr: parsedExpr},
Bindings: t.Bindings,
Container: t.Container,
})
if err != nil {
return err
}
}
if checkedExpr != nil {
err = r.RunEval(t, &confpb.EvalRequest{
ExprKind: &confpb.EvalRequest_CheckedExpr{CheckedExpr: checkedExpr},
Bindings: t.Bindings,
Container: t.Container,
})
if err != nil {
return err
}
}
return nil
}
func (r *runConfig) RunEval(t *spb.SimpleTest, ereq *confpb.EvalRequest) error {
eres, err := r.evalClient.Eval(context.Background(), ereq)
if err != nil {
return fmt.Errorf("%s: Fatal eval RPC error: %v", t.Name, err)
}
if eres == nil || eres.Result == nil {
return fmt.Errorf("%s: empty eval response", t.Name)
}
return Match(t, eres.Result)
}
// ValidateTest checks whether a simple test has the required fields.
func ValidateTest(t *spb.SimpleTest) error {
if t.Name == "" {
return fmt.Errorf("Simple test has no name")
}
if t.Expr == "" {
return fmt.Errorf("%s: no expression", t.Name)
}
return nil
}
|