glob(i.e. **) does not work in target match for policies

[xingao267]

When I use ** in the target field under match, Forseti Config Validator errors out.

Policy:

apiVersion: constraints.gatekeeper.sh/v1alpha1
kind: GCPStorageLocationConstraintV1
metadata:
  name: allow_some_storage_location
  annotations:
    description: Checks Cloud Storage bucket locations against allowed or disallowed
      locations.
spec:
  severity: high
  match:
    target: ["**/project/*"]
  parameters:
    mode: "allowlist"
    locations:
    - asia-southeast1
    exemptions: []

Error message snippet:

failed to evaluate review: core.dependencies.audit:25: eval_builtin_error: re_match: error parsing regexp: missing argument to repetition operator: `*`

From the CV regex matching tests, it seems such feature is supported.

It's a very useful feature to help make sure policy target can always match the ancestry path no matter what composite_root_resources in the forseti server config is.

[morgante]

@briantkennedy Any idea why this isn't working?

[morgante]

More generally, it looks like wildcards inside the path are not working at all.

For example, organization/816421441114/project/gcp-foundation-shared-devops will successfully exclude a project but organization/*/project/gcp-foundation-shared-devops won't even though they're supposed to be equivalent.

[briantkennedy]

I'll take a look, this may be due to some of the work I did for constraint backward compatibility.

[briantkennedy]

I looked into this and I'm not sure why this isn't quite working. I've created a PR that adds unit tests for these specific cases and it appears to be working as intended.

@xingao267 can you provide the version of config validator you're using, a link to template and constraint that is giving the error and a copy of resources that are problematic?

[xingao267]

@briantkennedy thanks for looking into this. I'm using Forseti deployed by Forseti Terraform module version 5.2.1, which is using Forseti version 2.25.1. I don't know what CV version is used by Forseti 2.25.1.

Error occurs whenever I use a constraint like the one pasted in this issue. It also gives error like the following:

  "serverMessage": "Error running scanner: ConfigValidatorScanner: 'Traceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 176, in review\n    return self.stub.Review(review_request).violations\n  File \"/usr/local/lib/python3.6/dist-packages/grpc/_channel.py\", line 565, in __call__\n    return _end_unary_response_blocking(state, call, False, None)\n  File \"/usr/local/lib/python3.6/dist-packages/grpc/_channel.py\", line 467, in _end_unary_response_blocking\n    raise _Rendezvous(state, None, None, deadline)\ngrpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"Received RST_STREAM with error code 2\"\n\tdebug_error_string = \"{\"created\":\"@1592250631.384809354\",\"description\":\"Error received from peer ipv6:[::1]:50052\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1052,\"grpc_message\":\"Received RST_STREAM with error code 2\",\"grpc_status\":13}\"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanner.py\", line 119, in run\n    scanner.run()\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_scanner.py\", line 203, in run\n    for flattened_violations in self._retrieve_flattened_violations():\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_scanner.py\", line 183, in _retrieve_flattened_violations\n    for violations in self.validator_client.paged_review(cv_assets):\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 113, in paged_review\n    violations = self.review(paged_assets)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 49, in wrapped_f\n    return Retrying(*dargs, **dkw).call(f, *args, **kw)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 206, in call\n    return attempt.get(self._wrap_exception)\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 247, in get\n    six.reraise(self.value[0], self.value[1], self.value[2])\n  File \"/usr/local/lib/python3.6/dist-packages/six.py\", line 703, in reraise\n    raise value\n  File \"/home/ubuntu/forseti-security/.eggs/retrying-1.3.3-py3.6.egg/retrying.py\", line 200, in call\n    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n  File \"/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/scanner/scanners/config_validator_util/validator_client.py\", line 183, in review\n    raise errors.ConfigValidatorAuditError(e)\ngoogle.cloud.forseti.scanner.scanners.config_validator_util.errors.ConfigValidatorAuditError: <_Rendezvous of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"Received RST_STREAM with error code 2\"\n\tdebug_error_string = \"{\"created\":\"@1592250631.384809354\",\"description\":\"Error received from peer ipv6:[::1]:50052\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1052,\"grpc_message\":\"Received RST_STREAM with error code 2\",\"grpc_status\":13}\"\n>\n'"

I can give you access to our forseti instance. Ping me if you need it.

[briantkennedy]

Now that I'm taking another look at the error message, this is quite strange. I had to go back in the history of the repo to find core.dependencies.audit and re_match which we haven't used since the scanning was switched over to constraint framework in #82

@gkowalski-google How would we get the FCV version associated with the terraform install to confirm that they are running an older version?

[gkowalski-google]

@briantkennedy Forseti v2.25.1 is using a static binary of CV that was included in the repo; I don't know exactly what version this was but it was some time in October 2019. The main branch has been updated to running CV as a docker container using this tag.