Test Package

Ensure that task producing the IMAGES_PROCESSED result contains the digests of the built image.

Solution: Found an image not processed by a task. Make sure that the task processes and includes the image digest of all images in the IMAGES_PROCESSED result.

Produce a warning if any informative tests have their result set to "FAILED". The result type is configurable by the "failed_tests_results" key, and the list of informative tests is configurable by the "informative_tests" key in the rule data.

Solution: There is a test that failed. Make sure that any task in the build pipeline with a result named 'TEST_OUTPUT' does not fail. More information about the test should be available in the logs for the build Pipeline.

Produce a violation if any tests have their result set to "ERROR". The result type is configurable by the "erred_tests_results" key in the rule data.

Solution: There is a test that erred. Make sure that any task in the build pipeline with a result named 'TEST_OUTPUT' does not err. More information about the test should be available in the logs for the build Pipeline.

Produce a violation if any non-informative tests have their result set to "FAILED". The result type is configurable by the "failed_tests_results" key, and the list of informative tests is configurable by the "informative_tests" key in the rule data.

Solution: There is a test that failed. Make sure that any task in the build pipeline with a result named 'TEST_OUTPUT' does not fail. More information about the test should be available in the logs for the build Pipeline.

Produce a warning if any tests have their result set to "WARNING". The result type is configurable by the "warned_tests_results" key in the rule data.

Solution: There is a task with result 'TEST_OUTPUT' that returned a result of 'WARNING'. You can find which test resulted in 'WARNING' by examining the 'result' key in the 'TEST_OUTPUT'. More information about the test should be available in the logs for the build Pipeline.

Produce a violation if any tests have their result set to "SKIPPED". A skipped result means a pre-requirement for executing the test was not met, e.g. a license key for executing a scanner was not provided. The result type is configurable by the "skipped_tests_results" key in the rule data.

Solution: There is a test that was skipped. Make sure that each task with a result named 'TEST_OUTPUT' was not skipped. You can find which test was skipped by examining the 'result' key in the 'TEST_OUTPUT'. More information about the test should be available in the logs for the build Pipeline.

Ensure all test data result values are in the set of known/supported result values.

Solution: The test results should be of a known value. Values can be set as a data source.

Confirm the expected rule data keys have been provided in the expected format. The keys are supported_tests_results, failed_tests_results, informative_tests, erred_tests_results, skipped_tests_results, and warned_tests_results.

Solution: If provided, ensure the rule data is in the expected format.

Ensure that at least one of the tasks in the pipeline includes a TEST_OUTPUT task result, which is where Conforma expects to find test result data.

Solution: Confirm at least one task in the build pipeline contains a result named TEST_OUTPUT.

Each test result is expected to have a results key. Verify that the results key is present in all of the TEST_OUTPUT task results.

Solution: There was at least one result named TEST_OUTPUT found, but it did not contain a key named 'result'. For a TEST_OUTPUT result to be valid, this key must exist.