diff --git a/README.md b/README.md index 4cb720a..b3d7ecd 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ # vet +## Automate Open Source Package Vetting in CI/CD `vet` is a tool for identifying risks in open source software supply chain. It helps engineering and security teams to identify potential issues in their open @@ -10,7 +11,7 @@ source dependencies and evaluate them against organizational policies. ## Demo -[![asciicast](https://asciinema.org/a/I60aD2VtVsETQtIFsYTCewJZ3.svg)](https://asciinema.org/a/I60aD2VtVsETQtIFsYTCewJZ3) +![vet Demo](docs/images/vet-demo.gif) ## TL;DR @@ -40,13 +41,13 @@ Alternatively, look at [Releases](https://github.com/safedep/vet/releases) for a pre-built binary for your platform. [SLSA Provenance](https://slsa.dev/provenance/v0.1) is published along with each binary release. -Get a trial API key for [Insights API](https://safedep.io/docs/concepts/raya-data-platform-overview) access +Get an API key for [Insights API](https://safedep.io/docs/concepts/raya-data-platform-overview) access ```bash vet auth trial --email john.doe@example.com ``` -> A time limited trial API key will be sent over email. +> A time limited API key will be sent over email. Configure `vet` to use API Key to access [Insights API](https://safedep.io/docs/concepts/raya-data-platform-overview) @@ -54,7 +55,7 @@ Configure `vet` to use API Key to access [Insights API](https://safedep.io/docs/ vet auth configure ``` -> Insights API is used to enrich OSS packages with meta-data for rich query and policy +> Insights API is used to enrich OSS packages with metadata for rich query and policy > decisions. Alternatively, the API key can be passed through environment > variable `VET_API_KEY` @@ -90,7 +91,59 @@ The default scan uses an opinionated [Summary Reporter](#) which presents a consolidated summary of findings. Thats NOT about it. Read more for expression based filtering and policy evaluation. -## Filtering +## Policy Control + +Policies are written using a DSL. A group of policies can be applied using +`vet` to build a security gate in CI/CD. + +Start by copying a sample policy + +```bash +curl -LO https://raw.githubusercontent.com/safedep/vet/main/samples/filter-suites/fs-generic.yml +``` + +Run a scan with policies and configure the scanner to fail in case of policy +violation + +```bash +vet scan -D /path/to/dir --filter-suite fs-generic.yml --filter-fail +``` + +Read more about underlying capability using which policy control is implemented +in [filtering guide](docs/filtering.md) + +## Exceptions Management + +Projects may have legacy libraries that will fail any reasonable security policy. +Legacy libraries can be added as time bounded exceptions to the policies to place +strict control on any new library while legacy library can be upgraded over +time. + +Exception rules can be generated using the `query` workflow to temporarily +ignore (or snooze) existing issues when using `vet` for the first time. This +helps in establishing security gating to prevent introduction of new security +issues while existing issues are being remediated. + +Use exception rules during scan to ignore specific packages + +```bash +vet scan -D /path/to/repo -e /path/to/exceptions.yml +``` + +For more information on generating exceptions, +refer to [exceptions guide](docs/exceptions.md) + +The generated exceptions file, when combined with policy control, can be used +to setup a security gate to prevent introducing new issues while ignoring the +existing backlog for a period of time. + +```bash +vet scan -D /path/to/dir \ + --filter-suite fs-generic.yml --filter-fail + -e /path/to/exceptions.yml +``` + +## Exploring OSS Risks using Filters Find dependencies that seems not very popular @@ -109,37 +162,10 @@ vet scan --lockfiles /path/to/pom.xml --report-summary=false \ > Use filtering along with `query` command for offline slicing and dicing of > enriched package manifests. Read [filtering guide](docs/filtering.md) - Learn more about [filtering with vet](docs/filtering.md). Look at [filter input spec](api/filter_input_spec.proto) on attributes available to the filter expression. -### Using Filter Suite - -Filter suites can be used to implement security gating in CI. [Example](samples/filter-suites/fs-generic.yml) -file suite contains rules to enforce generic OSS consumption best practices. - -```bash -vet scan -D /path/to/dir --filter-suite /path/to/suite.yml --filter-fail -``` - -Read more about filter suites in [filtering guide](docs/filtering.md) - -## Exceptions Management - -Exception rules can be generated using the `query` workflow to temporarily -ignore (or snooze) existing issues when using `vet` for the first time. This -helps in establishing security gating to prevent introduction of new security -issues while existing issues are being remediated. - -Use exception rules during scan to ignore specific packages - -```bash -vet scan -D /path/to/repo -e /path/to/exceptions.yml -``` - -For more information, refer to [exceptions guide](docs/exceptions.md) - ## FAQ ### How do I disable the stupid banner? diff --git a/api/filter_input_spec.proto b/api/filter_input_spec.proto index 7184a47..1cc150a 100644 --- a/api/filter_input_spec.proto +++ b/api/filter_input_spec.proto @@ -20,6 +20,7 @@ message Vulnerabilities { // OpenSSF Scorecard message Scorecard { map scores = 1; + float score = 2; } enum ProjectType { diff --git a/docs/filtering.md b/docs/filtering.md index 3daabfe..ee7ee0c 100644 --- a/docs/filtering.md +++ b/docs/filtering.md @@ -4,7 +4,7 @@ Filter command helps solve the problem of visibility for OSS dependencies in an application. To support various requirements, we adopt a generic [expressions language](https://github.com/google/cel-spec) for flexible filtering. -Example: +## Example ```bash vet scan -D /path/to/repo \ @@ -14,6 +14,20 @@ vet scan -D /path/to/repo \ The scan will list only packages that use the `MIT` license. +Find dependencies that seems not very popular + +```bash +vet scan --lockfiles /path/to/pom.xml --report-summary=false \ + --filter='projects.exists(x, x.stars < 10)' +``` + +Find dependencies with a critical vulnerability + +```bash +vet scan --lockfiles /path/to/pom.xml --report-summary=false \ + --filter='vulns.critical.exists_one(x, true)' +``` + ## Input Filter expressions work on packages (aka. dependencies) and evaluates to diff --git a/docs/images/vet-demo.gif b/docs/images/vet-demo.gif new file mode 100644 index 0000000..4971373 Binary files /dev/null and b/docs/images/vet-demo.gif differ diff --git a/gen/filterinput/filter_input_spec.pb.go b/gen/filterinput/filter_input_spec.pb.go index 7419d3c..8bd16bb 100644 --- a/gen/filterinput/filter_input_spec.pb.go +++ b/gen/filterinput/filter_input_spec.pb.go @@ -208,6 +208,7 @@ type Scorecard struct { unknownFields protoimpl.UnknownFields Scores map[string]float32 `protobuf:"bytes,1,rep,name=scores,proto3" json:"scores,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"fixed32,2,opt,name=value,proto3"` + Score float32 `protobuf:"fixed32,2,opt,name=score,proto3" json:"score,omitempty"` } func (x *Scorecard) Reset() { @@ -249,6 +250,13 @@ func (x *Scorecard) GetScores() map[string]float32 { return nil } +func (x *Scorecard) GetScore() float32 { + if x != nil { + return x.Score + } + return 0 +} + type ProjectInfo struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -490,48 +498,49 @@ var file_filter_input_spec_proto_rawDesc = []byte{ 0x28, 0x0b, 0x32, 0x0e, 0x2e, 0x56, 0x75, 0x6c, 0x6e, 0x65, 0x72, 0x61, 0x62, 0x69, 0x6c, 0x69, 0x74, 0x79, 0x52, 0x06, 0x6d, 0x65, 0x64, 0x69, 0x75, 0x6d, 0x12, 0x20, 0x0a, 0x03, 0x6c, 0x6f, 0x77, 0x18, 0x05, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x0e, 0x2e, 0x56, 0x75, 0x6c, 0x6e, 0x65, 0x72, - 0x61, 0x62, 0x69, 0x6c, 0x69, 0x74, 0x79, 0x52, 0x03, 0x6c, 0x6f, 0x77, 0x22, 0x76, 0x0a, 0x09, - 0x53, 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x12, 0x2e, 0x0a, 0x06, 0x73, 0x63, 0x6f, - 0x72, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x53, 0x63, 0x6f, 0x72, - 0x65, 0x63, 0x61, 0x72, 0x64, 0x2e, 0x53, 0x63, 0x6f, 0x72, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, - 0x79, 0x52, 0x06, 0x73, 0x63, 0x6f, 0x72, 0x65, 0x73, 0x1a, 0x39, 0x0a, 0x0b, 0x53, 0x63, 0x6f, - 0x72, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, - 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x02, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, - 0x3a, 0x02, 0x38, 0x01, 0x22, 0x87, 0x01, 0x0a, 0x0b, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, - 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x20, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x0c, 0x2e, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, - 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x73, 0x74, - 0x61, 0x72, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x73, 0x74, 0x61, 0x72, 0x73, - 0x12, 0x14, 0x0a, 0x05, 0x66, 0x6f, 0x72, 0x6b, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, - 0x05, 0x66, 0x6f, 0x72, 0x6b, 0x73, 0x12, 0x16, 0x0a, 0x06, 0x69, 0x73, 0x73, 0x75, 0x65, 0x73, - 0x18, 0x05, 0x20, 0x01, 0x28, 0x05, 0x52, 0x06, 0x69, 0x73, 0x73, 0x75, 0x65, 0x73, 0x22, 0x5c, - 0x0a, 0x0e, 0x50, 0x61, 0x63, 0x6b, 0x61, 0x67, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, - 0x12, 0x1c, 0x0a, 0x09, 0x65, 0x63, 0x6f, 0x73, 0x79, 0x73, 0x74, 0x65, 0x6d, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x09, 0x65, 0x63, 0x6f, 0x73, 0x79, 0x73, 0x74, 0x65, 0x6d, 0x12, 0x12, - 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, - 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0xc8, 0x01, 0x0a, - 0x0b, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x49, 0x6e, 0x70, 0x75, 0x74, 0x12, 0x21, 0x0a, 0x03, - 0x70, 0x6b, 0x67, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x0f, 0x2e, 0x50, 0x61, 0x63, 0x6b, - 0x61, 0x67, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x03, 0x70, 0x6b, 0x67, 0x12, - 0x26, 0x0a, 0x05, 0x76, 0x75, 0x6c, 0x6e, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x10, - 0x2e, 0x56, 0x75, 0x6c, 0x6e, 0x65, 0x72, 0x61, 0x62, 0x69, 0x6c, 0x69, 0x74, 0x69, 0x65, 0x73, - 0x52, 0x05, 0x76, 0x75, 0x6c, 0x6e, 0x73, 0x12, 0x28, 0x0a, 0x09, 0x73, 0x63, 0x6f, 0x72, 0x65, - 0x63, 0x61, 0x72, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x0a, 0x2e, 0x53, 0x63, 0x6f, - 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x52, 0x09, 0x73, 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, - 0x64, 0x12, 0x28, 0x0a, 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x18, 0x04, 0x20, - 0x03, 0x28, 0x0b, 0x32, 0x0c, 0x2e, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x49, 0x6e, 0x66, - 0x6f, 0x52, 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x12, 0x1a, 0x0a, 0x08, 0x6c, - 0x69, 0x63, 0x65, 0x6e, 0x73, 0x65, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x09, 0x52, 0x08, 0x6c, - 0x69, 0x63, 0x65, 0x6e, 0x73, 0x65, 0x73, 0x2a, 0x26, 0x0a, 0x0b, 0x50, 0x72, 0x6f, 0x6a, 0x65, - 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, - 0x4e, 0x10, 0x00, 0x12, 0x0a, 0x0a, 0x06, 0x47, 0x49, 0x54, 0x48, 0x55, 0x42, 0x10, 0x01, 0x42, - 0x28, 0x5a, 0x26, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x73, 0x61, - 0x66, 0x65, 0x64, 0x65, 0x70, 0x2f, 0x76, 0x65, 0x74, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x66, 0x69, - 0x6c, 0x74, 0x65, 0x72, 0x69, 0x6e, 0x70, 0x75, 0x74, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, - 0x33, + 0x61, 0x62, 0x69, 0x6c, 0x69, 0x74, 0x79, 0x52, 0x03, 0x6c, 0x6f, 0x77, 0x22, 0x8c, 0x01, 0x0a, + 0x09, 0x53, 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x12, 0x2e, 0x0a, 0x06, 0x73, 0x63, + 0x6f, 0x72, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x53, 0x63, 0x6f, + 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x2e, 0x53, 0x63, 0x6f, 0x72, 0x65, 0x73, 0x45, 0x6e, 0x74, + 0x72, 0x79, 0x52, 0x06, 0x73, 0x63, 0x6f, 0x72, 0x65, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x73, 0x63, + 0x6f, 0x72, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x02, 0x52, 0x05, 0x73, 0x63, 0x6f, 0x72, 0x65, + 0x1a, 0x39, 0x0a, 0x0b, 0x53, 0x63, 0x6f, 0x72, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, + 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, + 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x02, + 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x87, 0x01, 0x0a, 0x0b, + 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x6e, + 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, + 0x20, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x0c, 0x2e, + 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, + 0x65, 0x12, 0x14, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x72, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, + 0x52, 0x05, 0x73, 0x74, 0x61, 0x72, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x66, 0x6f, 0x72, 0x6b, 0x73, + 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x66, 0x6f, 0x72, 0x6b, 0x73, 0x12, 0x16, 0x0a, + 0x06, 0x69, 0x73, 0x73, 0x75, 0x65, 0x73, 0x18, 0x05, 0x20, 0x01, 0x28, 0x05, 0x52, 0x06, 0x69, + 0x73, 0x73, 0x75, 0x65, 0x73, 0x22, 0x5c, 0x0a, 0x0e, 0x50, 0x61, 0x63, 0x6b, 0x61, 0x67, 0x65, + 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x1c, 0x0a, 0x09, 0x65, 0x63, 0x6f, 0x73, 0x79, + 0x73, 0x74, 0x65, 0x6d, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x65, 0x63, 0x6f, 0x73, + 0x79, 0x73, 0x74, 0x65, 0x6d, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, + 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, + 0x69, 0x6f, 0x6e, 0x22, 0xc8, 0x01, 0x0a, 0x0b, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x49, 0x6e, + 0x70, 0x75, 0x74, 0x12, 0x21, 0x0a, 0x03, 0x70, 0x6b, 0x67, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x0f, 0x2e, 0x50, 0x61, 0x63, 0x6b, 0x61, 0x67, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, + 0x6e, 0x52, 0x03, 0x70, 0x6b, 0x67, 0x12, 0x26, 0x0a, 0x05, 0x76, 0x75, 0x6c, 0x6e, 0x73, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x10, 0x2e, 0x56, 0x75, 0x6c, 0x6e, 0x65, 0x72, 0x61, 0x62, + 0x69, 0x6c, 0x69, 0x74, 0x69, 0x65, 0x73, 0x52, 0x05, 0x76, 0x75, 0x6c, 0x6e, 0x73, 0x12, 0x28, + 0x0a, 0x09, 0x73, 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x0a, 0x2e, 0x53, 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x52, 0x09, 0x73, + 0x63, 0x6f, 0x72, 0x65, 0x63, 0x61, 0x72, 0x64, 0x12, 0x28, 0x0a, 0x08, 0x70, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x0c, 0x2e, 0x50, 0x72, 0x6f, + 0x6a, 0x65, 0x63, 0x74, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, + 0x74, 0x73, 0x12, 0x1a, 0x0a, 0x08, 0x6c, 0x69, 0x63, 0x65, 0x6e, 0x73, 0x65, 0x73, 0x18, 0x05, + 0x20, 0x03, 0x28, 0x09, 0x52, 0x08, 0x6c, 0x69, 0x63, 0x65, 0x6e, 0x73, 0x65, 0x73, 0x2a, 0x26, + 0x0a, 0x0b, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, + 0x07, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x10, 0x00, 0x12, 0x0a, 0x0a, 0x06, 0x47, 0x49, + 0x54, 0x48, 0x55, 0x42, 0x10, 0x01, 0x42, 0x28, 0x5a, 0x26, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, + 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x73, 0x61, 0x66, 0x65, 0x64, 0x65, 0x70, 0x2f, 0x76, 0x65, 0x74, + 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x69, 0x6e, 0x70, 0x75, 0x74, + 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/go.mod b/go.mod index 25f459a..190295e 100644 --- a/go.mod +++ b/go.mod @@ -6,13 +6,13 @@ require ( github.com/deepmap/oapi-codegen v1.12.4 github.com/golang/protobuf v1.5.2 github.com/google/cel-go v0.13.0 - github.com/google/osv-scanner v1.1.0 + github.com/google/osv-scanner v1.2.0 github.com/jedib0t/go-pretty/v6 v6.4.4 github.com/safedep/dry v0.0.0-20230222132026-c8b6cb976849 github.com/sirupsen/logrus v1.9.0 github.com/spf13/cobra v1.6.1 github.com/stretchr/testify v1.8.1 - golang.org/x/term v0.4.0 + golang.org/x/term v0.5.0 google.golang.org/protobuf v1.28.1 gopkg.in/yaml.v2 v2.4.0 ) @@ -32,8 +32,8 @@ require ( github.com/rivo/uniseg v0.2.0 // indirect github.com/spf13/pflag v1.0.5 // indirect github.com/stoewer/go-strcase v1.2.0 // indirect - golang.org/x/mod v0.7.0 // indirect - golang.org/x/sys v0.4.0 // indirect + golang.org/x/mod v0.8.0 // indirect + golang.org/x/sys v0.5.0 // indirect golang.org/x/text v0.5.0 // indirect google.golang.org/genproto v0.0.0-20221027153422-115e99e71e1c // indirect gopkg.in/yaml.v3 v3.0.1 // indirect diff --git a/go.sum b/go.sum index afeb37d..77d0817 100644 --- a/go.sum +++ b/go.sum @@ -24,8 +24,8 @@ github.com/google/cel-go v0.13.0 h1:z+8OBOcmh7IeKyqwT/6IlnMvy621fYUqnTVPEdegGlU= github.com/google/cel-go v0.13.0/go.mod h1:K2hpQgEjDp18J76a2DKFRlPBPpgRZgi6EbnpDgIhJ8s= github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= -github.com/google/osv-scanner v1.1.0 h1:6XL8tD8u4w8NFyiMo03Yd4xGG1VXhZXyrBESBuyWeUY= -github.com/google/osv-scanner v1.1.0/go.mod h1:w8BdEP4PJSosGhDfZ6W5RGMfIGb73rW38vCXB9DWA4c= +github.com/google/osv-scanner v1.2.0 h1:TWPfI5kDqO/wgxihVyRxX15JQFJeZ0NyPvx1UtR6g38= +github.com/google/osv-scanner v1.2.0/go.mod h1:2GmR3DxMxDW/HkkgOf+b/KCfspX2Ls4wcapeNnSNZoY= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc= @@ -46,8 +46,6 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= -github.com/safedep/dry v0.0.0-20230218045153-1a93b0397b55 h1:OBzggSWzjyEa7YaXp2DvpKDe1wYXtOEcFXQfDqkK7PI= -github.com/safedep/dry v0.0.0-20230218045153-1a93b0397b55/go.mod h1:odFOtG1l46k23IaCY6kdNkkLW8L+NT+EUVYYVphP59I= github.com/safedep/dry v0.0.0-20230222132026-c8b6cb976849 h1:5nO9ht1jn7XHFyNFRhUneDZbKmwh4kRr0w/EoWuOQQA= github.com/safedep/dry v0.0.0-20230222132026-c8b6cb976849/go.mod h1:odFOtG1l46k23IaCY6kdNkkLW8L+NT+EUVYYVphP59I= github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= @@ -70,14 +68,14 @@ github.com/stretchr/testify v1.7.4/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -golang.org/x/mod v0.7.0 h1:LapD9S96VoQRhi/GrNTqeBJFrUjs5UHCAtTlgwA5oZA= -golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.8.0 h1:LUYupSeNrTNCGzR/hVBk2NHZO4hXcVaW1k4Qx7rjPx8= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18= -golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/term v0.4.0 h1:O7UWfv5+A2qiuulQk30kVinPoMtoIPeVaKLEgLpVkvg= -golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ= +golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/text v0.5.0 h1:OLmvp0KP+FVG99Ct/qFiL/Fhk4zp4QQnZ7b2U+5piUM= golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/pkg/analyzer/filter/eval.go b/pkg/analyzer/filter/eval.go index 2243592..a94bad6 100644 --- a/pkg/analyzer/filter/eval.go +++ b/pkg/analyzer/filter/eval.go @@ -251,7 +251,12 @@ func (f *filterEvaluator) buildFilterInput(pkg *models.Package) (*filterinput.Fi // Scorecard scorecard := utils.SafelyGetValue(insight.Scorecard) - checks := utils.SafelyGetValue(utils.SafelyGetValue(scorecard.Content).Checks) + scorecardContent := utils.SafelyGetValue(scorecard.Content) + + // Aggregated score + fi.Scorecard.Score = utils.SafelyGetValue(scorecardContent.Score) + + checks := utils.SafelyGetValue(scorecardContent.Checks) for _, check := range checks { fi.Scorecard.Scores[string(utils.SafelyGetValue(check.Name))] = utils.SafelyGetValue(check.Score) diff --git a/pkg/parser/parser.go b/pkg/parser/parser.go index a750926..79ed8d1 100644 --- a/pkg/parser/parser.go +++ b/pkg/parser/parser.go @@ -8,6 +8,10 @@ import ( "github.com/safedep/vet/pkg/models" ) +const ( + customParserTypePyWheel = "python-wheel" +) + // We are supporting only those ecosystems for which we have data // for enrichment. More ecosystems will be supported as we improve // the capability of our Insights API @@ -18,6 +22,10 @@ var supportedEcosystems map[string]bool = map[string]bool{ models.EcosystemPyPI: true, } +var customExperimentalParsers map[string]lockfile.PackageDetailsParser = map[string]lockfile.PackageDetailsParser{ + customParserTypePyWheel: parsePythonWheelDist, +} + type Parser interface { Ecosystem() string Parse(lockfilePath string) (models.PackageManifest, error) @@ -53,6 +61,13 @@ func FindParser(lockfilePath, lockfileAs string) (Parser, error) { } } + if p, ok := customExperimentalParsers[lockfileAs]; ok { + pw := &parserWrapper{parser: p, parseAs: lockfileAs} + if pw.supported() { + return pw, nil + } + } + return nil, fmt.Errorf("no parser found with: %s for: %s", lockfileAs, lockfilePath) } @@ -93,6 +108,8 @@ func (pw *parserWrapper) Ecosystem() string { return models.EcosystemMaven case "buildscript-gradle.lockfile": return models.EcosystemMaven + case customParserTypePyWheel: + return models.EcosystemPyPI default: return "" } diff --git a/pkg/parser/pywheel.go b/pkg/parser/pywheel.go new file mode 100644 index 0000000..7a86b99 --- /dev/null +++ b/pkg/parser/pywheel.go @@ -0,0 +1,79 @@ +package parser + +import ( + "archive/zip" + "errors" + "io" + "net/mail" + "strings" + + "github.com/google/osv-scanner/pkg/lockfile" + "github.com/safedep/vet/pkg/common/logger" +) + +// https://packaging.python.org/en/latest/specifications/binary-distribution-format/ +func parsePythonWheelDist(pathToLockfile string) ([]lockfile.PackageDetails, error) { + details := []lockfile.PackageDetails{} + + r, err := zip.OpenReader(pathToLockfile) + if err != nil { + return details, err + } + + defer r.Close() + for _, file := range r.File { + if strings.HasSuffix(file.Name, ".dist-info/METADATA") { + fd, err := file.Open() + if err != nil { + return details, err + } + + defer fd.Close() + return parsePythonPkgInfo(fd) + } + } + + return details, errors.New("no METADATA found inside wheel") +} + +// https://packaging.python.org/en/latest/specifications/core-metadata/ +func parsePythonPkgInfo(reader io.Reader) ([]lockfile.PackageDetails, error) { + m, err := mail.ReadMessage(reader) + if err != nil { + return []lockfile.PackageDetails{}, err + } + + // https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use + if dists, ok := m.Header["Requires-Dist"]; ok { + details := []lockfile.PackageDetails{} + for _, dist := range dists { + p, err := parsePythonPackageSpec(dist) + if err != nil { + logger.Errorf("Failed to parse python pkg spec: %s err: %v", + dist, err) + continue + } + + details = append(details, p) + } + + return details, nil + } + + return []lockfile.PackageDetails{}, nil +} + +// https://peps.python.org/pep-0440/ +// https://peps.python.org/pep-0508/ +// Parsing python dist version spec is not easy. We need to use the spec grammar +// to do it correctly. Taking shortcut here by only using the name as the first +// iteration ignoring the version +func parsePythonPackageSpec(pkgSpec string) (lockfile.PackageDetails, error) { + name := strings.SplitN(pkgSpec, " ", 2)[0] + return lockfile.PackageDetails{ + Name: name, + Version: "0.0.0", + Ecosystem: lockfile.PipEcosystem, + CompareAs: lockfile.PipEcosystem, + }, nil +} diff --git a/pkg/reporter/markdown.template.md b/pkg/reporter/markdown.template.md index 1b79d6c..a6dec23 100644 --- a/pkg/reporter/markdown.template.md +++ b/pkg/reporter/markdown.template.md @@ -23,8 +23,8 @@ issues identified during the scan. {{ range $key, $value := .Remediations }} > {{ $key }} -| Package | Update Version | Risk Score | Issues | -|---------|----------------|------------|--------| +| Package | Update Version | Impact Score | Issues | +|---------|----------------|--------------|--------| {{- range $value }} | {{ .PkgRemediationName }} | {{ .Pkg.Insights.PackageCurrentVersion }} | {{ .Score }} | - | {{- end }} diff --git a/pkg/scanner/scanner.go b/pkg/scanner/scanner.go index 5751060..c540cce 100644 --- a/pkg/scanner/scanner.go +++ b/pkg/scanner/scanner.go @@ -56,7 +56,7 @@ func (s *packageManifestScanner) ScanDirectory(dir string) error { // automatic parser selection func (s *packageManifestScanner) ScanLockfiles(lockfiles []string, lockfileAs string) error { - logger.Infof("Scannding %d lockfiles as %s", len(lockfiles), lockfileAs) + logger.Infof("Scanning %d lockfiles as %s", len(lockfiles), lockfileAs) manifests, err := scanLockfilesForManifests(lockfiles, lockfileAs) if err != nil {