Veye-Checker

** THIS PROJECT IS NOT ACTIVELY MAINTAINED ANYMORE! **

It's a command-line util that scans packaged binaries and resolves their SHA digest values into the package information. The whole idea behind this utility is described in the Versioneye's blogpost "Identifying components by SHA values".

One can use this utility to lookup a version details of the package, fetch a license ID for the binary or get vulnerability details or automate due diligence process without installing any runtime or additional dependencies.

Default file extensions for package managers:

• Nuget (SHA512) - *.nupkg
• Maven (SHA1) - *.jar
• PYPI (MD5) - *.tar.gz, *.whl
• NPM (SHA1) - *.tgz

Usage

Download binaries from the releases and save it into your binaries folder

#NB! change version and op-sys
curl -s -L -o "${HOME}/bin/veye_checker" https://github.com/versioneye/veye-checker/releases/download/v0.2.0/veye_checker_osx

chmod a+x ~/bin/veye_checker

• resolve - scans the target folder recursively, translates a value of a file digest via VersionEye API into the product details and prints out results.

veye_checker resolve ../jars -a "api-key" -c "confs/veye_checker_local.toml"
VERSIONEYE_API_KEY="apitoken" veye_checker resolve ../jars
veye_checker resolve ../jars -o resolve.csv -a "api-key"

# only resolve binaries which are bigger than 5Kb, but smaller than 10Kb
veye_checker resolve tests/fixtures/files/ --max-file-size=10000 --min-file-size=5000

configure which digest algorithms to use commandline flags for blocking algos: no-md5, no-sha1, no-sha512 commandline options to overwrite list of file-extensions of a digest algos: ext-md5, ext-sha1, ext-sha512

veye_checker resolve ../jars -a "api-key" --no-md5 --ext-sha1="whl,jar,tgz"

• shas - scans the target folder recursively and outputs digests of supported packagefiles:

veye_checker shas ../jars/ 
veye_checker shas ../jars/ -o results.csv
VERSIONEYE_CSV_SEPARATOR="," veye_checker shas temp/bins/

# only generate diggest values for files which size in the range (5Kb, 10Kb)
veye_checker shas tests/fixtures/files/ --max-file-size=10000 --min-file-size=5000

It is possible to configure which digest algorithms to use. commandline flags for blocking algos: no-md5, no-sha1, no-sha512 commandline options to overwrite list of file-extensions of a digest algos: ext-md5, ext-sha1, ext-sha512 commandline flag -v / --verbose prints out scan configuration

# dont use MD5 for next scan and update file extensions to use for SHA1
veye_checker shas ../jars -a "api-key" -v --no-md5 --ext-sha1="whl,jar,tgz"

• lookup - fetches product details from VersionEye api by the SHA/digest value.

veye_checker lookup <SHA_STRING> -a <YOUR_API_KEY>

VERSIONEYE_API_KEY="apikey" veye_checker lookup <SHA_STRING>

API keys

All the commands ( lookup, resolve, etc ) requesting data from the VersionEye api require the API-key, which you can obtain from your's profile page.

It's possible to specify the api-key 3 ways:

• via environment variable VERSIONEYE_API_KEY

export VERSIONEYE_API_KEY="abcdef1234" veye_checker lookup SHA_VALUE_123

• add veye_checker.toml config file:

[api]
key = "abcdef1234"

• specify explicitly via command parameter

veye_checker lookup SHA_VALUE_123 -a abcdef1234

Configuration via ENV variable

It's possible to tweak a setting of the command-line tool with environmental variables, and all the variables follow a pattern: VERSIONEYE_GROUPID_VARIABLEID.

NB! Use cmd-line flags or config-file to configure file extensions used by a digest algo;

Configuration via config file

One can also put all the permanent configurations for the veye_checker tool into a veye_checker.toml file. By default the tool will lookup configuration file in the working directory, but you can always specify location with the -c flag or --config option after the subcommand.

All the fields in the configuration file are optional, and the commandline tool will use default values for unspecified fields.

# veye_checker.toml
[api]
key = "Your API key"
host = "www.versioneye.com"
path = "api/v2"
port = 80
scheme = "https"

[csv]
separator = ","
quote     = "'"
flexible  = 0

[proxy]
host = "127.0.0.1"
port = 3128
scheme = "http"

# configure file extensions
[digests.md5]
blocked = false
exts = ["whl", "gz"]

# Dont use SHA1
[digests.sha1]
blocked = true

# specify scan options
[scan]
max_file_size = 67108864
min_file_size = 1024

Build

cargo build
./target/debug/veye_checker

# or simpler command
cargo run

# or running tests
cargo test -- --test-threads=1

# test only api-calls 
VERSIONEYE_API_KEY="APIKEY" cargo test --features "api"

# or optimized production release
cargo build --release
./target/release/veye-checker

TESTING

• to run all the unit tests

cargo test -- --test-threads=1

--test-threads=1 is required for tests that are checking does reading configuration from ENV variables work;

• to run integration test against API configs

VERSIONEYE_API_KEY="your_api_key" cargo test --features="api"

• running integration tests against proxy

1. start squid proxy

docker pull sameersbn/squid:latest

docker run --name squid -d --restart=always \
  --publish 3128:3128 \
  --volume /veye-checker/temp/cache:/var/spool/squid3 \
  sameersbn/squid:latest
  
docker stop|run squid

2. run tests

    cargo test test_proxy --features=proxy

• to run acceptance tests

cd tests/acceptance

# on *nix machines
VERSIONEYE_API_KEY="your_api_key" ./tests.sh 

# on Macs
VERSIONEYE_API_KEY="your_api_key" ./tests_osx.sh

Contributing

It's opensource project and any kind of contribution is more than welcome.

Here's simple guideline to preferable workflow:

• open a issue
• implement after it lands into milestones
• write tests
• update docs
• make PR
• review

and your changes makes into next release