Rust CLI Project Template

A base project template for comfortably building small but reliable utilities in the Rust programming language.

Rationale

Historically, I have used Python for quick little scripts (from which my big, long-lived projects tend to unexpectedly develop), but maintaining large things written in Python risks turning into a gumption trap due to the need to either manually test every code path I affect when making changes or pour a lot of effort into writing an automated test suite to do it for me.

Rust's stronger compile-time guarantees remove most of that motivation-sapping busywork, but, since I've spent so many years getting comfortable with Python, working in a new language risks being a chicken-and-egg problem which kills off the will to change tasks. (Once I start something, it's easy to keep going.)

The purpose of this repository is to lower the friction for Rust-based development of such "scripts of unknown potential" as much as possible. (Hence the inclusion of aliases like just run which serve only to remove the need to think about whether to type just or cargo, and the wrappers for cargo-edit commands which regenerate the API documentation I'll have open in a browser tab after adding/removing/updating dependencies.)

In short, this is a "do it properly" answer to the workflow that evolved around the boiler snippet for Python that I keep in my UltiSnips.

License

This repository is licensed under your choice of the MIT or Apache 2.0 licenses with the exception of the license texts themselves. Please replace the LICENSE file in the template folder with your preferred license if you do not habitually start your new projects under the GNU GPLv3.

Features

• Uses StructOpt (with colourized --help output and "Did you mean...?" suggestions enabled) for argument parsing because gumdrop doesn't support OsString-based parsing for correct handling of non-UTF8 paths.
• Uses anyhow for unified error handling.
• Presents an app::main(opts: CliOpts) -> Result<()> function to keep your application logic cleanly separated from argument parsing and handling of fatal errors.
• Exposes clap's support for generating shell completions by providing a --dump-completions <shell> option.
• Enables almost all rustc and clippy lints without making clippy mandatory.
• A comprehensive set of just commands, easily customized via variables (eg. for cross-compilation), including install and uninstall, which also take care of shell completions and a manpage.
• just build-dist for a 100% static x86_64 binary that starts at roughly 272KiB (248KiB with panic="abort") in new projects.
• just install-deps to install all but two optional dependencies on Debian-family distros.
• just install-cargo-deps to install all distro-agnostic dependencies.
• A basic .travis.yml for use with Travis-CI and Nightli.es.
• The just fmt command always calls the nightly version of rustfmt to ensure access to the excessive number of customization options which are gated away as unstable.

Usage

1. Clone a copy of this repository (.git must exist, so no archive downloads)
2. Run apply.py path/to/new/project
3. Edit src/app.rs to implement your application logic

NOTE: As a safety measure, apply.py generates new projects from the git HEAD of the template repository (not the working tree), so any local changes you make will not be picked up until you commit them.

Supplementary Files

Justfile Reference

Variables (just --evaluate)

Commands (just --list)

Tips

• Edit the DEFAULT command. That's what it's there for.

• You can use just from any subdirectory in your project. It's like git that way.

• just path/to/project/ (note the trailing slash) is equivalent to (cd path/to/project; just)

• just path/to/project/command is equivalent to (cd path/to/project; just command)

• The simplest way to activate the bash completion installed by just install is to add this to your .bashrc:

  for script in ~/.bash_completion.d/*; do
    if [ -e "$script" ]; then
      . "$script"
    fi
  done
  

  foo
  

• The simplest way to activate the zsh completion installed by just install is to add this to your .zshrc:

  fpath=(~/.zsh/functions(:A) $fpath)
  

• When using clap/StructOpt validators for inputs such as filesystem paths, only use them to bail out early on bad input, not as your only check. They're conceptually similar to raw pointers and can be invalidated between when you check them and when you try to use them because Rust can't control what the OS and other programs do in the interim. See this blog post for more on this idea of references versus values in command-line arguments.

Build Behaviour

In order to be as suitable as possible for building compact, easy-to-distribute, high-reliability replacements for shell scripts, the following build options are defined:

If built via just build:

1. The default CARGO_BUILD_TARGET defined in the justfile will specify a 32-bit x86 build, statically linked against musl-libc for portability comparable to a Go binary. (Unless musl-gcc is installed, this will cause build failures if you depend on any crates which link to C or C++ code.)

If built via cargo build --release:

1. Full LTO (Link-Time Optimization) will be enabled. (lto = true)
2. The binary will be built with opt-level = "z" to further reduce file size.
3. If panic="abort" is uncommented in Cargo.toml, LTO will prune away the unwinding machinery to save even more space, but panics will not cause Drop implementations to be run and will be uncatchable.

If built via just build-dist:

1. Unless otherwise noted, all optimizations listed above. [1]
2. The binary will be stripped with --strip-unneeded and then with sstrip (a more aggressive companion used in embedded development) to produce the smallest possible pre-compression size.
3. The binary will be compressed via upx --ultra-brute. In my experience, this makes a file about 1/3rd the size of the input.

NOTE: --strip-unneeded removes all symbols that readelf --syms sees from the just build output, so it's not different from --strip-all in this case, but it's a good idea to get in the habit of using the safe option that's smart enough to just Do What I Mean™.

If built by just dist:

1. A packed binary will be built via build-dist and copied into dist/
2. Shell completion files and a manpage will also be built and saved into dist/

NOTE: Depending on who you're distributing precompiled binaries to, you may want get an overview of how virus scanners react to your binary using VirusTotal.

Especially with anything involving compression, small numbers of false positives are a fact of life in the world of virus detection. For example, when I tested the official installer for the NSIS authoring tools, which is used by various major companies including McAfeee, two or three no-name entries in the list of 60+ virus scanners they test reported it to have a virus.

If this proves problematic, you can either uninstall UPX or modify the justfile so the dist command always prefers the .stripped copy of the binary over the .packed one.

Experimental Cross-Compilation Support

I am currently in the process of extending this template to support generating Windows binaries, though I have no immediate plans to replace the justfile tasks so, for now, Windows-hosted development will have to settle for calling cargo commands directly.

NOTE: I haven't yet used a fresh Ubuntu install under VirtualBox to verify that I've correctly listed all the steps needed to achieve a working build environment.

To set up an environment where setting CARGO_BUILD_TARGET to x86_64-pc-windows-gnu will complete successfully and produce a .exe file which appears to work under my preliminary testing:

1. Install a MinGW package like Ubuntu's mingw-w64
2. Run rustup target add x86_64-pc-windows-gnu
3. Add the following two lines to ~/.cargo/config:

[target.x86_64-pc-windows-gnu]
linker = "/usr/bin/x86_64-w64-mingw32-gcc"

To make cargo test also work cross-platform:

1. Make sure your ~/.wine is 64-bit (indicated by an #arch=win64 comment in system.reg)
2. Make sure your kernel's binfmt_misc support is configured to allow running .exe files in the terminal via Wine as if they were native binaries.

NOTE: Wine is only suitable for "rapid iteration, approximate compatibility" testing. For proper testing of Windows binaries, the only reliable solution is to download one of the specially licensed "only for testing" Windows VMs that Microsoft offers for download from http://modern.ie/ and those cannot be used to make legally redistributable builds.

If you want to set up a Continuous Deployment-style workflow with testing against real Windows targets, the only viable option is to bypass just and call cargo directly under real Windows. I suggest a CI service like AppVeyor for this. (See also rust-cross.)

Dependencies

In order to use the full functionality offered by this boilerplate, the following dependencies must be installed:

• just add:
  • cargo-edit (cargo install cargo-edit)
• just bloat:
  • cargo-bloat (cargo install cargo-bloat)
• just build-dist:
  • The toolchain specified by the channel variable.
  • The target specified by the CARGO_BUILD_TARGET variable.
  • strip (Included with binutils)
  • sstrip (optional)
  • upx (optional, sudo apt-get install upx)
• just fmt and just fmt-check:
  • A nightly Rust toolchain (rustup toolchain install nightly)
  • rustfmt for the nightly toolchain (rustup component add rustfmt --toolchain nightly)
• just dist-supplemental:
  • help2man (sudo apt-get install help2man)
• just kcachegrind:
  • Valgrind (sudo apt-get install valgrind)
  • KCachegrind (sudo apt-get install kcachegrind)
• just kcov:
  • A Rust-compatible build of kcov
• just rm:
  • cargo-edit (cargo install cargo-edit)
• just test:
  • clippy (rustup component add clippy)
  • cargo-audit (cargo install cargo-audit)
  • cargo-deadlinks (cargo install cargo-deadlinks)
  • cargo-outdated (cargo install cargo-outdated)
• just update:
  • cargo-edit (cargo install cargo-edit)

Dependency Installation

• Debian/Ubuntu/Mint:

  export PATH="$HOME/.cargo/bin:$PATH"
  cargo install just
  just install-deps
  
  # ...and now manually install the following optional tools:
  #  - sstrip (from ELFkickers)
  #  - kcov (version 31 or higher with --verify support)
  

• Other distros:

  export PATH="$HOME/.cargo/bin:$PATH"
  cargo install just
  just install-cargo-deps
  
  # ...and now manually install the following optional tools:
  #  - help2man
  #  - kcachegrind
  #  - kcov (version 31 or higher with --verify support)
  #  - strip (from binutils)
  #  - sstrip (from ELFkickers)
  #  - upx
  #  - valgrind
  

TODO

• Add a .travis.yml at the top level to plumb the various test suites (template repo and generated project) into CI and then add a badge.

• Add a #[cfg(windows)] version of the path_output_dir validator and make the libc dependency conditional on not(windows) so that cross-compiling for Windows using the x86_64-pc-windows-gnu target can be a viable way to quickly fire off alpha/beta-testing builds to Windows-using peers.

• Investigate how flexible QuiCLI and its dependency on env_logger are and whether it'd be useful to rebase on it or whether I'd just be reinventing most of it anyway to force the exact look and feel I achieved with stderrlog. (eg. The Verbosity struct doesn't implement "-v and -q are mirrors of each other" and I'm rather fond of stderrlog's approach to timestamp toggling.)

  • What effect does QuiCLI have on the final binary size? (not a huge concern)

• Investigate why cargo-cov isn't hiding the components of the rust standard library and whether it can be induced to generate coverage despite some tests failing. If so, add a command for it.

• Read the callgrind docs and figure out how to exclude the Rust standard library from what Kcachegrind displays.

  • I may need to filter the output. [1]
  • Figure out how to add a just task for a faster but less precise profiler like gprof [1] [2], OProfile [1], or perf [1] to make it easy to leverage the various trade-offs. (And make sure to provide convenient access to flame graphs and at least one perf inspector GUI or TUI.)
  • Include references to these resources on how profilers can mislead in different ways. [1] [2] [3]
  • Look into options for making it as easy as possible to optimize and regression-test runtime performance. [1] [2] [3] [4]

• Test and enhance .travis.yml

  • Consider officially supporting Windows as a target (probably using cargo-make instead of Just) and, if I do, come up with an appveyor.yml... possibly the one from this project.

• Add a run-memstats Just task which swaps in jemalloc and sets MALLOC_CONF=stats_print:true

• Investigate commit hooks [1] [2] [3]

• Once I've cleared out these TODOs, consider using this space for a reminder list of best practices for avoiding "higher-level footguns" noted in my pile of assorted advice. (Things like "If you can find a way to not need path manipulation beyond 'pass this opaque token around', then you can eliminate entire classes of bugs")

• At least list a snip of example code for something like rustyline as the suggested way to do simple user prompting.

• Gather my custom clap validators into a crate, add some more, and have this depend on it:

  • Self-Contained data:
    • Boolean is 1/y/yes/t/true or 0/n/no/f/false (case-insensitive, include a utility function for actual parsing)
    • Integers:
      • Can be parsed as a decimal integer > 0 (eg. number of volumes)
      • Can be parsed as a decimal integer >= 0 (eg. number of bytes)
      • Number of bytes, with optional SI mebi- unit suffix (eg. 16m, including optional b, case-insensitive)
    • Floats:
      • Can be parsed as a float in the range 0.0 <= x <= 1.0
  • Invalidatable/Referential data:
    • Input files:
      • Directory exists and is browsable (+rX)
      • Path is a readable file or browsable directory (ie. read or recurse)
    • Output files:
      • Integers:
        • Augmented "number of bytes, with optional SI mebi- unit suffix" validator with upper limit for producing files representable by ISO9660/FAT32 filesystems on removable media. (2GiB, since some implementations use 32-bit signed offsets)
      • Paths:
        • File path is probably FAT32 writable
          • If file exists, access() says it's probably writable
          • If file does not exist, name is FAT32-valid and within a probably writable directory.
        • File path is probably FAT32 writable, with mkdir -p
          • Nonexistent path components are FAT32-valid
          • Closest existing ancestor is a probably writable directory
    • Network I/O:
      • Integers:
        • Successfully parses into a valid listening TCP/UDP port number (0-65535, I think)
        • Successfully parses into a valid, non-root, listening TCP/UDP port number (0 or 1024-65535, I think)
        • Successfully parses into a valid connecting TCP/UDP port number (1-65535, I think)
      • Strings:
        • Successfully parses into a SocketAddr (IP+port, may perform DNS lookup?)
        • Successfully parses into an IpAddr (may perform DNS lookup?)
      • URLs:
        • Is well-formed relative URL (external dependency behind a cargo feature)
        • Is well-formed absolute URL (external dependency behind a cargo feature)