An extremely fast FEC filing parser written in C
A C program to stream and parse Federal Election Commission (FEC) filings, writing output to CSV.
Download the latest release and place it on your path, or if you have Homebrew and are on Mac or Linux, you can install via:
brew install fastfec
You can also build a binary yourself following the development instructions below.
Once FastFEC has been installed, you can run the program by calling fastfec
in your terminal:
Usage: fastfec [flags] <id or file> [output directory=output] [override id]
[flags]
: optional flags which must come before other args; see below<file or id>
is either
--print-url
): prints the possible URLs the filing lives on the FEC docquery website[output directory]
is the folder in which CSV files will be written. By default, it is output/
.[override id]
is an ID to use as the filing ID. If not specified, this ID is pulled out of the first parameter as a numeric component that can be found at the end of the path.The CLI will read the specified filing from disk and then write output CSVs for each form type in the output directory. The paths of the outputted files are:
{output directory}/{filing id}/{form type}.csv
You can also pipe the output of another command in by following this usage:
[some command] | fastfec [flags] <id> [output directory=output]
The CLI supports the following flags:
--include-filing-id
/ -i
: if this flag is passed, then the generated output will include a column at the beginning of every generated file called filing_id
that gets passed the filing ID. This can be useful for bulk uploading CSVs into a database--silent
/ -s
: suppress all non-error output messages--warn
/ -w
: show warning messages (e.g. for rows with unexpected numbers of fields or field types that don't match exactly)--no-stdin
/ -x
: disable receiving piped input from other programs (stdin)--print-url
/ -p
: print URLs from docquery.fec.gov (cannot be specified with other flags)The short form of flags can be combined, e.g. -is
would include filing IDs and suppress output.
Parsing a local filing
fastfec -s 13360.fec fastfec_output/
fastfec_output/13360/
.Downloading and parsing a filing
Get the FEC filing URL needed:
fastfec -p 13360
If you have curl
installed, you can then run this command:
curl https://docquery.fec.gov/dcdev/posted/13360.fec | fastfec 13360
output/13360/
If you don't have curl installed, you can also download the filing from the URL (https://docquery.fec.gov/dcdev/posted/13360.fec), save the file, and run (is equivalent to the above):
fastfec 13360.fec
The following was performed on an M1 Macbook Air:
Filing | Size | Time | Memory usage | CPU usage |
---|---|---|---|---|
1464847.fec |
8.4gb | 1m 42s | 1.7mb | 98% |
Zig is used to build and compile the project. Download and install the latest version of Zig (>=0.11.0) by following the instructions on the website (you can verify it's working by typing zig
in the terminal and seeing help commands).
FastFEC has no external C dependencies. PCRE is bundled with the library to ensure compatibility with Zig's build system and cross-platform compilation.
From the root directory of the repo, run:
zig build
zig-out/bin/fastfec
and a shared library file in the zig-out/lib/
directory-Dlib-only=true
as a build option following zig build
-Dtarget=x86_64-windows
(see here for additional targets)Currently, there's C tests for specific parsing/buffer/write/CLI functionality and Python integration tests.
zig build test
cd python
pip install -r requirements-dev.txt
tox -e py
See the GitHub test workflow for more info
python scripts/generate_mappings.py
: A Python script to auto-generate C header files containing column header and type mappings