A terminal spreadsheet multitool for discovering and arranging data
This is a pretty small release of mostly bugfixes, which still has a few changes that merit a minor version bump.
The all-stars of this release are @ajkerrigan and @geekscrapy.
@ajkerrigan found, debugged, and fixed a Python 3.10 compatibility bug. Before v2.8, you could see many warnings related to COLOR-1 in the status message.
@geekscrapy contributed 7 PRs for v2.8. Thanks to his PRs, VisiData now gracefully skips blank lines in json files, and will not attempt to load plugins which failed to install.
The ever-vigilant @geekscrapy found a bug with Enter on the helpmenu #1196.
This is a small enough fix for a very important feature.
A bunch of improvements, bugfixes, and a couple of new loaders.
sheet.options.load_lazy is True (thanks @geekscrapy for PR #1193)save_foo function does not exist and saver fallsback to options.save_filetype (reported by @geekscrapy #1180)options.save_filetype default now 'tsv'lsv filetype for simple awk-like records (requested by @fourjay #1179)odf filetype for Open Document Format spreadsheetscellobject, fontcolor, fillcolor) if options.xlsx_meta_columns (default False) (thanks @hoclun-rigsep for PR #1098)WITHOUT ROWID tables (requested by @stephancb #1111)options.encoding and options.encoding_errors for decoding of sqlite db text (reported by @WesleyAC #1156)options.clean_names (reported by @davidwales #1122)Option+keyjoin-sheets operation hangs (reported by @agjohnson #1141)In preparation for the VisiData workshop at Strangeloop 2021, this patch was released with several minor bugfixes.
Features with fixes include:
On top of this, we have updated the homebrew and personal debian repositories to contain the most recent version of VisiData.
We appreciate everyone who has mentioned VisiData in several threads on HackerNews and reddit, recently. Talking about tools you love in public spaces, means a lot to everyone who dedicates time to work on them.
Happy VisiDating!
This release introduces a new[0] hierachical menu system for discovering and executing commands.
Press Ctrl+H to open the help menu, arrows and Enter to move around. Or use the mouse.
Here's a video demo of this new menu system, including a sample workflow within VisiData for finding which commands aren't in the menu (using the mouse almost entirely).
Also in this release are several performance improvements, feature enhancements, and bugfixes. See the CHANGELOG for full details.
[0] Back in v1.1 (2018), VisiData had an experimental one-line menu system, but it was removed in v1.3 (6 months later), as it wasn't feeling right.
First off, some non-technical notices:
Since 2017 when VisiData was started, I've created a number of plugins and tools that take advantage of the VisiData "platform". These have historically been separate from VisiData, but it became a fair amount of mental overhead tracking a dozen different repos when making changes to the VisiData core repo. So a little while ago I migrated some of my older projects, like vsh and vgit, into the saulpw/visidata repo, aiming at a "mono-repo". But they never really got integrated, because they didn't feel right either: some of them were wild experiments, some I don't want to release as open source, some I don't want to provide the level of support for, and some simply don't belong in the visidata repo (like galcon).
So now I'm migrating towards a "duo-repo" approach. The saulpw/visidata repo can remain focused on providing a reliable and flexible platform, and a separate repo, "vdplus", is where I put all my VisiData plugins and extras. (This vdplus repo is private and not open-source, but Patreon subscribers can get access to it and use it if they want.)
Secondly, the core VisiData repo now requires submitters to e-sign a Copyright Assignment Agreement (CAA). The core repo will always be open-source, but as the author I should have the legal authority to release it under a different license if I choose.
Finally, our IRC channel has moved from Freenode to libera.chat; come visit us on the new #visidata!
And now for the actual software changes:
quitguard ux improvement
[quit] add Shift+Q/quit-sheet-free to quit and free associated memory (thanks @cwarden)
New major keybinding!
x is now "cut" (copy and delete), like in vim.delete family was changed to delete without copying to the clipboard (previously it acted like "cut").[options] save to foo.visidatarc from OptionsSheet (thanks @njthomas #958)
.visidatarc (call it a security feature).Ctrl+Left/Right to go-left/right-page (thanks @davidwales #1002)options.unfurl_empty to include row for empty list/dict (thanks @frosencrantz #898)expand_col_scanrows; standardize on options.default_sample_size (thanks @jsvine)options.disp_canvas_charset to change displayed chars (thanks @albert-ying #963)
options.disp_canvas_charset = ' o', any points on the graph become an o, which is hopefully big enough to see. You can set this to use different characters, just make sure that the first character is a space. This option can also be configured more precisely if you want to vary the character by how many/which points are present (like the braille characters).edit the name or delete the row and then z Ctrl+S to commit as usual)Bugs fixed (34): #758 #895 #896 #898 #946 #952 #955 #958 #959 #961 #962 #963 #964 #966 #969 #971 #974 #975 #979 #982 #983 #987 #991 #1002 #1003 #1010 #1011 #1011 #1012 #1016 #1018 #1029 #1032 #1033
See the CHANGELOG for the comprehensive list of bugfixes and changes.
This release, hot on the heels of the 2.3 release last week, is primarily to address some bugs that were found in 2.3.
Originally it was slated to be a patch release (2.3.1), but the removal of options.use_default_colors with the fix for #939 and the semantics of splitwindow felt sufficient to warrant a minor version bump, and so we tossed the few other improvements that were sitting on the develop branch.
[color] remove options.use_default_colors (thanks @lxcode #939)
options.color_default can now have both fg and bgcolor_default
options.color_default=""
[splitwindow] stabilize sheet stack associations
Shift+Z pushes 'under sheet' (if any) onto other stackShift+Z does not swap panes anymoreg Tab changes which stack is in the top vs bottom panesoptions.disp_splitwin_pct cannot be sheet-specific[status] show nSelectedRows on rstatus
TABLE_XINFO for hidden/virtual columns (thanks @dotcs #945)This release adds a memory sheet and improves on several other features. Major changes are listed below. Please see the CHANGELOG for the full list of changes.
Memory sheet, for recording values usable in expressions. View the memory sheet with open-memory (Alt+Shift+M), add current cell with Alt+M, add single aggregated value with z+. Also reference the current clipboard value as clipval. #912
The split pane feature has been revamped. Now there is one sheet stack per pane, and sheets do not jump between panes. New sheets open in the current pane. Send the current sheet to the other pane with Shift+Z (same command as invoking splitpane in the first place). Ctrl+^ swaps sheets in the current pane. See the pane column on the Sheets Sheet (pane 0 means "not on any sheet stack", i.e. it's been quit). #894
Colors now may have a background color in addition to the fg color and attributes. Add e.g. "on blue" or "on 214" to a color to set the background color. There is still the (curses) limit of 256 active colors shown at one time. (This is one of several changes in 2.3 necessary to use the new darkdraw plugin.) #435
minor additions
jump-first (bound to g Ctrl+^) to push the bottom sheet (used repeatedly will cycle through sheets in the current pane).Ctrl+X within Ctrl+X (pyobj-expr) drops into a Python REPL.options.use_default_colors now defaults to True (was False).delete commands no longer move deleted values to clipboard #895zd and gzd (delete-cells) set the value to options.null_value instead of always using None.This is a small patch.
There are two bugfixes for setcol-fill (see #884), and a typo fix in the manpage.
Additionally, there is now a packaged visidata.1, so that man visidata will also bring up the VisiData manpage, when the manpage is successfully installed.
VisiData v2.2 has been packaged for Debian, and is slated to be part of the bullseye distribution. I am hoping I can sneak v2.2.1 in there, also.
Happy data-sleuthing!
This release adds new features and changes some existing behavior. Please see the CHANGELOG for the complete list of changes and bugfixes.
In v2.0 and v2.1, options given on the command-line were "sheet-specific" and only applied to files on the command-line, and -g (--global) had to be specified to make them apply to sheets created after launch. Now, options are "global" by default (affecting all sheets which don't have a sheet-specific override), and -n indicates that subsequent CLI options are sheet-specific. The -g option is now basically a no-op (though it can be given after -n (--nonglobal) to return to the default behavior after specifying some sheet-specific options). This fixes a few things that users (including myself) were consistently misinterpreting or getting frustrated with, and hopefully does not affect anyone adversely.
By popular demand, VisiData can now store user-provided input values so they persist between sessions. This is currently disabled by default, as it may have surprising privacy or security implications in some environments. Enable persistent input history by adding the following to visidatarc:
options.input_history='previnputs'
floatlocale
The VisiData float type uses the Python builtin float, which only parses numbers in "international" notation. To parse locale-specific numbers (which use e.g. a decimal comma instead of a decimal point), there is now the floatlocale type, which parses numbers based on the LC_NUMERIC environment variable (which must be set before starting vd). A separate type was created because locale-based parsing was measured as 20x slower than the base float type. The type-floatlocale and type-floatlocale-selected (on ColumnsSheet) commands were added without keybindings; users who need this functionality will probably want to add the following to their visidatarc:
bindkey('%', 'type-floatlocale')
bindkey('g%', 'type-floatlocale-selected')
The "fancy chooser", a tabular menu with helpstrings for choosing e.g. aggregators and jointypes, is now disabled by default. Choosing an aggregator or jointype now (again) uses the status/input line, and requires entering the value manually. To invoke the fancy chooser, use Ctrl+X. If the fancy chooser is preferred by default, set options.fancy_chooser=True.
This new single-file format can store typed data and column information from multiple sheets. Use g Ctrl+S (save-all or save-selected) to invoke the multisheet saver, and then specify a filename with a .vds extension. When you load the file into vd, the Column typing and formatting will be the same!
Thanks @dracos for PR #876 implementing geojson loading and saving!