The Predictive Ecosystem Analyzer (PEcAn) is an integrated ecological bioinformatics toolbox.
This is a major change:
get.rh
function in PEcAn.data.atmosphereparse
option to PEcAn.utils::read_web_config
had no effect when expand
was TRUE (#2421).PEcAn.DB::symmetric_setdiff
falsely report no differences (#2428).-T-
files have based on ED2 version (#2623)tmvtnorm
package with TruncatedNormal
package for speed up per #2621.created_at
or updated_at
timestamps. The database now updates these itself and ensures they are consistently in UTC (#1083).PEcAn.DB::insert_table
now uses DBI::dbAppendTable
internally instead of manually constructed SQL (#2552).plot.posterior.density
->plot_posterior.density
, plot.prior.density
->plot_prior.density
, plot.trait
->plot_trait
(#2439).plot.netcdf
->plot_netcdf
(#2526).Remote.Sync.launcher
-> Remote_Sync_launcher
(#2652)make check
and CI builds will now fail if R CMD check
returns any ERRORs or any "newly-added" WARNINGs or NOTEs. "Newly-added" is determined by strict string comparison against a check result saved 2019-09-03; messages that exist in the reference result do not break the build but will be fixed as time allows in future refactorings (#2404).ncdf4.helpers
package, which has been removed from CRAN (#2511).call_MODIS()
have been changed (issue #2519).PEcAn.data.atmosphere::download.NOAA_GEFS_downscale
. Precipitation was being downscaled via a spline which was causing fake rain events. Instead the 6 hr precipitation flux values from GEFS are preserved with 0's filling in the hours between.dbfile.input.insert
to work with inputs (i.e soils) that don't have start and end dates associated with themstop_on_error
is now TRUE
for non-ensemble runs; i.e., workflows that run only one model simulation (or omit the ensemble
XML group altogether) will fail if the model run fails. For ensemble runs, the old behavior is preserved; i.e., workflows will continue even if one of the model runs failed. This behavior can also be manually controlled by setting the new run -> stop_on_error
XML tag to TRUE
or FALSE
.PEcAn.utils
into other packages (#2830, #2857):
ensemble.filename
, get.results
, runModule.get.results
, read.sa.output
, sensitivity.filename
,
and write.sa.configs
have been moved to PEcAn.uncertainty
.create.base.plot
, dhist
, plot_data
and theme_border
have been moved to PEcAn.visualizaton
.priorfig
and plot_densities
(#2439).get_postgres_envvars
, which tries to look up connection parameters from Postgres environment variables (if they are set) and return them as a list ready to be passed to db.open
. It should be especially useful when writing tests that need to run on systems with many different database configurations (#2541).vacuum.bety.sh
and reindex.bety.sh
have been moved to the BeTY database repository (#2563).dump.pgsql.sh
and dump.mysql.sh
have been deleted. See the "BeTY database administration" chapter of the PEcAn documentation for current recommendations (#2563).check.dependencies.sh
, update.dependencies.sh
, and install_deps.R
have been deleted. Use generate_dependencies.R
and the automatic dependency handling built into make install
instead (#2563).PEcAn.utils
(#2830):
do_conversions
and runModule.run.write.configs
, run.write.configs
. These are now in PEcAn.workflow
get.ensemble.samples
, read.ensemble.output
, write.ensemble.configs
. These are now in PEcAn.uncertainty
logger.debug
, logger.error
, logger.getLevel
, logger.info
, logger.setLevel
,
logger.setOutputFile
, logger.setQuitOnSevere
, logger.setWidth
, logger.severe
, logger.warn
.
These are now in PEcAn.logger
rlang::UQ
syntax with the recommended !!
PEcAn.uncertainty::read.ensemble.output
in PEcAn.utils::get.results
. Otherwise, it would sometimes use the deprecated PEcAn.utils::read.ensemble.output
version.PEcAn.ED2::met2model.ED2
now skips processing of years for which all output files are already present (unless overwrite = TRUE
). This prevents a lot of unnecessary work when extending an existing ED met record.package::function
) throughout PEcAn.meta.analysis
. Otherwise, many of these functions would fail when trying to run a meta-analysis outside of the PEcAn workflow (i.e. without having loaded the packages first) (#2351).PEcAn.DB
tests create database connections, and make sure tests work with both the newer Postgres
and older PostgreSQL
drivers (#2351).rlang::UQ
syntax with the recommended !!
PEcAn.uncertainty::read.ensemble.output
in PEcAn.utils::get.results
. Otherwise, it would sometimes use the deprecated PEcAn.utils::read.ensemble.output
version.release
, the current public release of R (currently R 3.5). Build failures in this version are fixed before merging the change that caused them. When we say PEcAn is fully tested and working, this is the build we mean.devel
, the newest available development build of R. We will fix issues with this version before the next major R release.oldrel
, the previous major release of R (currently R 3.4). We will fix issues with this version as time allows, but we do not guarantee that it will stay compatible.PEcAn.utils
:
scripts/generate_dependencies.R
is now used to generate dependencies for make and dockerPEcAn.DB::get.trait.data
, if trait.names
is NULL
or missing, use the traits for which at least one prior is available among the input list of PFTs. (Previously, we were getting this from the PEcAn.utils::trait.dictionary
, which we are trying to deprecate #1747). (#2351)PEcAn.DB
related to getting trait data, including replacing many manual database queries with dplyr
calls.met2model.<MODEL>
functions for most models.RPostgres::Postgres
backend. The backend is officially supported by db.query
, and basic workflows run top-to-bottom with the Postgres
backend. However, RPostgreSQL
is still the default until we do more robust testing of all modules.PEcAn.DB::db.query
now optionally supports prepared statements (#395).PEcAn.DB::query_priors
that expands the functionality of query.priors
by (1) accepting PFTs by name or ID and (2) allowing the user to request all possible combinations of the input PFTs and traits (i.e. expand.grid(pfts, traits)
) or just the pairwise combinations (i.e. pft[1]-trait[1], pft[2]-trait[2]
). This function also comes with more robust error handling and a set of unit tests (#2351).PEcAn.DB::query_pfts
for finding PFT IDs and types from the PFT name and (optionally) model type (#2351).PEcAn.utils::load_local
reads Rdata
files into a named list (instead of into the current environment).PEcAn.visualization::points2county
, thus removing many indirect dependencies by no longer importing the earth
package.PEcAn.data.mining
from the Make build. It can still be installed directly from R if desired, but is skipped by default because it is in early development, does not yet export any functions, and creates a dependency on the (large, often annoying to install) ImageMagick library.MySQL
database driver. Now, only PostgreSQL
(and, experimentally, RPostgres
) are supported. With this, remove RMySQL
dependency in several places.Many updates to docker. Docker is now becoming more stable and can be used for new installations. Please check on the PEcAn slack channel if you want to start to use the docker installation.
This version replaces google maps in the web interface with leaflet. During this conversion we also sped up the rendering of the markers on the map. They are now grouped together, clicking a group marker will zoom into that area. You can always search in the box on the left to filter the markers if you know the sitename.
Below is a full list of all changes in this release.
read_web_config
which would filter out all variables.ed2in_tags
from XML. Now numbers (e.g. <TRAIT_PLASTICITY_SCHEME>0</TRAIT_PLASTICITY_SCHEME>
) and numeric vectors (e.g. <INCLUDE_THESE_PFT>9,10,11,12</INCLUDE_THESE_PFT>
) are correctly written to ED2IN without quotes.docker/receiver.py
to accept a pecan_json
object containing a JSON version of the PEcAn settings. Can now Use RabbitMQ HTTP API (called from R with httr
) to send a settings list (function submit_workflow
)fileServer
(basically, direct download), and NetCDF files are also readable through OpenDAP.
datasetScan
to the THREDDS catalog to search for dbfiles
. Now, model inputs (e.g. meteorology files) are accessible via THREDDS as well.method
) argument for data.atmosphere::download.CRUNCEP
, which defaults to opendap
(as it was), but can be switched to the slower but more robust NetCDF subset (ncss
).download.CRUNCEP
, check target coordinate against the land-sea mask. If sea, pick the nearest land pixel within 1 degree of target. This facilitates doing runs at coastal sites that may get masked out.pecan.xml
(#2183; fixes #2146).model
tag <all_pfts>
. If "false" (default), set ED2IN's INCLUDE_THESE_PFT
to only PFTs explicitly configured through PEcAn. If "true", use all 17 of ED2's PFTs.model
tag <barebones_ed2in>
. If "true", only write ED2IN tags, and do not include comment annotations. If "false" (default), try to transfer comments from ED2IN template to target ED2IN. Because of comments are written by matching line numbers, leaving this as "false" can lead to unexpected results whenever <ed2in_tags>
contains tags missing from the ED2IN
template.pecan.xml
tags.PEcAn.utils::do_conversions
has been moved to PEcAn.workflow::do_conversions
.
PEcAn.utils::do_conversions
still works for now with a warning, but is deprecated and will be removed in the future.r-base
to rocker/tidyverse:3.5.1
. This (1) saves build time (because many R packages and system dependencies are pre-installed), and (2) enhances reproducibility (because of the strict versioning rules of the rocker
packages)Fixed issue #2064 which sends one met path to write.sa.config.
Fixed issue #1939 which corrects output time vector for FATES output
Update to read.output to look for and read only PEcAn formatted .nc output based on the pecan standard filename format of YYYY.nc. Solves issues with models such as FATES and dvm-dos-tem where the original model output is also in .nc file format and was not ignored by read.output, causing errors with output parsing and plotting with Shiny. Removed deprecated function convert.outputs
PEcAn.data.atmosphere:
No geonamesUsername set
warning by updating geonames package to development versionED:
missing_port_bety
dataone_download.R:
Could not specify the port for BETY in config.php. Can now use db_bety_port
to specify port.
Data_Ingest_App:
updateSelectizeInput
to populate selectizeInput
with choices from BETYdb. This instantly loads the inputfields where other methods take minutes to load.PEcAn.undertainty gains one new function (input.ens.gen) and three functions moved from PEcAn.utils (see "Changed" below)
IC workflow now has functionality to generate ensembles.
You can now generate ensembles for parameters and met separatly and using different methods.
Soil process is now capable of reading in soil data from gSSURGO databse.
In modules/rtm new function foursail() to interface with the 4SAIL Fortran code. To enable the use of 4SAIL with any version of PROSPECT (i.e. 4, 5, 5b, D) and custom soil/background reflectance inputs
Shiny/Dependency explorer
Explore the interdependencies between pecan packages/functions.
From history you can now select an old run and show the curl command to re-execute this run. This only works with runs submitted through web interface right now.
Experimental support for docker (#1028)
dataone_download.R:
shiny/Data-Ingest
dbfile.input.insert
input.format.vars
to query the BETYdbhelper.R
auto.name.directory
This function uses the format_name and the site_id for a given input to create a directory name in the style of other dbfiles names.Next Step
buttons progress workflow programmaticallyshinytoastr
packageDepends
: PEcAn.visualization, shinytoastr, shinyWidgets, shinyjspecan/base/db
input.format.vars.R
. This function registers the format and the (optional) formats_variables record using db_merge_into
.data.atmosphere
check_met_input_file
-- Check that target met file conforms to PEcAn meteorology data standard.get_cf_variables_table
-- Retrieve CF variables table as a data.frame
docker:
(cd docker ; docker build -t pecan/depends:latest -f Dockerfile.depends .)
DEPENDS
on shinyFiles
or shinycssloaders
src/Makevars
. This should make the package Windows-compatible.Changed
section below.PEcAn.workflow
, for functions used to perform the each major step of the analysis. These were previously scattered in other base packages.PEcAn.logger
functions similar to stopifnot
to facilitate assertive programming: severeifnot
, errorifnot
, warnifnot
, infoifnot
, debugifnot
gpm()
function ("generalized plate model"), facilitating experimentation with different absorption coefficientsspectra
S3 class and methods for subsetting (e.g. myspec[[400:700]]
), plotting (plot()
and matplot()
), and combining spectra by wavelength.resample
functions for quickly resampling spectra (and, more generally, vectors and functions) to different dimensions.EDR
API has been revised. Setup has been refactored from EDR via new setup_edr
function, which relies on the ED utilities (see PEcAn.ED2
below), and the EDR
function now focuses only on execution. Also, added new params2edr
function to make it easy to convert complex EDR parameters list to flat parameter vector required by invert_bt
(or other optimization functions).SolarR
is now solar
and WS
is now windspeed
. Previously generated met files with the old names will still work for BioCro 0.9 runs, but will need to be renamed before using them with BioCro 1.0.models/biocro/inst/extdata/defaults
will not work when using BioCro 1.0.rename.jags.columns
renamed to rename_jags_columns
to avoid conflict with S3 method naming conventionsrhdf5
library with hdf5r
, a more modern alternative that is available on CRAN.runModule.get.trait.data
has been moved to the new PEcAn.workflow package to avoid a circular package dependency between PEcAn.DB and PEcAn.settings.Rmd
files in all subdirectories of the documentation source are rendered by default. The overall structure of the documentation has been revised for clarity and cohesiveness.site.id=site$id
instead of site.id=site, as site is an object not just the idPEcAn.utils
now lazy-loads data for faster execution of functions that consult lookup tables, especially to_ncvar
.PEcAn.BIOCRO
daily and yearly results: Was calculating every row from whole simulation instead of that day (#1738)PEcAn.utils::model2netcdf
, which has been deprecated since PEcAn 1.3.7. Use model2netcdf.<YOURMODEL>
in the appropriate model package instead.PEcAn.utils
package. It now loads more quietly and is much less likely to mask functions in a package you loaded earlier.PEcAn.utils
into other PEcAn packages. The PEcAn.utils
versions still work with a deprecation warning, but will be removed in next release.
listToXml
and SafeList
moved to PEcAn.settings
fqdn
moved to PEcAn.remote
Documentation