An interface to Google's BigQuery from R.
bigrquery is now MIT licensed (#453).
Deprecated functions (i.e. those not starting with bq_) have been
removed (#551). These have been superseded for a long time and were formally
deprecated in bigrquery 1.3.0 (2020).
bq_table_download() now returns unknown fields as character vectors.
This means that BIGNUMERIC (#435) and JSON (#544) data is downloaded into
R for you to process as you wish.
It now parses dates using the clock package. This leads to a considerable performance improvement (#430) and ensures that dates prior to 1970-01-01 are parsed correctly (#285).
bigquery datasets and tables will now appear in the connection pane when
using dbConnect (@meztez, #431).
dbAppendTable() (#539), dbCreateTable() (#483), and dbExecute (#502)
are now supported.
dbGetQuery()/dbSendQuery() gains support for parameterised queries via
the params argument (@byapparov, #444).
dbReadTable(), dbWriteTable(), dbExistsTable(), dbRemoveTable(),
and dbListFields() now all work with DBI::Id() (#537).
bigrquery now uses 2nd edition of dbplyr interface (#508) and is compatible with dbplyr 2.4.0 (#550).
Joins now work correctly across bigrquery connections (#433).
grepl(pattern, x) is now correctly translated to
REGEXP_CONTAINS(x, pattern) (#416).
median() gets a translation that works in summarise() and a clear
error if you use it in mutate() (#419).
tbl() now works with views (#519), including the views found in the
INFORMATION_SCHEMA schema (#468).
tbl(con, sql("...")) now works robustly once more (#540), fixing the
"URL using bad/illegal format or missing URL" error.
runif(n()) gains a translation so that slice_sample() can work
(@mgirlich, #448).
Google API URLs have been aligned with the Google Cloud Discovery docs. This enables support for Private and Restricted Google APIs configurations (@husseyd, #541)
Functions generally try to do a better job of telling you when you've
supplied the wrong type of input. Additionally, if you supply SQL() to
a query, you no longer get a weird warning (#498).
If bq_job_wait() receives a 503 response, it now waits for 2 seconds and
tries again (#535).
dbFetch() now respects the quiet setting from the connection (#463).
dbGetRowCount() and dbHasComplete() now return correct values when you
try to fetch more rows than actually exist (#501).
New dbQuoteLiteral() method for logicals reverts breaking change introduced
by DBI 1.1.2 (@meztez, #478).
dbWriteTable() now correct uses the billing value set in the
connection (#486).
Sync up with the current release of gargle (1.4.0). Recently gargle introduced some changes around OAuth and bigrquery is syncing with up that:
bq_oauth_client() is a new function to replace the now-deprecated
bq_oauth_app().client argument of bq_auth_configure() replaces the
now-deprecated client argument.bq_auth_configure() emphasizes that the preferred
way to "bring your own OAuth client" is by providing the JSON downloaded
from Google Developers Console.op_table.lazy_select_query() now returns a string instead of a list, which
fixes an error seen when printing or using functions like head() or
dplyr::glimpse() (@clente, #509).
Fix for R CMD check in R-devel (#511)
bigrquery is now compatible with dbplyr 2.2.0 (@mgirlich, #495).
brio is new in Imports, replacing the use of the Suggested package readr,
in bq_table_download() (@AdeelK93, #462).
bq_table_download() has been heavily refactored (#412):
max_results argument has been deprecated in favor of n_max, which
reflects what we actually do with this number and is consistent with the
n_max argument elsewhere, e.g., readr::read_csv().page_size is no longer fixed and, instead, is
determined empirically. Users are strongly recommended to let bigrquery
select page_size automatically, unless there's a specific reason to do
otherwise.The BigQueryResult object gains a billing slot (@meztez, #423).
collect.tbl_BigQueryConnection() honours the bigint field found in a connection object created with DBI::dbConnect() and passes bigint along to bq_table_download(). This improves support for 64-bit integers when reading BigQuery tables with dplyr syntax (@zoews, #439, #437).
BigQuery BYTES and GEOGRAPHY column types are now supported via
the blob and
wk packages, respectively
(@paleolimbot, #354, #388).
When used with dbplyr >= 2.0.0, ambiguous variables in joins will get
suffixes _x and _y (instead of .x and .y which don't work with
BigQuery) (#403).
bq_table_download() works once again with large row counts
(@gjuggler, #395). Google's API has stopped accepting startIndex
parameters with scientific formatting, which was happening for large
values (>1e5) by default.
New bq_perform_query_dry_run() to retrieve the estimated cost of
performing a query (@Ka2wei, #316).
Old functions (not starting with bq_) are deprecated (@byapparov, #335)
When bq_perform_*() fails, you now see all errors, not just the first (#355).
bq_perform_query() can now execute parameterised query with parameters
of ARRAY type (@byapparov, #303). Vectors of length > 1 will be
automatically converted to ARRAY type, or use bq_param_array() to
be explicit.
bq_perform_upload() works once again (#361). It seems like the generated
JSON was always incorrect, but Google's type checking only recently become
strict enough to detect the problem.
dbExecute() is better supported. It no longer fails with a spurious
error for DDL queries, and it returns the number of affected rows for
DML queries (#375).
dbSendQuery() (and hence dbGetQuery()) and collect() passes on ...
to bq_perform_query(). collect() gains page_size and max_connection
arguments that are passed on to bq_table_download() (#374).
copy_to() now works with BigQuery (although it doesn't support temporary
tables so application is somewhat limited) (#337).
str_detect() now correctly translated to REGEXP_CONTAINS
(@jimmyg3g, #369).
Error messages inlude hints for common problems (@deflaux, #353).
bigrquery's auth functionality now comes from the gargle package, which provides R infrastructure to work with Google APIs, in general. The same transition is underway in several other packages, such as googledrive. This will make user interfaces more consistent and makes two new token flows available in bigrquery:
Where to learn more:
bq_auth() all that most users need
OAuth2 tokens are now cached at the user level, by default, instead of in .httr-oauth in the current project. The default OAuth app has also changed. This means you will need to re-authorize bigrquery (i.e. get a new token). You may want to delete any vestigial .httr-oauth files lying around your bigrquery projects.
The OAuth2 token key-value store now incorporates the associated Google user when indexing, which makes it easier to switch between Google identities.
bq_user() is a new function that reveals the email of the user associated with the current token.
If you previously used set_service_token() to use a service account token, it still works. But you'll get a deprecation warning. Switch over to bq_auth(path = "/path/to/your/service-account.json"). Several other functions are similarly soft-deprecated.
R 3.1 is no longer explicitly supported or tested. Our general practice is to support the current release (3.6), devel, and the 4 previous versions of R (3.5, 3.4, 3.3, 3.2).
gargle and rlang are newly Imported.
Fix test failure with dbplyr 1.4.0.
bq_field() can now pass description parameter which will be applied
in bq_table_create() call (@byapparov, #272).