Opennem Versions Save

Energy market data access platform

3.15.1

1 year ago

New Stations

Data

#109 #173 - OpenNEM Volume-Weighted Price for entire history of the NEM
#261 - Burrendong facility information updates (thank you to Matt)
#264 - Market value for historic monthly solar rooftop data was incorrect in stripes

Features

#262 - Flows endpoint now supports filtering by network region
#260 - All API and static JSON outputs now contain a list of enabled feature flags (see the wiki) and optional messages for developers from OpenNEM

Fixes

#33 - Gaps in interconnector data would cause an error in the flows API endpoint
#258 - Some distillate stations on the SWIS were not correctly mapped (thank you @deathtrapz)
#239 - Significant improvements in API response times, particularly on fueltech and facility return endpoints
#256 - Some facilities were missing historic market value's in facility response page.
#206 - Stripes data consistency in monthly and daily buckets
#263- Facility feedback API endpoint was causing an exception
#266 - Additional health checks and monitoring on API endpoint and data delays (thank you @macbryla)

API Access

IMPORTANT Starting soon and with our next release the OpenNEM API will be placed behind authorization and will require an API key to access. We will be providing a free level of access for researchers.

We plan on gradually and completely deprecating the current API that is open to all (the API endpoints will not change)

If you are a current API user, or a potential API user, please fill in this form to register for access to the API. We will be in touch with you with an email with the various setup steps and access details.

The new API and registration can be discussed here

Documentation

We are steadily improving our documentation of the OpenNEM backend and data sources on the wiki

Running

Note: Container registry has been switched to ghcr.io

PyPi:

$ pip install -U opennem-backend

OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull ghcr.io/opennem/opennem/opennem:3.15.1

3.14.0

1 year ago

Features

This release is largely focused on new data and improving data quality.

Imported NEM demand data from 2005-04-01 to 2010-01-01
Imported all NEM non-scheduled generation data from 2006-07-01 to today (from METER_DATA_GEN_DUID)
Rooftop solar generation improvements (same output in Stripes and Energy views):
- From 2005-04-01 to 2018-02-28 : data is estimated using OpenNEM algorithm (monthly only)
- From 2015-01-01 to 2018-02-28 (and to the present for WEM): data from APVI
- From 2018-03-01 to present: data from AEMO (NEM only)

Fixes

#246 - Eror where no period_human was specified in some API endpoints
#240 - Squared off solar_rooftop and solar_utility start dates in stripes view
#245 - Updated incorrect emission factors on some coalmine waste facilities
#122 - Corrected error where some waste coalmine facilities were not being completely displayed in aggregate views
#221 - Further optimization and indexes for outputting from aggregates

API Access

IMPORTANT Starting in April and with our next release the OpenNEM API will be placed behind authorization and will require an API key to access. We will be providing a free level of access for researchers.

We plan on gradually and completely deprecating the current API that is open to all.

The new API and registration can be discussed here

Documentation

We are steadily improving our documentation of the OpenNEM backend and data sources on the wiki

Running

Note: Container registry has been switched to ghcr.io

PyPi:

$ pip install -U opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull ghcr.io/opennem/opennem/opennem:3.14

3.13.0

1 year ago

Features

We have reduced the time taken to process new intervals from AEMO and overnight data processing. Overnight energy data is generally available now via the API and the OpenNEM frontend approx 7am AEST.
Many internal architecture updates and optimizations.
Automated processing of AEMO facility data from registration and exemption list and general information lists

Data

#187 - CPI data has been updated
#149 - Stripes solar data is now correctly backfilled to beginning of rooftop data using a derived OPENNEM_ROOFTOP network from APVI data bucketed by day.

Fixes

#208 - Missing import and export data for the first half of 2009
#183 - Solar market value data on stripes
#182 - Blending AEMO NEM rooftop and derived rooftop data from APVI in stripes and other outputs
#179 - Better handling of corrupt zip files from upstream data providers
#228 - Handling of gaps in solar rooftop data
#154 - Fixed emission factor endpoint for network (NOTE: Endpoint now restricted to one day of data)
#226 - Delays in WEM data optimized
#228 - Rooftop generation data had trailing 0 values when intervals with generation didn't align (note: you can forward-fill rooftop data with provided forecasts)

Running

Note: Container registry has been switched to ghcr.io

PyPi:

$ pip install -U opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull ghcr.io/opennem/opennem/opennem:3.13

v3.12.0

1 year ago

Features

OpenNEM now calculates all NEM interconnector flows and emissions per-interval, and publishes that data live in the website power series along with emissions and intensity data on the API. Previously the import and export data was the AEMO-provided NET_INTERCHANGE value for each region of the NEM. See #160
OpenNEM Stripes data for both the NEM and WEM now contains historic daily solar generation data #119
Overnight processing of AEMO NEM data has been optimized - it is now published on OpenNEM and available via the API at approximately 8am AEST daily. The goal with the next release will be to bring the publish date even earlier #169
Every station and facility now has an emission intensity factor - #107
A dozen new stations mapped in the NEM, with South Australias temporary generation units re-assigned to their respective stations (#180). For the WEM, the Muja retired units and the various Kwinana units have been corrected. (#84)
AEMO statuses have been remapped where units are automatically graduated from commissioning with generation data to being operational - #152
Many optimizations to back-end processes. One of the most intensive returns - the entire history of generation for a station (as displayed on the facilities page, has now been optimized #113

Fixes

#130 - Weather data now returned in local timezone
#157 - Bug in the total value of rooftop for Australia when summing NEM + SWIS
#145 - Bug where the last value in daily and monthly series data was null
#127 - Make all ids consistent across the JSON outputs and the API
#185 - South Australia imports/exports were inverted
#144 - NEM Tasmania imports and exports had the emission factor from the source inverted displaying an incorrect total emissions and intensity
#156 - Altered method of how gaps are filled in rooftop solar data
#155 - Exception in API output for stations with some parameters. Other API exceptions.
#163 - Null values in energy sums

Thank You

@aonaus for his report and suggestion of migrating away from AEMO NEM net_interchange data for imports and exports and to per-interval flow and emission calculations.
@aleith for providing valuable station and emission information
@andrewbusch7 - on highlighting errors in how gaps were filled in solar data

Running

PyPi:

$ pip install -U opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.12

v3.11.0

1 year ago

Features

OpenNEM price is now volume-weighted (#109)
Facilities updated and kept up-to-date (#79)
Historic per-interval data (generated, price, emissions, demand, etc.) exported for each network (#65) (#68)
Performance of facility pages and API returns dramatically improved (#113)
load_opennem_dataset_from_file and load_opennem_dataset_from_url methods that return structured Pydantic models (see schema.py)
Reduced latency of AEMO regulator data in all returns - does not exceed 30 seconds in 99% of cases.

Fixes

#130 - Weather data now returned in local timezone
#129 - Validators applied to prevent incorrect series lengths in JSON and API outputs
#103 - Facilities committed and commissioning status are preserved from regulator status
#86 - Battery charging and discharging being inverted
#123 - HTTP3/QUIC, Brotli and TLS v1.2 enabled on all frontends
#127 - id names in outputs are consistent across all JSON outputs and API

Interval Data API

The output format for historic interval data is:

https://data.opennem.org.au/v3/stats/historic/weekly/{network_code}/{network_region/year/{year}/week/{week_no}.json

ex.

https://data.opennem.org.au/v3/stats/historic/weekly/NEM/NSW1/year/2022/week/32.json

https://data.opennem.org.au/v3/stats/historic/weekly/WEM/WEM/year/2022/week/32.json

Running

PyPi:

$ pip install opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.11

v3.10.0

1 year ago

Features

Remove dependency on pytz and replaced with zoneinfo. pytz had edge cases that were incorrect (see this blog post)
Upgraded to use Python v3.10 (primarily for use of structural pattern matching to simplify parsers)
Price API endpoint now includes forecasts if parameter provided. (see /stats/price/NEM?forecasts=1) (Issue #66 )
Optimized AEMO NEM parser for nemweb and MMS data. See [documentation]
Full MMS data mirror in JSON on AWS
More intelligent gap filling (#83)

Fixes

#86 - Battery charging and discharging inverted
#77 - APVI gap filling bug - rooftop data on WEM
#76 - Bug with all regions not adding up
#79 - Missing facilities on OpenNEM and updating facilities
#55 - Price and market value output now matches AEMO and also provides OpenNEM price (see docs)
#83 - Finding and filling data gaps
#72 - Rooftop solar sums in daily and monthly aggregates

Running

PyPi:

$ pip install opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.10

v3.9.0

2 years ago

Features

Network flows between network regions now display emissions (example for South Australia)
Price API and other new endpoints (see /stats/price/nem)
New crawler infrastructure (see opennem/crawlers - crawl.py) can now run as AWS Lambdas
Concrete crawler definitions (see example)
Weather backup data from Willy Weather (client code)
Aggregate tables for network flows (See: ORM model)
AEMO MMS parser and schemas for all the primary tables and output into JSON
Status page at https://status.opennem.org.au
Developer documentation at https://developers.opennem.org.au/
New infrastructure setup

Fixes

Timeouts on API requests
Patch old interconnectors and code renames
Double counting of old rooftop data because of backfill
SWIS data consistency and updates
More regular updates from crawlers and faster processing of all outputs for frontend website
Patch to new APVI API
API endpoint for fueltech mix by network and region fixed
Negative emissions removed from sums and flows
Legacy code removed

Running

PyPi:

$ pip install opennem-backend

New OpenNEM cli available:

$ opennem -h

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.9

Upgrading

Apply database migrations:

$ alembic upgrade head

Next Release v3.10

Planned for a fortnight to three weeks. To contribute to the planning of new features or enhancements see the list of issues

v3.8.0

2 years ago

Features

Emissions for flows on NEM network
5 minute price intervals for NEM
Added the Victoria Big Battery and other stations

Fixes

NEM + SWIS were not summing correctly in energy views for all regions on OpenNEM (#15)
Temperature data was misaligned because of timezone issues (parsing BoM timezones) (#16)
Better date parsing in API parameters (#25)
Committed and comissioning stations were not being imported for NEM due to changes to the AEMO excel dumps (#32)
Exports for ElectricityMaps was not showing all of Australia's solar data (#40)
On the Stripes pages on some outputs the latest day was showing as empty/null (#38)
Move to new GitHub package host
Improved test coverage + coverage reports

v3.5.0

3 years ago

FEAT: Multiiple new material views speeding up access queries (including live API queries)
FEAT: Approve multiple stations such as Yandin, Warradarge and Merredin
FEAT: Interconnector flows, interconnectors and net out methods
FEAT: Emission values for all stations, facilities, regions and networks
FEAT: Track government stats in a new stats table starting with AU CPI and Inflation
FEAT: BoM imports of archive data for Australia
FEAT: Alerts infra for new DUIDs
FIX: NEM price data
FIX: Trim time series to all data
FIX: Year turnover fixes with timezones and outputs
META: Update all dependencies

Running

PyPi:

$ pip install opennem

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.5

Upgrading

Run:

$ scripts/upgrade-v3.5.sh

v3.4.0

3 years ago

Largely optimisations and bug fixes from the 3.3.4 release and new stations from AEMO manually approved.

FEAT: Material views for energy and power data per fueltech, network and per station. see mv_fueltech_energy_hour mv_network_energy_hour and mv_network_facility_hour FEAT: Facility API endpoints updated to utilise material views FEAT: New summary outputs of all daily station data and all regions FEAT: 15 new stations manually approved in data set from AEMO REL and AEMO GI latest FEAT: Optimised date parsing improving crawling performance by 30%+ FEAT: New MMS archive crawlers FIX: Dev server and API server reloads would be slow FIX: Organised scripts/ directory with new scripts to manage and run tasks FIX: Join optimisations in ORM models with lazy loading and most and inner joins where required FIX: API outputs always have interval set

Export Output

New monthly and daily summary data for entire network and region history:

Replace NSW1 in above for other NEM regions

Running

PyPi:

$ pip install opennem

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.4

Upgrading

Run:

$ scripts/upgrade-v3.4.sh