Opennem Versions Save

FEAT: API tasks endpoint to queue crawl and worker tasks and check queues FEAT: API prometheus /metrics endpoint FEAT: Local scrapyd for development FEAT: scrapyd control library in opennem.utils.scrapyd FIX: Handle negative small float values in clean_float

FIX: Small floats would be stringified to exp. notation and then back to the wrong float value in some cases FEAT: Scrapyd queue script

Changelog

FEAT: Interconnectors and region flow data on NEM FEAT: More emission mappings FEAT: APVI rooftop data now a separate network FEAT: Can now be accessed as a library opennem and tools used separately FEAT: More export maps for opennem.org.au FIX: Double-counting of NEM rooftop data FIX: Additional database indicies

Running

PyPi:

$ pip install opennem

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.2

Changelog

FEAT: Pollution maps for AU, NEM and WEM and regions in each FEAT: v3 export paths in opennem.org.au (behind feature flag) FEAT: AEMO rooftop data for NEM now in v3 outputs FEAT: MMS data from AEMO imported into production opennem.org.au behind v3 paths

Running

PyPi:

$ pip install opennem

Docker:

$  docker pull docker.pkg.github.com/opennem/opennem/opennem:3.2

On dockerhub as opennem/opennem:3.2

https://hub.docker.com/r/opennem/opennem

• FEAT: Export maps for data.opennem defined as schema and exported to metadata.json in root of object store
• FIX: Better coalescing of existing energy data with energy_sum to save calculations in WEM
• FIX: WEM energy data back to 2006 for SWIS
• FEAT: Compact formatting of output numbers and new numbers module for handling
• FEAT: Australia-wide energy + power statistics
• FIX: Cache common queries like Scada date ranges for networks and network regions
• FIX: Remove slow xpath parsing in spiders on directory listings
• META: Update all external deps and dev deps

• FEAT: Notifications module for AEMO data with slack as first notification type

• FEAT: Full support for SWIS network with crawlers and outputs
• FIX: Rooftop Solar in SWIS only uses two-digit postcodes that are approx. in SWIS
• FEAT: New bulk inserter for CSV records from AEMO
• FEAT: energy_sum SQL functions to calculate energy totals with support for variable interval sizes
• FEAT: APVI rooftop solar data
• FEAT: New API output schema format and export JSON's on data.opennem.org.au

• FIX: aiofiles dependency for static file hosting in API
• FIX: WEM monitor checks fieldnames from DictReader rather than attempting to read the length of a generator