TvgrabpyAPI Versions Save

Adding cookie support and alternate useragent list for the updated humo source fixed some bugs;

• disable_source in the configuration file was called before the sources where known from the main json file,
• the detail source functionality was incompletely disabled on a disabled source.

With two major fixes:

• For sources like tvgids.tv that use an offset in their url that gets updated somwhere in the early morning resulting in wrong programme dates.
• For sources like npo.nl that need data from a detail page to determine a proper stop-time. They now if set as prime_source do grab the detailpages prior to starting the merge process. You can disable this behavior, that can lengthen a grab, by either making sure that this source is not set as prime_source or by setting the new option "pre_merge" in your configuration either global or for single channels to False.

With some minor fixes especially concerning the database With two new start-up options: --clear-source <sourceid> --clear-channel <chanid> to either remove all programme data from one source or one channel from the database. You can use it to remove the faulty data from tvgids.tv while leaving all other data intact. tv_grab_nl3.py --clear-source 1

And a fix on the --use-only-cache option that also conflicts with the new "pre_merge" mode. So using --use-only-cache disables the new "pre_merge" mode.

Some minor updates

This version mainly implements all the changes in DataTreeGrab v1.4 which can potentially more then double the speed.

I saw my average fetch-time of 68 min with DataTreeGrab 1.3.2 reduce to 48 min with DataTreeGrab 1.3.3 and now to 23.5 min.! (65%)

Also on some points the fetch process has been streamlined. Further some minor bug-fixes have been implemented. A review of the complete merge process will follow in a next version.

• With a fix on a race-condition between the source and channel threads that in specific situations can lead to channels giving up to early waiting on a source delivering its data
• A new link function to fix an issue with titles on oorboekje.nl. The updated datadef implementing this function will be released when this version reaches stability.
• A better fix on quotes in attributes
• And some other small cosmetic fixes

Here the first release deemed stable. See the Wiki for more details
If you already had any of the previous beta releases installed, it is advised, as database data structures have changed, to run --clear-cache and --clear-ttvdb. Missing fields and tables will get added but outdated fields are not removed and could possibly cause an error. Those commands will dump and rebuild the tables.
On future changes to the data structure we will also handle outdated field removal etc., so existing data won't get lost.
To prevent the first runs on an empty database from running to long, the number of days retrieved from internet has for most sources been limited to 4. So it will take a few runs/days to come up to the maximum of 14 days. If you run ones a day that will take 6 runs ( the first day 4 and every next day the days in cache, 3 new + the current day)