Automating packaging and software distribution on macOS.
Since AutoPkg 3.0 RC2, I've rewritten the entire Recipe loading logic. This is currently stored in the dev_new_recipe_logic
branch, and is being tested for merging into the dev
branch here in https://github.com/autopkg/autopkg/pull/926. Since this is not yet merged into dev
, GitHub's automatic comparison probably won't do the right thing and generate proper comparison notes.
For now, I am keeping track of the major issues people are experiencing on this Trello board: https://trello.com/b/NlGjpVJm/autopkg-30 (I will also probably move that somewhere else in the future to reduce the spread of information, which is already fragmented enough).
I have successfully tested the rewritten recipe logic on a bunch of recipes, but I need a lot more data, especially from people whose setups don't look like mine. AutoPkg's beauty is that it's flexible enough to run in all kinds of environments and all kinds of ways, and my curse is that I'm largely insulated from the effects of changes on those - so I need all of you to test it for me and tell what's not working.
The ultimate goal here, of course, is to get the new Recipe Map and the new Recipe loading logic finalized so that I can start tackling the major blockers identified in issues already - where it's not loading recipes the way it did previously. The release of AutoPkg 3.0 should essentially be a big performance improvement without any major changes to the way people use AutoPkg (something I haven't always been particularly good at).
Please, test, and generate Discussions or file Issues for things that are blocking you so I can track them and we can work on them accordingly.
Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC2...v3.0.0RC3 See also https://github.com/autopkg/autopkg/pull/926 for the full change in the recipe logic rewrite.
Full Changelog: https://github.com/autopkg/autopkg/compare/v3.0.0RC1...v3.0.0RC2
ISSUES:
Given the massive number of fundamental changes in AutoPkg behavior, we have decided to promote this version bump to 3.0.0.
Full Changelog from 2.7.2: https://github.com/autopkg/autopkg/compare/master...v3.0.0RC1
The Github integrations were previously based on curling the GitHub API directly. For various reasons, that's become both unwieldy and unreliable, especially with how much GitHub has applied rate limiting to API calls from unauthenticated sources. The entire stack has been rewritten to rely on the PyGithub module instead.
autopkglib.github
now provides a utility class GitHubSession
that will store information about a given repo (as well as the AutoPkg repo and orgs itself by default), and can be used as an interface for working with the PyGithub module. Note that the PyGithub module does require installation of another module, so you will need to use this RC's version of Python (which includes it), or do a pip install on your own, in order to support the new functionality.
GitHubReleasesInfoProvider processor has also been completely rewritten to use the new module and class.
As described in the previous 2.8.1 RC6's notes:
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH
. The old GITHUB_TOKEN
preference key is no longer used anywhere. You must now store your GH token in a flat text file.
Known issues: the current implementation of the code does not do any rate limit checking or backoff. If you are denied access to the API calls because of rate limits or lack of access, you will get a Python traceback (which is still similar to the behavior from before, but now the error does not come from curl). I hope to add a little bit better of an experience for API rate limiting issues in a future version of AutoPkg.
AutoPkg no longer searches through the folders on disk every time it looks for a recipe or processor. Instead, it now builds a recipe map (stored on disk as a JSON file) that tells AutoPkg where to find recipes by shortname and identifier. The recipe map contains four top level keys:
Each top level key contains a list of key/value pairs corresponding to the full path of a recipe file. Example:
"identifiers": {
"com.github.autopkg.AutoPkg1GitMaster": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/AutoPkg/AutoPkg1GitMaster.pkg.recipe",
...
"shortnames": {
...
"Adium.download": "/Users/nmcspadden/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes/Adium/Adium.download.recipe",
...
This recipe map is automatically built whenever you add or delete a repo, or create a new recipe. The recipe map automatically maps everything in your RECIPE_SEARCH_DIRS preferences. The map is consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos).
For existing clients, simply re-adding an existing repo will build your map: autopkg repo-add recipes
IMPORTANT NOTE: Searching for recipes on disk no longer happens - only the recipe map is consulted! If you experience issues with locating recipes, please file a Github Issue with steps to reproduce the problem.
This has no effect on AutoPkg usage whatsoever, but it's an important note for anyone familiar with the code. Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.
autopkglib.common
contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs
has all the Preference objects and getting/saving preferences code.
WARNING: if you have any custom processors that loaded functions from autopkglib, such as log
or is_mac()
, they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed. If you are a custom processor author and are not sure how to address this, please contact the AutoPkg maintainers and/or ask in Slack for advice.
More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once the logic is fully rewritten.
https://github.com/autopkg/autopkg/compare/v2.8.1RC5...dev?expand=1
The Recipe map is now consulted only on verbs that require it (such as anything involving listing, auditing, generating, or handling recipes and repos). The Recipe map is built when you add or remove a repo, or if you generate a new recipe.
If you do not have a recipe map, or the recipe map is invalid when it is needed, autopkg now tells you what to do next (add or remove a repo to fix it). In the future, I'll add a preference or a command line argument to automatically rebuild vs. failing. (In a CI environment, you probably want it to fail rather than to attempt to build one, as it may have incomplete state at that point. In a regular desktop environment, you probably want it to automatically build and proceed).
Rather than throw everything into primarily two files (autopkg and init.py), it's now separated into some smaller libraries - at the moment, prefs and common.
autopkglib.common
contains some basic common functions - determining platform, and all default path constants.
autopkglib.prefs
has all the Preference objects and getting/saving preferences code.
WARNING: if you have any custom processors that loaded functions from autopkglib, such as log
or is_mac()
, they will need to be adjusted for this change in order to work. You may need to wrap your import in a try/except block while we make this transition. All Core processors have already been fixed.
More sub-libraries will come as I continue to break the monolith apart and add more tests. AutoPkg still generally assumes "autopkglib" is one big folder of processors, so the Core processors will likely move to their own subfolder once I fix the logic everywhere.
The only option now for reading a GitHub token is via the preference GITHUB_TOKEN_PATH. The old GITHUB_TOKEN preference key is no longer used anywhere. You must now store your GH token in a flat text file.
The GitHub logic is being completely rewritten in a different branch, so I've only barely touched it here, and there may be things that break. I don't intend to spend any more effort on the existing GitHub code except to keep it from breaking as I touch things around it. Once this dev branch is done, I'll merge in those changes and work on that next. In the meantime, if you do have existing issues with anything GitHub-related specifically, please file an Issue and I'll make sure that is tested against the new branch.
The recipe map is now created automatically when you use any autopkg verb aside from "help" or "version". This also handles situations where we attempted to read an invalid recipe map (such as would be created in between RC3 and RC4) without rebuilding one first.
This makes the autopkg recipe search feature much more seamless, and is a lot closer to how the existing recipe location mechanism works in the release version (2.7.2).
Fixes #874.
Fixes:
Known Issues:
~/Library/AutoPkg
, others convert to an absolute path upon preference storage like /Users/nmscpadden/Library/AutoPkg
.Fixes:
New features:
Fixes:
Full Changelog: https://github.com/autopkg/autopkg/compare/v2.7.1..v2.7.2