Last active
June 11, 2022 00:57
-
-
Save coordt/43a6b9f2b2d6598a3d5b84a95b49ca51 to your computer and use it in GitHub Desktop.
Revisions
-
coordt revised this gist
Jun 10, 2022 . 1 changed file with 61 additions and 10 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,10 +1,13 @@ # Releasing Software ## What is a software release? A *release* is the immutable packaging of a software product. Software releases are an essential component of modern software development and the software development life cycle for two main reasons: - **They are intentional.** Someone decided what changes are required to give value and when the changes are ready for others to use. - **They include context.** Information about the difference between the code at two points is conveyed. Whether it is simply by the version indicating the scope or including additional artifacts such as a changelog, the user has more information than "something changed." The developer plays a key role in a software release because a thoughtful decision is required. ## What is release management? @@ -13,16 +16,20 @@ Active software releases are intentional. The initial automation setup may take - **Release Policies**: The definition of release types, standards, and governance requirements. - **Release Unit:** The set of artifacts released together. - **Release Pipeline**: A repeatable workflow process that includes human and automated activities and follows the release policies. - **Release Value Stream:** Extra processes that add or create value across the release pipeline. - **Pre-releasing:** Ability to release software before finalizing the code for integration testing with other services. - **Parallel development:** The ability for parallel and non-blocking development on the same repository. ## An example release management processs This section provides a concrete example of a working software relase management system. ### Release policies A software release consists of release unit(s) tagged with a version number. #### Version numbering A software release's version number should: - convey some contextual information. @@ -59,21 +66,59 @@ For development releases, only the `DEV` version is incremented. The `MAJOR`, `M ### Release pipeline A git tag on a commit in the main/master branch using the version number as the label indicates a release. The tagged commit ideally should contain all artifacts (changelog, release notes, test reports, package metadata) pertaining to the release, but they may be included in previous commits. _All subsequent commits on that branch are considered part of the next release._ The recommended method is: 1. Development is done in one or more feature branches 2. A pull request for the feature branch is created and reviewed 3. When the code in the pull request is finished and _could_ be released, it is merged into main/master 4. The merged branches are deleted 5. A developer checks out the latest main/master branch 6. The developer runs a release command (see below) to trigger the appropriate release type. We will set up automated tooling below. ### Release value stream There are two parts of the value stream: 1. Building the artifacts required before the tag. 2. Building the artifacts as a result of the tag. The first part requires input from the developer: the "type" of release (major, minor, patch). The method of getting this information should be: - Easy to set up - Easy to maintain - Easy to learn - The easiest way to release - Resilient for errors and mistakes We feel a `Makefile` with commands for releasing major, minor or patch versions fills this role the best, based on the above criteria. The second part is run by a continuous integration platform, such as GitHub Actions or Azure Pipelines. These pipelines can detect new tags in the repository and automate additional steps. ### Pre-releasing The ability to create a non-final software release for beta testing or integration testing is useful to prevent regressions or confirm a bug was fixed in a real-world scenario. When on a feature branch, developers can release a development version. The resulting format is `<BASE VERSION>+BRANCH_NAME-DEV`. For example a development release on a branch named "bug-fix-451" that is branched from version `1.2.3` would release versions `1.2.3+BUGFIX451-0` and `1.2.3+BUGFIX451-1` and so on. Including the branch name and the version the branch is based off allows parallel development without conflicts. ### Parallel development Working on a feature should not block fixing a bug in the latest release. Development releases do not conflict due to naming conventions. Conflicts for final release is managed using git itself via the automated tools. Since the release pipeline allows the developer to specify the release type and not the version, the tools pull down the latest commits and increments the version according to the latest data in git. If two developers tried to do a final release at the exact same time, the first one who pushes to origin would cause a git error for the other when they pushed. ## Version management implementation ### Store the version in your package @@ -372,3 +417,9 @@ to ``` The three additional lines makes sure that your branch is up to date, generates the changelog and then brings up the changelog for editing (if your `EDITOR` environment variable is set). ## Continuous integration and releasing ## Nested PRs for parallel feature development -
coordt revised this gist
Jun 9, 2022 . 1 changed file with 17 additions and 17 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -4,7 +4,7 @@ A *release* is the immutable packaging of a software product. All software used by more than one person is released. You can release passively by giving others access to every change as it occurs. You can release actively by deciding what changes are required and when the changes are ready for others to use. Active software releases are intentional. The initial automation setup may take longer than a passive release, but subsequent releases to do not take more or less time than passive software releases; they just include the intent of what value the release gives to its users and the statement "this is ready." Active software releases are an essential component of modern software development and the software development life cycle. ## What is release management? @@ -99,7 +99,7 @@ This allows us to: ### Update our packaging information Our packaging will reference the `__version__` when it builds in the `[metadata]` section to the `setup.cfg` file in our repo: **setup.cfg:** @@ -117,7 +117,7 @@ The `attr: mypackage.__version__` on line 3 tells `setuptools` to use the `__ver ## Release version management via tools We will do the following to manage our versions: - Use [bump2version](https://pypi.org/project/bump2version/) for version management. - Add configuration to `setup.cfg` to configure how to manage our version progressions. @@ -153,25 +153,25 @@ message = Version updated from {current_version} to {new_version} Here is what the lines mean: - **`current_version:`** The current version of the software package before bumping. [Documentation.](https://github.com/c4urself/bump2version#current_version) - **`commit:`** Create a commit after bumping. [Documentation.](https://github.com/c4urself/bump2version#commit--true--false) - **`commit_args:`** Extra arguments to pass to commit command. `--no-verify` prevents git commit hooks from running during commit. [Documentation.](https://github.com/c4urself/bump2version#commit_args-) - **`tag:`** Create a tag named as the `tag_name` setting. [Documentation.](https://github.com/c4urself/bump2version#tag--true--false) - **`tag_name:`** Set the name of the tag to the new version. [Documentation.](https://github.com/c4urself/bump2version#tag_name-) - **`parse:`** Regular expression (using [Python regular expression syntax](https://docs.python.org/3/library/re.html#regular-expression-syntax)) on how to find and parse the version string. - **`serialize:`** Template specifying how to serialize the version parts back to a version string. `bumpversion` will try the serialization formats beginning with the first and choose the last one where all non-zero values can be represented. In this case, it will render the version as `{major}.{minor}.{patch}` if the `dev` value is `0`. `{$BRANCH_NAME}` references an environment variable that is covered later. [Documentation.](https://github.com/c4urself/bump2version#serialize-) - **`message:`** The commit message to use when creating a commit. [Documentation.](https://github.com/c4urself/bump2version#message-) - **`[bumpversion:file:mypackage/__init__.py]:`** This tells `bumpversion` to look in the `mypackage/__init__.py` for the current version string update the string to the new version. [Documentation.](https://github.com/c4urself/bump2version#configuration-file----file-specific-configuration) ### Automation with make @@ -237,7 +237,7 @@ get-version: # Sets the value after release-version to the VERSION The variables at the top are used to configure the behavior: - **`.DEFAULT_GOAL`** The target to run when no target is passed. If someone simply runs `make` it will show the help text. - **`BRANCH_NAME`** The current branch name. - **`SHORT_BRANCH_NAME`** The first 20 characters of the branch name. Keeps versions from getting too long. - **`PRIMARY_BRANCH_NAME`** The name of the branch for doing non-development releases. @@ -256,7 +256,7 @@ The Makefile's [phony targets](https://www.gnu.org/software/make/manual/html_nod ## Updating a changelog when releasing Now that we have a set of commands to automate a version change and software release, let's add additional value by updating a changelog using the git commits as the basis and the version-tagged commits as the basis for grouping commits into releases. ### generate-changelog configuration @@ -288,7 +288,7 @@ We are adding the `repo_url` variable so we can add additional links in our temp ### Overriding the changelog templates We are going to add links to each commit and a link to a diff between releases. Make sure the default location for extra templates `.github/changelog_templates` exists: ```shell mkdir -p .github/changelog_templates @@ -323,7 +323,7 @@ Line 3 is the addition to this template. It uses the `repo_url` variable to make ### Update bumpversion configuration When the changelog is generated, the latest commits are grouped under the title "`Unreleased`" and the diff link references the difference between the previous version and `HEAD`. These two things need replacing with the new version. Bumpversion can do this for us. Add the following lines to `setup.cfg`: @@ -371,4 +371,4 @@ to git push --tags; ``` The three additional lines makes sure that your branch is up to date, generates the changelog and then brings up the changelog for editing (if your `EDITOR` environment variable is set). -
coordt created this gist
Jun 7, 2022 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,374 @@ # Release tooling ## What is a software release? A *release* is the immutable packaging of a software product. All software used by more than one person is released. You can release passively by giving others access to every change as it occurs. You can release actively by deciding what changes are required and when the changes are ready for others to use. Active software releases are intentional. They do not take more or less time than passive software releases; they just include the intent of what value the release gives to its users and the statement "this is ready." Active software releases are an essential component of modern software development and the software development life cycle. ## What is release management? *Release management* is how releasing software fits into your development process. Some things to consider: - **Release Policies**: The definition of release types, standards, and governance requirements. - **Release Unit:** The set of artifacts released together. - **Release Pipeline**: A repeatable workflow process that includes human and automated activities and follows the release policies. - **Release Value Stream:** Extra processes that add or create value across the release pipeline. - **Pre-releasing:** Ability to release software before finalizing the code for integration testing with other services. - **Parallel development:** The ability for parallel and non-blocking development on the same repository. For our purposes we will define these below. ### Release policies A software release's version number should: - convey some contextual information. - easily link the resulting artifacts and the git commit used to generate the artifacts. #### Versioning methods Software that delivers features ad-hoc, will use a [semantic versioning](https://semver.org) system. The version will provide some contextual information about each release's difference from another version. Software that delivers regular, periodic updates, will use a [calendar versioning](https://calver.org) system. The version will provide information about how current an installed version is from the current version. Software that does not satisfy the above conditions should use the first 7 characters of the git commit SHA for version reporting. A form of calendar version could also be used if the date-time contextual information is useful. #### Linking the version to the git commit For semantic and calendar versioning, tag the git commit that contains all the finished code and documentation and should be used to generate the final artifacts with the version number. Continuous integration processes can use the tag as a trigger to automate the packaging. #### Semantic versioning We will use the format `MAJOR.MINOR.PATCH` for final releases and `MAJOR.MINOR.PATCH+BRANCH_NAME-DEV` for development releases while on feature branches. For final releases, increment the: - `PATCH` version when you make backwards compatible bug fixes. - `MINOR` version when you add functionality in a backwards compatible manner - `MAJOR` version when you make incompatible API changes For development releases, only the `DEV` version is incremented. The `MAJOR`, `MINOR`, and `PATCH` segments indicate the version this development release is based on. ### Release unit - Package for libraries - Docker container for services ### Release pipeline This includes the tooling we will set up below and some continuous integration to package the release. ### Release value stream Once we have an easy way to manage and trigger a release, we will add additional things. ### Pre-releasing The ability to create a non-final software release for beta testing or integration testing is useful to prevent regressions or confirm a bug was fixed in a real-world scenario. ### Parallel development Working on a feature should not block fixing a bug in the latest release. ## Initial setup ### Store the version in your package We are going to store the version in our code for convenience. We'll add a `__version__` attribute to our package for this. Modify the `__init__.py` file in your package such as for `mypackage`: **mypackage/\__init__.py:** ``` python """This is my package; there are many like it but this one is mine.""" __version__: str = "0.1.0" ``` This allows us to: ```pycon >>> import mypackage >>> print(mypackage.__version__) 0.1.0 ``` ### Update our packaging information We can have our packaging reference that version when it build by adding or updating a `[metadata]` section to the `setup.cfg` file in your repo: **setup.cfg:** ```ini [metadata] name = mypackage version = attr: mypackage.__version__ description = "This is my package; there are many like it but this one is mine." long_description = file:README.md long_description_content_type = text/markdown ``` The `attr: mypackage.__version__` on line 3 tells `setuptools` to use the `__version__` attribute that we set up previously, making it easier to manage our release version. ## Release version management via tools We will do the following in order to manage our versions: - Use [bump2version](https://pypi.org/project/bump2version/) for version management. - Add configuration to `setup.cfg` to configure how to manage our version progressions. - Add a `Makefile` to automate the process ### Bumpversion (bump2version) Bump version is a small command line tool to simplify releasing software by updating all version strings in your source code by the correct increment. It also creates commits and tags. First you need to add `bump2version` to your _development_ requirements. > Note > > If you don't manage development requirements separately from production requirements, then you can put it in the production requirements; it just isn't required for your code to work. **setup.cfg:** ```ini [bumpversion] current_version = 0.1.0 commit = True commit_args = --no-verify tag = True tag_name = {new_version} parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\+[-_\w]+?-(?P<dev>\d+))? serialize = {major}.{minor}.{patch}+{$BRANCH_NAME}-{dev} {major}.{minor}.{patch} message = Version updated from {current_version} to {new_version} [bumpversion:file:mypackage/__init__.py] ``` Here is what the lines mean: - **`current_version`** The current version of the software package before bumping. [Documentation.](https://github.com/c4urself/bump2version#current_version) - **`commit`** Create a commit after bumping. [Documentation.](https://github.com/c4urself/bump2version#commit--true--false) - **`commit_args`** Extra arguments to pass to commit command. `--no-verify` prevents git commit hooks from running during commit. [Documentation.](https://github.com/c4urself/bump2version#commit_args-) - **`tag`** Create a tag named as the `tag_name` setting. [Documentation.](https://github.com/c4urself/bump2version#tag--true--false) - **`tag_name`** Set the name of the tag to the new version. [Documentation.](https://github.com/c4urself/bump2version#tag_name-) - **`parse`** Regular expression (using [Python regular expression syntax](https://docs.python.org/3/library/re.html#regular-expression-syntax)) on how to find and parse the version string. - **`serialize`** Template specifying how to serialize the version parts back to a version string. `bumpversion` will try the serialization formats beginning with the first and choose the last one where all non-zero values can be represented. In this case, it will render the version as `{major}.{minor}.{patch}` if the `dev` value is `0`. `{$BRANCH_NAME}` references an environment variable that is covered later. [Documentation.](https://github.com/c4urself/bump2version#serialize-) - **`message`** The commit message to use when creating a commit. [Documentation.](https://github.com/c4urself/bump2version#message-) - **`[bumpversion:file:mypackage/__init__.py]`** This tells `bumpversion` to look in the `mypackage/__init__.py` for the current version string update the string to the new version. [Documentation.](https://github.com/c4urself/bump2version#configuration-file----file-specific-configuration) ### Automation with make This creates a set of commands: - **`make release-dev`** Release a new development version *only if you are on a feature branch.* (1.1.1 to 1.1.1+branchname-1) - **`make release-patch`** Release a new patch version. (1.1.1 to 1.1.2) - **`make release-minor`** Release a new minor version. (1.1.1 to 1.2.0) - **`make release-major`** Release a new major version. (1.1.1 to 2.0.0) - **`make release-version <new version number>`** Set version to `<new version number>` and release. **Makefile:** ```makefile .DEFAULT_GOAL := help BRANCH_NAME := $(shell git rev-parse --abbrev-ref HEAD) SHORT_BRANCH_NAME := $(shell echo $(BRANCH_NAME) | cut -c 1-20) PRIMARY_BRANCH_NAME := master BUMPVERSION_OPTS := help: @grep '^[a-zA-Z]' $(MAKEFILE_LIST) | sort | awk -F ':.*?## ' 'NF==2 {printf "\033[36m %-25s\033[0m %s\n", $$1, $$2}' release-dev: RELEASE_KIND := dev release-dev: do-release ## Release a new development version: 1.1.1 -> 1.1.1+branchname-1 release-patch: RELEASE_KIND := patch release-patch: do-release ## Release a new patch version: 1.1.1 -> 1.1.2 release-minor: RELEASE_KIND := minor release-minor: do-release ## Release a new minor version: 1.1.1 -> 1.2.0 release-major: RELEASE_KIND := major release-major: do-release ## Release a new major version: 1.1.1 -> 2.0.0 release-version: get-version do-release ## Release a specific version: release-version 1.2.3 # # Helper targets. Not meant to use directly # do-release: @if [[ "$(BRANCH_NAME)" == "$(PRIMARY_BRANCH_NAME)" ]]; then \ if [[ "$(RELEASE_KIND)" == "dev" ]]; then \ echo "Error! Can't bump $(RELEASE_KIND) while on the $(PRIMARY_BRANCH_NAME) branch."; \ exit; \ fi; \ elif [[ "$(RELEASE_KIND)" != "dev" ]]; then \ echo "Error! Must be on the $(PRIMARY_BRANCH_NAME) branch to bump $(RELEASE_KIND)."; \ exit; \ fi; \ export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \ git push origin $(BRANCH_NAME); \ git push --tags; get-version: # Sets the value after release-version to the VERSION $(eval VERSION := $(filter-out release-version,$(MAKECMDGOALS))) $(eval BUMPVERSION_OPTS := --new-version=$(VERSION)) %: # NO-OP for unrecognized rules @: ``` The variables at the top are used to configure the behavior: - **`.DEFAULT_GOAL`** The target to run when no target is passed. If one simply runs `make` it will show the help text. - **`BRANCH_NAME`** The current branch name. - **`SHORT_BRANCH_NAME`** The first 20 characters of the branch name. Keeps versions from getting too long. - **`PRIMARY_BRANCH_NAME`** The name of the branch for doing non-development releases. - **`BUMPVERSION_OPTS`** Additional options passed to `bumpversion`. The Makefile's [phony targets](https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets) require more explanation. Some of the targets are not meant to be called directly. - **`help`** (lines 7-8) parses and displays the text after `##` comments. - **`release-*`** (lines 10-22) creates the release type specified in the name. - **`do-release`** (lines 28-40) does the actual release work. Lines 29-37 check the current branch to determine if you can do a development release or not. Lines 38-40 run the commands to change the version, commit, tag and push the result. *Don't call this target directly!* - **`get-version`** (lines 42-44) parses the version from the `make` command and updates the `BUMPVERESION_OPTS` variable to the correct option for `bump2version`. - **`%`** (lines 46-47) does nothing if an unrecognized target is passed to the `make` command. This is required so that the version passed in with `release-version` doesn't generate an error. ## Updating a changelog when releasing Now that we have a set of commands to automate a version change and software release, lets add additional value by updating a changelog using the git commits as the basis and the version-tagged commits as the basis for grouping commits into releases. ### generate-changelog configuration Add `generate-changelog` to your development requirements. Then install it. ```shell pip install generate-changelog ``` You can generate the entire default configuration file by running: ```shell generate-changelog --generate-config ``` But you can also just create a configuration that overrides the default values. We are going to do that here to highlight the changes we will make. **.changelog-config.yaml:** ```yaml variables: changelog_filename: CHANGELOG.md repo_url: https://github.com/<username>/mypackage ``` The default configuration sets the `changelog_filename` variable. Since we are overriding the `variables` section, we need to include it. We are adding the `repo_url` variable so we can add additional links in our templates. ### Overriding the changelog templates We are going to add links to each commit and a link to a diff between releases. Make sure the default location for extra templates `.github/changelog_templates` is there: ```shell mkdir -p .github/changelog_templates ``` Now add two files, `commit.md.jinja` and `version_heading.md.jinja`: **commit.md.jinja:** ```jinja - {{ commit.summary }} [{{ commit.short_sha }}]({{ repo_url }}/commit/{{ commit.sha }}) {{ commit.body|indent(2, first=True) }} {% for key, val in commit.metadata["trailers"].items() %} {% if key not in VALID_AUTHOR_TOKENS %} **{{ key }}:** {{ val|join(", ") }} {% endif %} {% endfor %} ``` The change in this template is on the first line. After the commit summary, the short git SHA is made into a link, using the `repo_url` variable as the root. **version_heading.md.jinja:** ```jinja ## {{ version.label }} ({{ version.date_time.strftime("%Y-%m-%d") }}) [Compare the full difference.]({{ repo_url }}/compare/{{ version.previous_tag }}...{{ version.tag }}) ``` Line 3 is the addition to this template. It uses the `repo_url` variable to make a link to the difference between this version and the previous. ### Update bumpversion configuration When the changelog is generated, the latest commits are grouped under the title `Unreleased` and the diff link references the difference between the previous version and `HEAD`. These two things need replacing with the new version. Bumpversion can do this for us. Add the following lines to `setup.cfg`: **setup.cfg addition:** ```ini [bumpversion:file(version heading):CHANGELOG.md] search = Unreleased [bumpversion:file(diff link):CHANGELOG.md] search = {current_version}...HEAD replace = {current_version}...{new_version} ``` ### Add changelog generation to release automation We just need to make a few changes to our `Makefile` to enable changelog generation and optional editing of the changelog before commit and release. **Makefile:** Add (after line 5) ```makefile EDIT_CHANGELOG := $(shell if [[ -n $$EDITOR ]] ; then echo "$$EDITOR CHANGELOG.md" ; else echo "true" ; fi) ``` This adds the command to edit the changelog if the `EDITOR` environment variable is set when `$(call EDIT_CHANGELOG)` is run within the `do-release` target. If `EDITOR` is not set, it puts in the `true` command, which causes it to move to the next step. In the `do-release` target, change the lines: ```Makefile export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \ git push origin $(BRANCH_NAME); \ git push --tags; ``` to ```Makefile git fetch -p --all; \ generate-changelog; \ $(call EDIT_CHANGELOG); \ export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \ git push origin $(BRANCH_NAME); \ git push --tags; ``` The three additional lines makes sure that your branch is up-to-date, generates the changelog and then brings up the changelog for editing (if your `EDITOR` environment variable is set).