Skip to content

Instantly share code, notes, and snippets.

@maelvls
Last active February 23, 2025 20:02
Show Gist options
  • Save maelvls/068af21911c7debc4655cdaa41bbf092 to your computer and use it in GitHub Desktop.
Save maelvls/068af21911c7debc4655cdaa41bbf092 to your computer and use it in GitHub Desktop.

Revisions

  1. maelvls revised this gist May 25, 2019. No changes.
  2. maelvls revised this gist Oct 4, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions How-to-automate-build-bottles-your-homebrew-tap.md
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,8 @@
    How to automate the build of bottles on your Homebrew tap
    ========================================================

    Note on Oct 4, 2018: due to a [change](https://github.com/Homebrew/brew/commit/59c858a8e8362e7cdc063e107d0360be9dc85d0e#diff-04e700045721a197c725bc15b7f2a173) in Homebrew's `brew test-bot` behaviour, the user must set `HOMEBREW_TRAVIS_CI` and `HOMEBREW_TRAVIS_SUDO` appropriately (it was previously using Travis-CI-provided `TRAVIS` and `TRAVIS_SUDO`).

    This tutorial is a follow-up to the discussion we had on https://github.com/davidchall/homebrew-hep/pull/114.
    It relies on a fork of the `test-bot` provided by davidchall; you can get it with `brew tap maelvalais/test-bot`.
    First:
  3. maelvls revised this gist Sep 3, 2018. 1 changed file with 59 additions and 28 deletions.
    87 changes: 59 additions & 28 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -14,37 +14,48 @@ language: ruby
    if: tag IS blank AND NOT type IN (push)
    env:
    global:
    - HOMEBREW_LOGS=/tmp # Prevent ~/.cache/Homebrew/Logs from being rebuilt
    - HOMEBREW_NO_AUTO_UPDATE=yes # Prevents redundant auto-updates from brew

    ###### Bintray configuration ######
    - TAP_BOTTLE_ORG=touist # Used in this script only
    - TAP_BOTTLE_ROOT_URL=https://dl.bintray.com/$TAP_BOTTLE_ORG/bottles-touist
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist
    - AWS_REGION=eu-west-1 # needed because the aws-sdk-s3 gem errors without it
    # HOMEBREW_BINTRAY_KEY # Secret variable set in Travis CI

    ###### AWS S3 configuration ######
    - AWS_REGION=eu-west-1 # aws-sdk-s3 gem errors without it
    - BUCKET=homebrew-touist-travis
    # Beyond these global variables, you also need to set some secret variables
    # in your travis-ci settings:
    # - AWS_ACCESS_KEY_ID
    # - AWS_SECRET_ACCESS_KEY (for the AWS S3 bucket)
    # - GITHUB_TOKEN (for pushing bottle DSL commits)
    # - HOMEBREW_BINTRAY_KEY (for the Bintray repo)

    # (1) remember tu use 'brew cleanup -s' at the end of the travis script;
    # AWS_ACCESS_KEY_ID # Secret variable set in Travis CI
    # AWS_SECRET_ACCESS_KEY # Secret variable set in Travis CI

    ###### Github configuration ######
    # This config allows test-bot to push tags (eg. pr-1234) and DSL commits.
    # They will be pushed to github.com/$GITHUB_USER/homebrew-%TAP%.
    - GITHUB_USER=touist
    # GITHUB_TOKEN # (Secret) For pushing bottle DSL commits

    # (1) remember to use 'brew cleanup -s' at the end of the travis script;
    # otherwise, the $HOME/Library/Caches/Homebrew folder will be enormous
    # (~2GB for the 4 jobs). This is because this folder is already present on
    # osx images and it is big.
    # (2) I also removed $HOME/.cache/Homebrew (cache of Linuxbrew) because it
    # was also storing some logs, making the cache rebuilt every time.
    # (2) For a time, I had removed $HOME/.cache/Homebrew (cache of Linuxbrew)
    # because it was also storing some logs in $HOME/.cache/Homebrew/Logs, making
    # the cache rebuilt every time.
    cache:
    directories:
    - "$HOME/.cache/pip"
    - "$HOME/.gem/ruby"
    #- "$HOME/Library/Caches/Homebrew" # (1)
    #- "$HOME/.cache/Homebrew" # (2)
    - "$HOME/Library/Caches/Homebrew" # (1)
    - "$HOME/.cache/Homebrew" # (2)

    install:
    # the official test-bot won't let you run inside TravisCI, so we use
    # davidchall's one. David's test-bot cannot push the commit using
    # Oauth github + https (only ssh) so I use my own.
    - brew tap maelvalais/test-bot
    # Install the ruby AWS gem so that I can upload bottles to S3
    - sudo gem install aws-sdk-s3
    - rvm default do gem install aws-sdk-s3
    # IMPORTANT STEP: link the tap inside brew to our current travis-cloned tap
    # Step: 1) create the intermediate folders <user>/<repo> so that
    # we can 2) remove <repo> and 3) replace it with a sym link
    @@ -78,23 +89,24 @@ script:
    if [[ $TRAVIS_EVENT_TYPE =~ cron|api ]]; then
    brew install *.rb && brew test *.rb && brew linkage --test *.rb || exit 123
    fi
    - if [[ $TRAVIS_EVENT_TYPE =~ pull_request ]]; then brew test-bot; fi
    - if [[ $TRAVIS_EVENT_TYPE =~ pull_request ]]; then brew test-bot --root-url=$TAP_BOTTLE_ROOT_URL; fi
    - ls *.bottle*.* || echo "==> No bottle created here"

    jobs:
    include:
    - &run-osx
    os: osx
    osx_image: xcode9.3beta
    env: OS=sierra-10.13
    osx_image: xcode9.4
    env: OS=high_sierra-10.13
    # We must use 'rvm: system' because the system ruby doesn't rely on
    # Homebrew, which allows us to reinstall Homebrew without having a
    # damaged ruby. We need ruby because we use the 'aws-sdk-s3' gem.
    rvm: system
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    # First we uninstall any outdated versions of xquartz; otherwise,
    # Homebrew will complain of of older version (2.9.7) being outdated
    # even though we install a new version.
    # even though we install a new version. Remember that
    # /usr/local/Caskroom will also be deleted below.
    - brew cask outdated xquartz || brew cask uninstall xquartz
    # Three reasons not to use the /usr/local and Homebrew installations
    # that come in the Travis CI images:
    @@ -106,16 +118,26 @@ jobs:
    # Homebrew anyway, so why not start fresh.
    - mkdir ~/usr_local && sudo mv /usr/local/* ~/usr_local
    - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    - brew cask reinstall xquartz

    # Either xquartz was already installed at its latest version or it has
    # been uninstalled. First, we put the cask back in place if it has not
    # been uninstalled.
    - |
    if [ -d ~/usr_local/Caskroom/xquartz ];
    then sudo mv ~/usr_local/Caskroom /usr/local/Caskroom;
    else travis_retry brew cask install xquartz; fi
    # We still need the Homebrew ruby on macOS 10.12 and 10.11 because the
    # system ruby uses an old openssl version ("tlsv1 alert protocol").
    - travis_retry brew install libyaml gmp [email protected] openssl

    - <<: *run-osx
    os: osx
    osx_image: xcode8.3
    env: OS=high_sierra-10.12
    osx_image: xcode9.2
    env: OS=sierra-10.12

    - <<: *run-osx
    os: osx
    osx_image: xcode7.3
    osx_image: xcode8
    env: OS=el_capitan-10.11

    - &run-on-linux
    @@ -140,8 +162,7 @@ jobs:
    # Fix a `brew doctor` error on "config" scripts for some reason
    - sudo rm -f /home/travis/.phpenv/shims/php-config
    /opt/pyenv/shims/*-config /usr/local/clang-*/bin/llvm-config
    before_cache:
    - brew cleanup

    # Uploading to bintray (as unpublished bottle) and pushing a tag to the
    # repo are only enabled on pull_requests.
    - <<: *run-on-linux
    @@ -152,7 +173,7 @@ jobs:
    script:
    # First step: download the *.json and *.tar.gz from the S3 bucket
    - |
    ruby <<EOF || true
    rvm default do ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    s3.bucket("$BUCKET").objects(prefix:"$TRAVIS_BUILD_NUMBER/").each { |o| o.download_file(o.key.to_s.gsub(/^\d*\/(.*)/, '\1')) }
    @@ -170,7 +191,7 @@ jobs:
    - |
    if ls *.{json,tar.gz}; then # (1)
    sed -i 's:/usr/local:/home/linuxbrew/.linuxbrew:g' *.json; # (2)
    brew test-bot --ci-upload --git-name=maelvalais [email protected] --bintray-org=maelvalais --verbose;
    brew test-bot --ci-upload --root-url=$TAP_BOTTLE_ROOT_URL --git-name=$GITHUB_USER [email protected] --bintray-org=$TAP_BOTTLE_ORG --verbose;
    else
    echo "==> No bottle found in the bucket, skipping --ci-upload";
    fi
    @@ -181,6 +202,16 @@ after_script:
    # available, we don't want the build to fail.
    # Here, the 'testing' jobs have finished and must upload their *.{json,tar.gz}
    - |
    ruby <<EOF || true
    rvm default do ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    Dir["*.{json,tar.gz}"].each { |f| s3.bucket("$BUCKET").put_object(key:"$TRAVIS_BUILD_NUMBER/#{f}").upload_file(f) }
    before_cache:
    # Scrub cache so that travis only caches stuff for installed formulae.
    - brew cleanup -s
    - brew cask cleanup
    # Remove temporary stuff (idk why they put that in ~/.cache... that should be in /tmp)
    - rm -f ~/Library/Caches/Homebrew/linkage.db ~/.cache/Homebrew/linkage.db
    # List the formulae so that I understand why the cache is huge sometimes
    - brew list
    - du -h ~/Library/Caches/Homebrew || du -h ~/.cache/Homebrew/linkage.db
  4. maelvls renamed this gist Jun 21, 2018. 1 changed file with 0 additions and 0 deletions.
  5. maelvls revised this gist Jun 21, 2018. No changes.
  6. maelvls revised this gist Jun 21, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    How to automate the build of bottles on your tap
    ===============================================
    How to automate the build of bottles on your Homebrew tap
    ========================================================

    This tutorial is a follow-up to the discussion we had on https://github.com/davidchall/homebrew-hep/pull/114.
    It relies on a fork of the `test-bot` provided by davidchall; you can get it with `brew tap maelvalais/test-bot`.
  7. maelvls revised this gist Jun 18, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ I propose to use the same repo. Don't worry, the `pr-1234` tags won't show in pe

    [![homebrew-touist workflow, a tap](https://www.lucidchart.com/publicSegments/view/0ae31518-840c-4207-9276-4e3717a4bc6c/image.png)](https://www.lucidchart.com/documents/view/d8a214f4-d978-4be5-9f28-85086dbd26c8)

    <p align="right">_Drawings made using Lucidchart (not free)_</p>
    <p align="right">Drawings made using Lucidchart (not free).</p>

    When a PR is opened and the formula needs a bottle, Travis CI will build the bottle (`brew test-bot`) and then
    run `brew test-bot --ci-upload` which will:
  8. maelvls revised this gist Jun 18, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ I propose to use the same repo. Don't worry, the `pr-1234` tags won't show in pe

    [![homebrew-touist workflow, a tap](https://www.lucidchart.com/publicSegments/view/0ae31518-840c-4207-9276-4e3717a4bc6c/image.png)](https://www.lucidchart.com/documents/view/d8a214f4-d978-4be5-9f28-85086dbd26c8)

    <small>Drawings made using Lucidchart (not free)</small>
    <p align="right">_Drawings made using Lucidchart (not free)_</p>

    When a PR is opened and the formula needs a bottle, Travis CI will build the bottle (`brew test-bot`) and then
    run `brew test-bot --ci-upload` which will:
  9. maelvls revised this gist Jun 18, 2018. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -42,12 +42,14 @@ that uses it and its [`.travis.yml` (homebrew-touist repo)](https://github.com/t

    The Homebrew/homebrew-core has the following workflow:

    ![homebrew-core workflow](https://www.lucidchart.com/publicSegments/view/525e9c0f-3a81-49e6-9879-97da4d543c0f/image.png)
    [![homebrew-core workflow](https://www.lucidchart.com/publicSegments/view/525e9c0f-3a81-49e6-9879-97da4d543c0f/image.png)](https://www.lucidchart.com/documents/view/d6a4384f-2faf-4fef-b9c2-ceb1dd417037)

    Instead of pushing `pr-1234` (containing the updated DSL commit) to a fork (in core's case, the fork is BrewTestBot/homebrew-core),
    I propose to use the same repo. Don't worry, the `pr-1234` tags won't show in people's clones. Here is the workflow:

    ![homebrew-touist workflow, a tap](https://www.lucidchart.com/publicSegments/view/0ae31518-840c-4207-9276-4e3717a4bc6c/image.png)
    [![homebrew-touist workflow, a tap](https://www.lucidchart.com/publicSegments/view/0ae31518-840c-4207-9276-4e3717a4bc6c/image.png)](https://www.lucidchart.com/documents/view/d8a214f4-d978-4be5-9f28-85086dbd26c8)

    <small>Drawings made using Lucidchart (not free)</small>

    When a PR is opened and the formula needs a bottle, Travis CI will build the bottle (`brew test-bot`) and then
    run `brew test-bot --ci-upload` which will:
  10. maelvls revised this gist Mar 20, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -15,9 +15,9 @@ if: tag IS blank AND NOT type IN (push)
    env:
    global:
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist
    - AWS_REGION=eu-west-1 # needed because the aws-sdk-s3 gem errors without it
    - BUCKET=homebrew-amc
    - BUCKET=homebrew-touist-travis
    # Beyond these global variables, you also need to set some secret variables
    # in your travis-ci settings:
    # - AWS_ACCESS_KEY_ID
  11. maelvls revised this gist Mar 20, 2018. 2 changed files with 93 additions and 83 deletions.
    62 changes: 42 additions & 20 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -3,16 +3,18 @@ How to automate the build of bottles on your tap

    This tutorial is a follow-up to the discussion we had on https://github.com/davidchall/homebrew-hep/pull/114.
    It relies on a fork of the `test-bot` provided by davidchall; you can get it with `brew tap maelvalais/test-bot`.
    Two things to do first: create a Github and Bintray projects:
    First:

    1. the Github project must be of the form `https://github.com/<user>/homebrew-<tap>` with the following tree:
    1. the Github project must be of the form `https://github.com/<user>/homebrew-<tap>` with the following tree
    (I give the example of one of my formulas, _touist_):

    .
    ├── your-formula.rb
    ├── touist.rb
    └── .travis.yml


    2. the Bintray project must be of the form `https://bintray.com/<user>/bottles-<tap>`
    3. the S3 bucket can have any name

    ## Use Travis CI to test and bottle your formulae

    @@ -36,6 +38,34 @@ where bottles are built for linux, sierra and el\_capitan.
    You can also see an example of tap, [touist/homebrew-touist](https://github.com/touist/homebrew-touist),
    that uses it and its [`.travis.yml` (homebrew-touist repo)](https://github.com/touist/homebrew-touist/blob/master/.travis.yml).

    ## Workflow for updating formulas

    The Homebrew/homebrew-core has the following workflow:

    ![homebrew-core workflow](https://www.lucidchart.com/publicSegments/view/525e9c0f-3a81-49e6-9879-97da4d543c0f/image.png)

    Instead of pushing `pr-1234` (containing the updated DSL commit) to a fork (in core's case, the fork is BrewTestBot/homebrew-core),
    I propose to use the same repo. Don't worry, the `pr-1234` tags won't show in people's clones. Here is the workflow:

    ![homebrew-touist workflow, a tap](https://www.lucidchart.com/publicSegments/view/0ae31518-840c-4207-9276-4e3717a4bc6c/image.png)

    When a PR is opened and the formula needs a bottle, Travis CI will build the bottle (`brew test-bot`) and then
    run `brew test-bot --ci-upload` which will:

    - upload the bottle as "unpublished".
    - commit and push the DSL to BrewTestBot/Homebrew-core with the pr-1234 tag.

    Then a Homebrew maintainer (me) will do

    brew pull --bottle --bintray-org=touist --test-bot-user=touist https://github.com/touist/homebrew-touist/pull/5

    which will do two things:

    - switch the bottle in bintray from "unpublished" to "published"
    - fetch the `pr-1234` tag, rebase it on top of Homebrew/homebrew-core

    then the mainainer push the merged PR to Homebrew/homebrew-core.

    ### Drawbacks w.r.t. bottles when maintaining a tap relying on core formulae

    On the homebrew-core repo, whenever a formula on which some other formulae
    @@ -45,17 +75,14 @@ number). This seems to be automatically taken care of on the homebrew-core
    tap.

    On your own tap, this revision bump whenever a dependency bottle gets rebuilt
    can lead to broken bottles. This problem has not been solved yet, but:
    can lead to broken bottles. This problem has not been solved yet (see
    https://github.com/Homebrew/brew/issues/3346 and
    https://github.com/Homebrew/brew/issues/2572). But we can:

    1) may be we could rely on a cron job that tests the bottles every week to check
    if the bottle still works;
    1) run a cron job that tests the bottles every night to check
      if the bottle still works, using `brew test *.rb` and `brew linkage --test *.rb`;
    2) or we could subscribe to the changes on the formulae we depend on and
    run a script on a Travis CI build

    Both solutions require a lot a thoughts, that is why I did not do anything
    about it. I only rely on `gmp` at runtime so maybe it is fine... Note that
    this problem has been also descussed [here](https://github.com/Homebrew/brew/issues/2572#issuecomment-298725510)
    and [here](https://github.com/Homebrew/brew/issues/3346).
    run a script on a Travis CI build (I could not find a way to do that yet).

    ### What you need

    @@ -77,7 +104,7 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    to make sure everything runs correctly locally (except for `brew doctor` which generally always
    fails on my mac) using:

    brew test-bot Formula/touist.rb
    brew test-bot touist.rb

    2) On Travis CI, mac builds are costly (in time). For trying to make the
      `.travis.yml` work, I really recommend to use `os: linux` instead of a mac
    @@ -94,13 +121,8 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    5) the `--keep-old` option when uploading seems to be a good idea (= keeps bottles you
    previously built but that you don't produce anymore) but it keeps failing on stupid
    errors all the time; I don't use it anymore.
    6) when you try to make a formula work (on master) and you want to rebuid the bottle of the formula,
    just edit somewhere in the formula and push. This is going to increment "rebuild 1" in the
    bottle DSL and reupload the bottles.
    7) same thing on a PR or a branch: to make the formula rebuild, the push has to only contain edits on
    a Formula/formula.rb file
    8) adding "revision 1" to a formula does not seem to be appropriate for forcing a bottle to be rebuil
    9) the tests/building of the bottle only happens when you push a series of commit on ONE single formula;
    7) adding "revision 1" to a formula does not seem to be appropriate for forcing a bottle to be rebuild
    8) the tests/building of the bottle only happens when you push a series of commit on ONE single formula;
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):
    114 changes: 51 additions & 63 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -5,35 +5,38 @@
    # - cron/api (api is when you select 'trigger build' in travis): in this mode,
    # all the formulas are tested to make sure that there is no shared lib
    # breakage
    # - push to master: in this mode, the updated formulas are built and bottled
    # and the bottles are pushed to bintray, commited and pushed.
    # - pull-request: same as 'push on master' but without bintray and commit.
    # - push to master: does nothing
    # - pull-request: in this mode, the updated formulas are built and bottled
    # and the bottles are pushed to bintray as 'unpublished'. The DSL commit
    # is pushed as a tag, e.g., 'pr-1234', which us unrelated to master.

    language: ruby
    if: tag IS blank # build is disabled on tags (no rationale actually)
    if: tag IS blank AND NOT type IN (push)
    env:
    global:
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/maelvalais
    - AWS_REGION=eu-west-1 # needed because the aws-sdk-s3 gem errors without it
    - BUCKET=homebrew-touist-travis
    - BUCKET=homebrew-amc
    # Beyond these global variables, you also need to set some secret variables
    # in your travis-ci settings:
    # - AWS_ACCESS_KEY_ID
    # - AWS_SECRET_ACCESS_KEY (for the AWS S3 bucket)
    # - GITHUB_TOKEN (for pushing bottle DSL commits)
    # - HOMEBREW_BINTRAY_KEY (for the Bintray repo)

    # Note: I removed $HOME/Library/Caches/Homebrew from the caches
    # because it was enormous (~2GB for the 4 jobs). This is because
    # this folder is already present on osx images and it is big.
    # Note2: I also removed $HOME/.cache/Homebrew (cache of Linuxbrew)
    # because it was also storing some logs, making the cache rebuilt
    # every time.
    # (1) remember tu use 'brew cleanup -s' at the end of the travis script;
    # otherwise, the $HOME/Library/Caches/Homebrew folder will be enormous
    # (~2GB for the 4 jobs). This is because this folder is already present on
    # osx images and it is big.
    # (2) I also removed $HOME/.cache/Homebrew (cache of Linuxbrew) because it
    # was also storing some logs, making the cache rebuilt every time.
    cache:
    directories:
    - "$HOME/.cache/pip"
    - "$HOME/.gem/ruby"
    #- "$HOME/Library/Caches/Homebrew" # (1)
    #- "$HOME/.cache/Homebrew" # (2)

    install:
    # the official test-bot won't let you run inside TravisCI, so we use
    @@ -62,28 +65,28 @@ install:
    # brew cask uninstall... it will fail on the deprecation notice.
    # Unless I am in 'brew test-bot', I don't want to fail on warnings.

    # WARNING: I discovered that 'set -e' is breaking Travis CI. I was using for
    # stopping a multiple-lines command as soon as one of the commands fails.
    # After having 'set -e', internal travis commands may stop working, e.g.:
    # /Users/travis/.travis/job_stages: line 57: shell_session_update: command not found
    # is an internal error that shouldn't stop the script for continuing, but
    # because of 'set -e', the whole internal Travis CI script stops.
    # Conclusion: avoid using 'set -e' and more generally avoid for-loops and
    # multi-line commands that should fail when one of the commands fails.
    script:
    - |
    if [[ $TRAVIS_EVENT_TYPE =~ cron|api ]]; then
    brew install *.rb && brew test *.rb || exit 123
    for f in *.rb; do
    if ! brew linkage --test ${f/\.rb/}; then
    rev=$(grep revision $f | sed 's/^[^0-9]*\([0-9]*\)$/\1/')
    sed -i "s/revision [0-9]*/revision $(expr $rev + 1)" $f
    git commit -a -m "${f/\.rb/}: bump from revision $rev to $(expr $rev + 1) (brew linkage)"
    git log -n2 -p
    git push https://${GITHUB_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git master
    fi
    done
    brew install *.rb && brew test *.rb && brew linkage --test *.rb || exit 123
    fi
    - if [[ $TRAVIS_EVENT_TYPE =~ push|pull_request ]]; then brew test-bot; fi
    - if [[ $TRAVIS_EVENT_TYPE =~ pull_request ]]; then brew test-bot; fi
    - ls *.bottle*.* || echo "==> No bottle created here"

    jobs:
    include:
    - os: osx
    osx_image: xcode8.3
    env: OS=sierra-10.12
    - &run-osx
    os: osx
    osx_image: xcode9.3beta
    env: OS=sierra-10.13
    # We must use 'rvm: system' because the system ruby doesn't rely on
    # Homebrew, which allows us to reinstall Homebrew without having a
    # damaged ruby. We need ruby because we use the 'aws-sdk-s3' gem.
    @@ -105,22 +108,15 @@ jobs:
    - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    - brew cask reinstall xquartz

    - &run-on-osx-old-xcode # only for xcode6.4 and xcode7.3
    - <<: *run-osx
    os: osx
    osx_image: xcode8.3
    env: OS=high_sierra-10.12

    - <<: *run-osx
    os: osx
    osx_image: xcode7.3
    env: OS=el_capitan-10.11
    rvm: system
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    - brew cask outdated xquartz || brew cask uninstall xquartz
    - mkdir ~/usr_local && sudo mv /usr/local/* ~/usr_local
    - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    - brew cask reinstall xquartz

    # - <<: *run-on-osx-old-xcode
    # os: osx
    # osx_image: xcode6.4
    # env: OS=yosemite-10.10
    # rvm: system

    - &run-on-linux
    os: linux
    @@ -146,10 +142,11 @@ jobs:
    /opt/pyenv/shims/*-config /usr/local/clang-*/bin/llvm-config
    before_cache:
    - brew cleanup

    # Uploading to bintray (as unpublished bottle) and pushing a tag to the
    # repo are only enabled on pull_requests.
    - <<: *run-on-linux
    stage: deploy
    if: branch = master AND NOT type IN (pull_request, cron, api)
    if: branch = master AND type IN (pull_request)
    os: linux
    env: OS=any_linux
    script:
    @@ -158,35 +155,26 @@ jobs:
    ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    s3.bucket("$BUCKET").objects(prefix:"$TRAVIS_BUILD_NUMBER/").each { |o| o.download_file(o.key.to_s.gsub(/^\d*\/(.*)/, '\1')) }
    # This hook will add [ci skip] the the commit messages for bottle DSL
    # so that travis-ci does not re-build a duplicate bottle on pushing.
    - echo 'echo "\n\n[ci skip]" >> "$1"' > .git/hooks/commit-msg; chmod +x .git/hooks/commit-msg
    # We fail 'silently' if no bottle is found.

    # (1) We fail 'silently' if no bottle is found.
    # (2) A trick so that 'prefix' and 'cellar' are removed if they are
    # equal to DEFAULT_PREFIX and DEFAULT_CELLAR even though we run
    # 'brew bottle' on a linux setup.
    # (3) In 'brew --ci--upload', --git-name is the name of the fork
    # where the pr-1234 (or testing-98 when not in a PR) tag is pushed.
    # In my case, I use the 'main' instead of a fork.
    - |
    if ls *.{json,tar.gz}; then
    # This is a trick so that 'prefix' and 'cellar' are removed if they
    # are equal to DEFAULT_PREFIX and DEFAULT_CELLAR even though we run
    # 'brew bottle' on a linux setup.
    sed -i 's:/usr/local:/home/linuxbrew/.linuxbrew:g' *.json;
    # Travis does not provide a high_sierra image yet, but I can use the
    # sierra one and pretend it is a high_sierra bottle. I test them regularly in
    # case it doesn't work.
    ls *.sierra.bottle.json | while read json; do
    sed 's/sierra/high_sierra/g' $json > ${json/sierra/high_sierra};
    bottle=`ls ${json/bottle*/}*.tar.gz`;
    cp ${bottle} ${bottle/sierra/high_sierra};
    done;
    # Note that --git-name also determines where the push will be done!
    # In my case, I disabled the push using '--no-push' so it wont push
    # to maelvalais/homebrew-touist.
    git fetch && git checkout master;
    brew test-bot --ci-upload --no-push --overwrite --git-name=touist --bintray-org=touist --verbose;
    # git fetch && git rebase && git push https://${GITHUB_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git master;
    if ls *.{json,tar.gz}; then # (1)
    sed -i 's:/usr/local:/home/linuxbrew/.linuxbrew:g' *.json; # (2)
    brew test-bot --ci-upload --git-name=maelvalais [email protected] --bintray-org=maelvalais --verbose;
    else
    echo "==> No bottle found in the bucket, skipping --ci-upload";
    fi
    after_script:
    # nothing
    after_script: true # nothing

    after_script:
    # In case the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY are not
  12. maelvls revised this gist Mar 20, 2018. 1 changed file with 1 addition and 2 deletions.
    3 changes: 1 addition & 2 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -8,8 +8,7 @@ Two things to do first: create a Github and Bintray projects:
    1. the Github project must be of the form `https://github.com/<user>/homebrew-<tap>` with the following tree:

    .
    ├── formula1.rb
    ├── formula2.rb
    ├── your-formula.rb
    └── .travis.yml


  13. maelvls revised this gist Mar 20, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -6,10 +6,12 @@ It relies on a fork of the `test-bot` provided by davidchall; you can get it wit
    Two things to do first: create a Github and Bintray projects:

    1. the Github project must be of the form `https://github.com/<user>/homebrew-<tap>` with the following tree:

    .
    ├── formula1.rb
    ├── formula2.rb
    └── .travis.yml


    2. the Bintray project must be of the form `https://bintray.com/<user>/bottles-<tap>`

  14. maelvls revised this gist Mar 20, 2018. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,16 @@ How to automate the build of bottles on your tap
    ===============================================

    This tutorial is a follow-up to the discussion we had on https://github.com/davidchall/homebrew-hep/pull/114.
    It relies on the `test-bot` provided by davidchall; you can get it with `brew tap davidchall/test-bot`.
    It relies on a fork of the `test-bot` provided by davidchall; you can get it with `brew tap maelvalais/test-bot`.
    Two things to do first: create a Github and Bintray projects:

    1. the Github project must be of the form `https://github.com/<user>/homebrew-<tap>` with the following tree:
    .
    ├── formula1.rb
    ├── formula2.rb
    └── .travis.yml

    2. the Bintray project must be of the form `https://bintray.com/<user>/bottles-<tap>`

    ## Use Travis CI to test and bottle your formulae

  15. maelvls revised this gist Mar 12, 2018. 1 changed file with 28 additions and 15 deletions.
    43 changes: 28 additions & 15 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -15,6 +15,8 @@ env:
    global:
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist
    - AWS_REGION=eu-west-1 # needed because the aws-sdk-s3 gem errors without it
    - BUCKET=homebrew-touist-travis
    # Beyond these global variables, you also need to set some secret variables
    # in your travis-ci settings:
    # - AWS_ACCESS_KEY_ID
    @@ -34,13 +36,12 @@ cache:
    - "$HOME/.gem/ruby"

    install:
    # Fix travis log truncated (https://github.com/travis-ci/travis-ci/issues/8920)
    - python -c "import fcntl; fcntl.fcntl(1, fcntl.F_SETFL, 0)"
    # the official test-bot won't let you run inside TravisCI, so we use
    # davidchall's one. David's test-bot cannot push the commit using
    # Oauth github + https (only ssh) so I use my own.
    - brew tap maelvalais/test-bot

    # Install the ruby AWS gem so that I can upload bottles to S3
    - sudo gem install aws-sdk-s3
    # IMPORTANT STEP: link the tap inside brew to our current travis-cloned tap
    # Step: 1) create the intermediate folders <user>/<repo> so that
    # we can 2) remove <repo> and 3) replace it with a sym link
    @@ -53,10 +54,6 @@ install:
    - rm -rf $(brew --repo $TRAVIS_REPO_SLUG) # 2)
    - ln -s $PWD $(brew --repo $TRAVIS_REPO_SLUG) # 3)
    - git fetch --unshallow || true # 4)

    - brew install awscli
    - aws s3 sync s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} ~/shared || true

    # 'brew doctor' must be run under HOMEBREW_DEVELOPER=1. Otherwise,
    # it returns 1 with the message "this osx version is outdated" on old osx ver.
    - HOMEBREW_DEVELOPER=1 brew doctor
    @@ -68,7 +65,7 @@ install:
    script:
    - |
    if [[ $TRAVIS_EVENT_TYPE =~ cron|api ]]; then
    brew install *.rb || exit 123
    brew install *.rb && brew test *.rb || exit 123
    for f in *.rb; do
    if ! brew linkage --test ${f/\.rb/}; then
    rev=$(grep revision $f | sed 's/^[^0-9]*\([0-9]*\)$/\1/')
    @@ -80,13 +77,16 @@ script:
    done
    fi
    - if [[ $TRAVIS_EVENT_TYPE =~ push|pull_request ]]; then brew test-bot; fi
    - cp *.bottle*.* ~/shared 2>/dev/null || echo "==> No bottle created here"
    - ls *.bottle*.* || echo "==> No bottle created here"

    jobs:
    include:
    - os: osx
    osx_image: xcode8.3
    env: OS=sierra-10.12
    # We must use 'rvm: system' because the system ruby doesn't rely on
    # Homebrew, which allows us to reinstall Homebrew without having a
    # damaged ruby. We need ruby because we use the 'aws-sdk-s3' gem.
    rvm: system
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    # First we uninstall any outdated versions of xquartz; otherwise,
    @@ -133,7 +133,7 @@ jobs:
    # Because we cannot do 'umask 002' just before travis clones the repo,
    # I set umask afterwards (1) and I change the permission of
    # already cloned files from 664 to 644 (2).
    - umask 022 # (1)
    - umask 022 # (1)
    - chmod 0644 *.rb # (2)

    # Instal linuxbrew
    @@ -153,12 +153,17 @@ jobs:
    os: linux
    env: OS=any_linux
    script:
    # First step: download the *.json and *.tar.gz from the S3 bucket
    - |
    ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    s3.bucket("$BUCKET").objects(prefix:"$TRAVIS_BUILD_NUMBER/").each { |o| o.download_file(o.key.to_s.gsub(/^\d*\/(.*)/, '\1')) }
    # This hook will add [ci skip] the the commit messages for bottle DSL
    # so that travis-ci does not re-build a duplicate bottle on pushing.
    - echo 'echo "\n\n[ci skip]" >> "$1"' > .git/hooks/commit-msg; chmod +x .git/hooks/commit-msg
    # We fail 'silently' if no bottle is found.
    - |
    if cp -v ~/shared/* . ; then
    if ls *.{json,tar.gz}; then
    # This is a trick so that 'prefix' and 'cellar' are removed if they
    # are equal to DEFAULT_PREFIX and DEFAULT_CELLAR even though we run
    # 'brew bottle' on a linux setup.
    @@ -171,15 +176,23 @@ jobs:
    bottle=`ls ${json/bottle*/}*.tar.gz`;
    cp ${bottle} ${bottle/sierra/high_sierra};
    done;
    brew test-bot --ci-upload --git-name=maelvalais [email protected] --bintray-org=touist --verbose;
    # Note that --git-name also determines where the push will be done!
    # In my case, I disabled the push using '--no-push' so it wont push
    # to maelvalais/homebrew-touist.
    git fetch && git checkout master;
    brew test-bot --ci-upload --no-push --overwrite --git-name=touist --bintray-org=touist --verbose;
    # git fetch && git rebase && git push https://${GITHUB_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git master;
    else
    echo "==> No bottle found in the bucket, skipping --ci-upload";
    fi
    after_script:
    # Clean the bucket
    #- aws s3 rm s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} --recursive
    # nothing

    after_script:
    # In case the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY are not
    # available, we don't want the build to fail.
    - aws s3 sync ~/shared s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} || true
    # Here, the 'testing' jobs have finished and must upload their *.{json,tar.gz}
    - |
    ruby <<EOF || true
    require 'aws-sdk-s3'; s3 = Aws::S3::Resource.new
    Dir["*.{json,tar.gz}"].each { |f| s3.bucket("$BUCKET").put_object(key:"$TRAVIS_BUILD_NUMBER/#{f}").upload_file(f) }
  16. maelvls revised this gist Jan 27, 2018. 2 changed files with 67 additions and 49 deletions.
    19 changes: 7 additions & 12 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -110,15 +110,11 @@ WARNING: in my solution, in the deploy state, I do on a linux build:
    - `brew test-bot --ci-upload`

    which packs all bottles at the same time and I get weird `prefix: /usr/local` and
    `cellar: :any_skip_relocation`. Intead, I should do:
    `cellar: :any_skip_relocation`. I fixed this with a little trick: change every
    `/usr/local` with `/home/linuxbrew/.linuxbrew` so that `brew test-bot --ci-upload`
    removes the `cellar:` field when it is the default cellar.

    - first consider the mac bottles and do `brew test-bot --ci-upload`
    - then consider the linux bottle and do `brew test-bot --ci-upload --keep-old` (like in the Linuxbrew thing)

    If on macos has `:any_skip_relocation`, then the linux one will add
    `cellar :any_skip_relocation if OS.mac?` if it is different on linux.

    Example:
    Example of such problem:
    MacOS `lingeling-151109.el_capitan.bottle.json` and `lingeling-151109.sierra.bottle.json` have:
    ```json
    {
    @@ -133,7 +129,8 @@ but the linux bottle `lingeling-151109.x86_64_linux.bottle.json` has:
    "cellar": "/home/linuxbrew/.linuxbrew/Cellar",
    }
    ```
    and it wrongly became the following bottle DSL:
    and it wrongly became the following bottle DSL, probably because the linux `bottle.json`
    is the last to be read, so it takes precedence over the previous ones:

    ```ruby
    bottle do
    @@ -160,8 +157,6 @@ It should have been

    As mentionned by Steven Peters [here](https://github.com/Homebrew/brew/issues/3346#issuecomment-339059005), when a dependency is updated, we must also update the formula that depends on it. In 'core', this is done by incrementing the _revision_ number for all formulas that rely on the updated formula.

    In this tap, a cron is daily testing every formula (`brew install <formula> && brew test <formula>`). If travis-ci reports a failure, I read the travis log and edit the faulty formula by incrementing its revision number. I then commit with the message (for example with qute that depends on boost, and boost has been updated; actual commit [here](https://github.com/touist/homebrew-touist/commit/29248c08fb78dd244f597a1190a5748e6bf49601)):

    qute: revision for boost
    In this tap, a cron is daily testing every formula (`brew install <formula> && brew linkage --test <formula>`). If a linkage breakage is detected, it automatically increments the revision number. It has been working fine for now, the only problem is to have travis-ci working every single day (I am getting better and better at that).


    97 changes: 60 additions & 37 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,16 @@
    # Inspired from the .travis.yml in
    # https://github.com/davidchall/homebrew-hep/

    # This script can be executed in three different modes:
    # - cron/api (api is when you select 'trigger build' in travis): in this mode,
    # all the formulas are tested to make sure that there is no shared lib
    # breakage
    # - push to master: in this mode, the updated formulas are built and bottled
    # and the bottles are pushed to bintray, commited and pushed.
    # - pull-request: same as 'push on master' but without bintray and commit.

    language: ruby
    if: tag IS blank
    if: tag IS blank # build is disabled on tags (no rationale actually)
    env:
    global:
    - HOMEBREW_BINTRAY_USER=maelvalais
    @@ -26,6 +34,8 @@ cache:
    - "$HOME/.gem/ruby"

    install:
    # Fix travis log truncated (https://github.com/travis-ci/travis-ci/issues/8920)
    - python -c "import fcntl; fcntl.fcntl(1, fcntl.F_SETFL, 0)"
    # the official test-bot won't let you run inside TravisCI, so we use
    # davidchall's one. David's test-bot cannot push the commit using
    # Oauth github + https (only ssh) so I use my own.
    @@ -57,14 +67,19 @@ install:

    script:
    - |
    set -e;
    if [ $TRAVIS_EVENT_TYPE = cron -o $TRAVIS_EVENT_TYPE = api ]; then
    for f in Formula/*.rb; do
    brew install $f && brew test $f;
    done;
    else
    brew test-bot;
    if [[ $TRAVIS_EVENT_TYPE =~ cron|api ]]; then
    brew install *.rb || exit 123
    for f in *.rb; do
    if ! brew linkage --test ${f/\.rb/}; then
    rev=$(grep revision $f | sed 's/^[^0-9]*\([0-9]*\)$/\1/')
    sed -i "s/revision [0-9]*/revision $(expr $rev + 1)" $f
    git commit -a -m "${f/\.rb/}: bump from revision $rev to $(expr $rev + 1) (brew linkage)"
    git log -n2 -p
    git push https://${GITHUB_TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git master
    fi
    done
    fi
    - if [[ $TRAVIS_EVENT_TYPE =~ push|pull_request ]]; then brew test-bot; fi
    - cp *.bottle*.* ~/shared 2>/dev/null || echo "==> No bottle created here"

    jobs:
    @@ -73,40 +88,33 @@ jobs:
    osx_image: xcode8.3
    env: OS=sierra-10.12
    rvm: system
    before_install:
    # Sometimes, brew update fails with the error `Homebrew must be run
    # under Ruby 2.3! You are running 2.0.0' (althouth ruby-portable
    # is there). Workaround (A): run double brew update (see
    # https://github.com/Homebrew/brew/issues/3299). This seems to happen
    # as soon as the macos image on Travis is 'too old'.
    # The issue of broken symlink (B) also happens at the same time.
    - brew update | grep "==>" || brew update # (A)
    # (brew doctor) fix some random broken symlinks
    - brew prune # (B)
    # (brew doctor) fix the missing deps
    - brew missing | cut -d':' -f1 | xargs brew uninstall
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    # First we uninstall any outdated versions of xquartz; otherwise,
    # Homebrew will complain of of older version (2.9.7) being outdated
    # even though we install a new version.
    - brew cask outdated xquartz || brew cask uninstall xquartz
    # Three reasons not to use the /usr/local and Homebrew installations
    # that come in the Travis CI images:
    # 1) because Travis CI has installed many non-homebrew things into
    # /usr/local that randomly cause 'brew doctor' to fail;
    # 2) after time, the osx image contains an outdated Homebrew that
    # has weird 'unlinked kegs' and such;
    # 3) also because it takes a long time to 'brew update' from an old
    # Homebrew anyway, so why not start fresh.
    - mkdir ~/usr_local && sudo mv /usr/local/* ~/usr_local
    - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    - brew cask reinstall xquartz

    - &run-on-osx-old-xcode # only for xcode6.4 and xcode7.3
    os: osx
    osx_image: xcode7.3
    env: OS=el_capitan-10.11
    rvm: system
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    # Sometimes, brew update fails with the error `Homebrew must be run
    # under Ruby 2.3! You are running 2.0.0' (althouth ruby-portable
    # is there). Workaround (A): run double brew update (see
    # https://github.com/Homebrew/brew/issues/3299). This seems to happen
    # as soon as the macos image on Travis is 'too old'.
    # The issue of broken symlink (B) also happens at the same time.
    - brew update | grep "==>" || brew update # (A)
    # (brew doctor) fix some random broken symlinks
    - brew prune # (B)
    # (brew doctor) fix the many missing deps on gnupg and others
    - brew missing | cut -d':' -f1 | xargs brew uninstall
    # (brew doctor) fix "your XQuartz (2.7.9) is outdated" if it is installed
    - brew cask uninstall xquartz || true
    # (brew test-bot on travis) fix libyaml 0.1.6_1 already installed
    - brew uninstall libyaml || true; brew uninstall md5deep || true
    - brew cask outdated xquartz || brew cask uninstall xquartz
    - mkdir ~/usr_local && sudo mv /usr/local/* ~/usr_local
    - /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    - brew cask reinstall xquartz

    # - <<: *run-on-osx-old-xcode
    # os: osx
    @@ -126,7 +134,7 @@ jobs:
    # I set umask afterwards (1) and I change the permission of
    # already cloned files from 664 to 644 (2).
    - umask 022 # (1)
    - chmod 0644 Formula/*.rb # (2)
    - chmod 0644 *.rb # (2)

    # Instal linuxbrew
    - export PATH="/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:$PATH";
    @@ -146,8 +154,23 @@ jobs:
    env: OS=any_linux
    script:
    # This hook will add [ci skip] the the commit messages for bottle DSL
    # so that travis-ci does not re-build a duplicate bottle on pushing.
    - echo 'echo "\n\n[ci skip]" >> "$1"' > .git/hooks/commit-msg; chmod +x .git/hooks/commit-msg
    - if cp -v ~/shared/* . ; then
    # We fail 'silently' if no bottle is found.
    - |
    if cp -v ~/shared/* . ; then
    # This is a trick so that 'prefix' and 'cellar' are removed if they
    # are equal to DEFAULT_PREFIX and DEFAULT_CELLAR even though we run
    # 'brew bottle' on a linux setup.
    sed -i 's:/usr/local:/home/linuxbrew/.linuxbrew:g' *.json;
    # Travis does not provide a high_sierra image yet, but I can use the
    # sierra one and pretend it is a high_sierra bottle. I test them regularly in
    # case it doesn't work.
    ls *.sierra.bottle.json | while read json; do
    sed 's/sierra/high_sierra/g' $json > ${json/sierra/high_sierra};
    bottle=`ls ${json/bottle*/}*.tar.gz`;
    cp ${bottle} ${bottle/sierra/high_sierra};
    done;
    brew test-bot --ci-upload --git-name=maelvalais [email protected] --bintray-org=touist --verbose;
    else
    echo "==> No bottle found in the bucket, skipping --ci-upload";
  17. maelvls revised this gist Jan 8, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -21,10 +21,10 @@ The idea is to use _staged build_ in Travis CI, where the two stages will be

    ![Screenshot of the staged build status on Travis CI](https://user-images.githubusercontent.com/2195781/32568431-83a25464-c4be-11e7-8c9f-31202d07fd79.png)

    You can see a detailed example of such a configuration in the [`.travis.yml`](#file-travis-yml) of this gist,
    You can see a detailed example of such a configuration in the [`.travis.yml` (this gist)](#file-travis-yml),
    where bottles are built for linux, sierra and el\_capitan.
    You can also see an example of tap, [touist/homebrew-touist](https://github.com/touist/homebrew-touist),
    that uses it and its [`.travis.yml`](https://github.com/touist/homebrew-touist/blob/master/.travis.yml).
    that uses it and its [`.travis.yml` (homebrew-touist repo)](https://github.com/touist/homebrew-touist/blob/master/.travis.yml).

    ### Drawbacks w.r.t. bottles when maintaining a tap relying on core formulae

  18. maelvls revised this gist Jan 8, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -21,7 +21,7 @@ The idea is to use _staged build_ in Travis CI, where the two stages will be

    ![Screenshot of the staged build status on Travis CI](https://user-images.githubusercontent.com/2195781/32568431-83a25464-c4be-11e7-8c9f-31202d07fd79.png)

    You can see a detailed example of such a configuration in the [`.travis.yml`][#file-travis-yml] of this gist,
    You can see a detailed example of such a configuration in the [`.travis.yml`](#file-travis-yml) of this gist,
    where bottles are built for linux, sierra and el\_capitan.
    You can also see an example of tap, [touist/homebrew-touist](https://github.com/touist/homebrew-touist),
    that uses it and its [`.travis.yml`](https://github.com/touist/homebrew-touist/blob/master/.travis.yml).
  19. maelvls revised this gist Jan 8, 2018. 1 changed file with 10 additions and 0 deletions.
    10 changes: 10 additions & 0 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -155,3 +155,13 @@ It should have been
    ```
    ### Notes on my discoveries of Linuxbrew
    - When a formula is linuxbrew-only, it has the comment `# tag "linuxbrew"` in it.

    ### What to do when some dependencies are updated?

    As mentionned by Steven Peters [here](https://github.com/Homebrew/brew/issues/3346#issuecomment-339059005), when a dependency is updated, we must also update the formula that depends on it. In 'core', this is done by incrementing the _revision_ number for all formulas that rely on the updated formula.

    In this tap, a cron is daily testing every formula (`brew install <formula> && brew test <formula>`). If travis-ci reports a failure, I read the travis log and edit the faulty formula by incrementing its revision number. I then commit with the message (for example with qute that depends on boost, and boost has been updated; actual commit [here](https://github.com/touist/homebrew-touist/commit/29248c08fb78dd244f597a1190a5748e6bf49601)):

    qute: revision for boost


  20. maelvls revised this gist Jan 8, 2018. 1 changed file with 49 additions and 26 deletions.
    75 changes: 49 additions & 26 deletions travis.yml
    Original file line number Diff line number Diff line change
    @@ -5,30 +5,31 @@ language: ruby
    if: tag IS blank
    env:
    global:
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist
    # Beyond these global variables, you also need to set some secret variables
    # in your travis-ci settings:
    # in your travis-ci settings:
    # - AWS_ACCESS_KEY_ID
    # - AWS_SECRET_ACCESS_KEY (for the AWS S3 bucket)
    # - GITHUB_TOKEN (for pushing bottle DSL commits)
    # - HOMEBREW_BINTRAY_KEY (for the Bintray repo)
    - HOMEBREW_BINTRAY_USER=maelvalais
    - TAP_BOTTLE_DOMAIN=https://dl.bintray.com/touist

    # Note: I removed $HOME/Library/Caches/Homebrew from the caches
    # because it was enormous (~2GB for the 4 jobs). This is because
    # this folder is already present on osx images and it is big.
    # The linuxbrew one isn't as big because it is created from scratch.
    # Note2: I also removed $HOME/.cache/Homebrew (cache of Linuxbrew)
    # because it was also storing some logs, making the cache rebuilt
    # every time.
    cache:
    directories:
    - "$HOME/.cache/Homebrew" # cache of Linuxbrew
    - "$HOME/.cache/pip"
    - "$HOME/.gem/ruby"

    install:
    # the official test-bot won't let you run inside TravisCI, so we use
    # davidchall's one. David's test-bot cannot push the commit using
    # Oauth github + https (only ssh) so I use my own.
    - brew tap davidchall/test-bot
    - brew tap maelvalais/test-bot

    # IMPORTANT STEP: link the tap inside brew to our current travis-cloned tap
    # Step: 1) create the intermediate folders <user>/<repo> so that
    @@ -46,40 +47,62 @@ install:
    - brew install awscli
    - aws s3 sync s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} ~/shared || true

    script:
    # 'brew doctor' must be run under HOMEBREW_DEVELOPER=1. Otherwise,
    # it returns 1 with the message "this osx version is outdated" on old osx ver.
    - HOMEBREW_DEVELOPER=1 brew doctor
    # Note on HOMEBREW_DEVELOPER: I don't want to put in env.global because
    # it should be '1' only during test-bot. If it is '1' during
    # brew cask uninstall... it will fail on the deprecation notice.
    # Unless I am in 'brew test-bot', I don't want to fail on warnings.
    # Also, 'brew doctor' must be run under HOMEBREW_DEVELOPER=1. Otherwise,
    # it returns 1 with the message "this osx version is outdated" on old osx ver.
    - export HOMEBREW_DEVELOPER=1
    - brew doctor # needs HOMEBREW_DEVELOPER because of outdated osx returning 1
    - brew test-bot

    script:
    - |
    set -e;
    if [ $TRAVIS_EVENT_TYPE = cron -o $TRAVIS_EVENT_TYPE = api ]; then
    for f in Formula/*.rb; do
    brew install $f && brew test $f;
    done;
    else
    brew test-bot;
    fi
    - cp *.bottle*.* ~/shared 2>/dev/null || echo "==> No bottle created here"

    # https://docs.travis-ci.com/user/reference/osx/#OS-X-Version
    jobs:
    include:
    - os: osx
    osx_image: xcode8.3
    env: OS=sierra-10.12
    rvm: system
    before_install:
    # Sometimes, brew update fails with the error `Homebrew must be run
    # under Ruby 2.3! You are running 2.0.0' (althouth ruby-portable
    # is there). Workaround (A): run double brew update (see
    # https://github.com/Homebrew/brew/issues/3299). This seems to happen
    # as soon as the macos image on Travis is 'too old'.
    # The issue of broken symlink (B) also happens at the same time.
    - brew update | grep "==>" || brew update # (A)
    # (brew doctor) fix some random broken symlinks
    - brew prune # (B)
    # (brew doctor) fix the missing deps
    - brew missing | cut -d':' -f1 | xargs brew uninstall

    - &run-on-osx-old-xcode # only for xcode6.4 and xcode7.3
    os: osx
    osx_image: xcode7.3
    env: OS=el_capitan-10.11
    rvm: system
    before_install: # IMPORTANT: HOMEBREW_DEVELOPER must not be set here.
    # Sometimes, brew update fails with the error 'Homebrew must be run
    # under Ruby 2.3! You're running 2.0.0 (althouth ruby-portable is there).
    # Workaround: double brew update: https://github.com/Homebrew/brew/issues/3299
    - brew update | grep "==>" || brew update
    # Sometimes, brew update fails with the error `Homebrew must be run
    # under Ruby 2.3! You are running 2.0.0' (althouth ruby-portable
    # is there). Workaround (A): run double brew update (see
    # https://github.com/Homebrew/brew/issues/3299). This seems to happen
    # as soon as the macos image on Travis is 'too old'.
    # The issue of broken symlink (B) also happens at the same time.
    - brew update | grep "==>" || brew update # (A)
    # (brew doctor) fix some random broken symlinks
    - brew prune # (B)
    # (brew doctor) fix the many missing deps on gnupg and others
    - brew missing | cut -d':' -f1 | xargs brew uninstall
    # (brew doctor) fix broken symlinks (oclint and cloog)
    - brew prune
    # (brew doctor) fix "your XQuartz (2.7.9) is outdated" if it is installed
    - brew cask uninstall xquartz || true
    # (brew test-bot on travis) fix libyaml 0.1.6_1 already installed
    @@ -94,7 +117,6 @@ jobs:
    - &run-on-linux
    os: linux
    env: OS=x86_64_linux
    rvm: 2.3
    before_install:
    # Fix the permission problem on linux (664 instead of 644) during
    # git clone (the one done by travis-ci). Homebrew needs formulas to be
    @@ -108,26 +130,27 @@ jobs:

    # Instal linuxbrew
    - export PATH="/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:$PATH";
    - yes '' | sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)";
    # I added 'brew vendor-install ruby' because sometimes the install would
    # fail on 'Homebrew must be run under Ruby 2.3!' error.
    - yes | sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)" || (brew vendor-install ruby && brew update --force)
    # Fix a `brew doctor` error on "config" scripts for some reason
    - sudo rm -f /home/travis/.phpenv/shims/php-config
    /opt/pyenv/shims/*-config /usr/local/clang-3.9.0/bin/llvm-config
    /opt/pyenv/shims/*-config /usr/local/clang-*/bin/llvm-config
    before_cache:
    - brew cleanup

    - <<: *run-on-linux
    stage: deploy
    if: branch = master AND type != pull_request
    if: branch = master AND NOT type IN (pull_request, cron, api)
    os: linux
    rvm: 2.3
    env: OS=any_linux
    script:
    # This hook will add [ci skip] the the commit messages for bottle DSL
    - echo 'echo "\n\n[ci skip]" >> "$1"' > .git/hooks/commit-msg; chmod +x .git/hooks/commit-msg
    - if cp -v ~/shared/* . ; then
    brew test-bot --ci-upload --git-name=maelvalais [email protected] --bintray-org=touist --verbose;
    else
    echo "==> No bottle downloaded, cannot run --ci-upload";
    echo "==> No bottle found in the bucket, skipping --ci-upload";
    fi
    after_script:
    # Clean the bucket
    @@ -136,4 +159,4 @@ jobs:
    after_script:
    # In case the AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY are not
    # available, we don't want the build to fail.
    - aws s3 sync ~/shared s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} || true
    - aws s3 sync ~/shared s3://homebrew-touist-travis/${TRAVIS_BUILD_NUMBER} || true
  21. maelvls revised this gist Dec 8, 2017. 1 changed file with 52 additions and 15 deletions.
    67 changes: 52 additions & 15 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -47,21 +47,6 @@ about it. I only rely on `gmp` at runtime so maybe it is fine... Note that
    this problem has been also descussed [here](https://github.com/Homebrew/brew/issues/2572#issuecomment-298725510)
    and [here](https://github.com/Homebrew/brew/issues/3346).

    ## Note on wierd `prefix:` and `cellar:` in bottle DSL

    WARNING: in my solution, in the deploy state, I do on a linux build:

    - `brew test-bot --ci-upload`

    which packs all bottles at the same time and I get weird `prefix: /usr/local` and
    `cellar: :any_skip_relocation`. Intead, I should do:

    - first consider the mac bottles and do `brew test-bot --ci-upload`
    - then consider the linux bottle and do `brew test-bot --ci-upload --keep-old` (like in the Linuxbrew thing)

    If on macos has `:any_skip_relocation`, then the linux one will add
    `cellar :any_skip_relocation if OS.mac?` if it is different on linux.

    ### What you need

    1) the name of the Github repo for your tap must be `<user>/homebrew-<tap>`
    @@ -118,3 +103,55 @@ If on macos has `:any_skip_relocation`, then the linux one will add
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
    will disappear (as they are only there when the formula is new).

    ### Note on wierd `prefix:` and `cellar:` in bottle DSL

    WARNING: in my solution, in the deploy state, I do on a linux build:

    - `brew test-bot --ci-upload`

    which packs all bottles at the same time and I get weird `prefix: /usr/local` and
    `cellar: :any_skip_relocation`. Intead, I should do:

    - first consider the mac bottles and do `brew test-bot --ci-upload`
    - then consider the linux bottle and do `brew test-bot --ci-upload --keep-old` (like in the Linuxbrew thing)

    If on macos has `:any_skip_relocation`, then the linux one will add
    `cellar :any_skip_relocation if OS.mac?` if it is different on linux.

    Example:
    MacOS `lingeling-151109.el_capitan.bottle.json` and `lingeling-151109.sierra.bottle.json` have:
    ```json
    {
    "prefix": "/usr/local",
    "cellar": "any_skip_relocation",
    }
    ```
    but the linux bottle `lingeling-151109.x86_64_linux.bottle.json` has:
    ```json
    {
    "prefix": "/home/linuxbrew/.linuxbrew",
    "cellar": "/home/linuxbrew/.linuxbrew/Cellar",
    }
    ```
    and it wrongly became the following bottle DSL:

    ```ruby
    bottle do
    root_url "https://dl.bintray.com/touist/bottles-touist"
    sha256 "141f132d5ed58f930ada22fbe81d63e4115b9f78b8fb2ca3fa266e30e5fdf0a3" => :sierra
    sha256 "be1e33155185e36cbfbc48a743efd14d65edbfc174d2e31e4386414ea2153233" => :el_capitan
    sha256 "b24481b3ac4a7b02114ea5d627aff7cd6e2bb1551e4b4f073d3aa1d580f787c8" => :x86_64_linux
    end
    ```
    It should have been
    ```ruby
    bottle do
    root_url "https://dl.bintray.com/touist/bottles-touist"
    cellar :any_skip_relocation if OS.mac?
    sha256 "141f132d5ed58f930ada22fbe81d63e4115b9f78b8fb2ca3fa266e30e5fdf0a3" => :sierra
    sha256 "be1e33155185e36cbfbc48a743efd14d65edbfc174d2e31e4386414ea2153233" => :el_capitan
    sha256 "b24481b3ac4a7b02114ea5d627aff7cd6e2bb1551e4b4f073d3aa1d580f787c8" => :x86_64_linux
    end
    ```
    ### Notes on my discoveries of Linuxbrew
    - When a formula is linuxbrew-only, it has the comment `# tag "linuxbrew"` in it.
  22. maelvls revised this gist Dec 8, 2017. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -59,7 +59,8 @@ which packs all bottles at the same time and I get weird `prefix: /usr/local` an
    - first consider the mac bottles and do `brew test-bot --ci-upload`
    - then consider the linux bottle and do `brew test-bot --ci-upload --keep-old` (like in the Linuxbrew thing)


    If on macos has `:any_skip_relocation`, then the linux one will add
    `cellar :any_skip_relocation if OS.mac?` if it is different on linux.

    ### What you need

  23. maelvls revised this gist Dec 8, 2017. 1 changed file with 13 additions and 0 deletions.
    13 changes: 13 additions & 0 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -47,6 +47,19 @@ about it. I only rely on `gmp` at runtime so maybe it is fine... Note that
    this problem has been also descussed [here](https://github.com/Homebrew/brew/issues/2572#issuecomment-298725510)
    and [here](https://github.com/Homebrew/brew/issues/3346).

    ## Note on wierd `prefix:` and `cellar:` in bottle DSL

    WARNING: in my solution, in the deploy state, I do on a linux build:

    - `brew test-bot --ci-upload`

    which packs all bottles at the same time and I get weird `prefix: /usr/local` and
    `cellar: :any_skip_relocation`. Intead, I should do:

    - first consider the mac bottles and do `brew test-bot --ci-upload`
    - then consider the linux bottle and do `brew test-bot --ci-upload --keep-old` (like in the Linuxbrew thing)



    ### What you need

  24. maelvls revised this gist Nov 13, 2017. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -96,9 +96,9 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

    * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)
    * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)

      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
  25. maelvls revised this gist Nov 13, 2017. 1 changed file with 5 additions and 4 deletions.
    9 changes: 5 additions & 4 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -94,10 +94,11 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    9) the tests/building of the bottle only happens when you push a series of commit on ONE single formula;
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):
    * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

    * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)

      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
  26. maelvls revised this gist Nov 13, 2017. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -94,10 +94,11 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    9) the tests/building of the bottle only happens when you push a series of commit on ONE single formula;
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):
      * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):
    * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)

      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
    will disappear (as they are only there when the formula is new).
  27. maelvls revised this gist Nov 13, 2017. 1 changed file with 4 additions and 6 deletions.
    10 changes: 4 additions & 6 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -94,12 +94,10 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    9) the tests/building of the bottle only happens when you push a series of commit on ONE single formula;
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

      * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)

    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):
      * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)
      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
    will disappear (as they are only there when the formula is new).
  28. maelvls revised this gist Nov 13, 2017. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -95,10 +95,10 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

      - New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    - GitHub fork (not canonical repository)
    - GitHub repository too new (<30 days old)
      * New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    * GitHub fork (not canonical repository)
    * GitHub repository too new (<30 days old)

      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
  29. maelvls revised this gist Nov 13, 2017. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -96,9 +96,9 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

      - New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    - GitHub fork (not canonical repository)
    - GitHub repository too new (<30 days old)
      - New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    - GitHub fork (not canonical repository)
    - GitHub repository too new (<30 days old)

      First, make sure the bottles build without errors and that the only step failing is `brew audit --online`.
    Then, accept the PR and make a small change to the formula to force the rebuild of the bottle. The errors
  30. maelvls revised this gist Nov 13, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion How-to-automate-build-bottles-your-tap.md
    Original file line number Diff line number Diff line change
    @@ -95,7 +95,7 @@ and [here](https://github.com/Homebrew/brew/issues/3346).
    if you include some edits to .travis.yml for example, the push won't trigger build of the bottle. This
    caused me a lot of time lost on try and errors.
    9) when creating a new formula, you may get these kind of errors during the build (on PR, branch and master):

      - New formulae should not require patches to build. Patches should be submitted and accepted upstream first.
    - GitHub fork (not canonical repository)
    - GitHub repository too new (<30 days old)