OP (oppenheimer) Ompragash

Docker Buildx Bake with Harness CI: Advanced Multi-Architecture Build Guide

Introduction

Docker Buildx Bake is a powerful, declarative build orchestration tool designed for advanced Docker users who need to manage complex, multi-target build scenarios. This guide demonstrates how to leverage Bake's capabilities within Harness CI pipelines for sophisticated container image workflows.

What is Docker Buildx Bake?

Declarative Configuration: Define complex build scenarios using HCL (HashiCorp Configuration Language) files, enabling version-controlled, reproducible build definitions (Docker Buildx Bake Documentation)
Multi-Target Orchestration: Build multiple related images simultaneously with shared configurations, inheritance, and dependency management

Developer: Your environment includes ast-grep. For searches requiring syntax-aware or structural matching, always use ast-grep --lang rust -p '', adjusting the --lang flag for the relevant language. Only use plain-text tools (such as rg or grep) if specifically instructed to perform a plain-text search. Begin with a concise checklist (3-7 bullets) of planned sub-tasks before starting any search. Before running any search command, succinctly state the purpose and the search pattern to be used. After executing a search, briefly validate whether the results satisfy your intent and, if not, self-correct before proceeding.

Using OIDC Authentication with Drone-gcs Plugin for Download Operations

In this gist, we'll explore how to leverage OIDC-based authentication with the Drone plugins/gcs plugin, focusing specifically on download operations. While the plugin's default behavior is to upload files from the local harness build node to a specified GCS bucket, we can reverse this behavior for download purposes when needed.

Understanding the Upload and Download Modes of Drone GCS Plugin

1. Default Operation (Upload Mode)

By default (i.e. when Download is false), the Drone-gcs plugin uploads files. In this mode, it treats:

Source: The local file system on the harness build node.
Target: The destination GCS bucket (extracted from the Target configuration).

What is InstructLab?

InstructLab is an open-source AI project developed by IBM and Red Hat. It focuses on enhancing the fine-tuning of Large Language Models (LLMs) by minimizing dependence on GPUs and human-generated data. This approach facilitates easier integration and ongoing model improvement.

InstructLab incorporates several key processes:

Taxonomy-driven data curation
Large-scale synthetic data generation
A multi-phased instruction-tuning method

Prerequisites

Argo CD Quick Start Guide

1. Install Kubernetes (k3s - Lightweight, Production-Grade)

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.29.6+k3s2 sh -s - --write-kubeconfig-mode 644

2. Install Argo CD CLI

VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

Drone AWS OIDC Plugin

When the plugins/aws-oidc plugin is used in a Harness CI pipeline stage, CI Manager automatically retrieves the OIDC token during pipeline execution. This token is passed to the PLUGIN_OIDC_TOKEN_ID environment variable.
The plugin uses this token (PLUGIN_OIDC_TOKEN_ID) to generate temporary AWS credentials, which can be utilized in subsequent steps to perform AWS operations.
These temporary credentials are exported as masked secrets. They can be referenced in later steps using <+execution.steps.awsOidc.output.outputVariables.AWS_ACCESS_KEY_ID> but cannot be viewed directly.
See the sample pipeline YAML below for a complete example.

Sample Harness CI Pipeline YAML

            steps:

sudo docker push fossmatters.jfrog.io/ossops-docker/ubuntu:1.0.0
The push refers to repository [fossmatters.jfrog.io/ossops-docker/ubuntu]
a30a5965a4f7: Pushed
1.0.0: digest: sha256:19bc204df71f4086020b609089ebf49b332c2e373ec31e3512644b8ad9615001 size: 529

cat query.json
{

	version: 3
	domain: philosophy
	created_by: ompragash
	seed_examples:
	- context: \|
	Aristotle's view of the universe included a geocentric model where Earth is at the center. He described a cosmos divided into two distinct realms: the terrestrial (sublunar) and the celestial. Terrestrial included four elements: earth, water, air, and fire, with earth and water being heavy and sinking toward the center, while air and fire were light and moved outward. Celestial realm was thought to be perfect and immutable, composed of a fifth element, or quintessence. Aristotle's physics or natural philosophy was deeply intertwined with his metaphysical ideas, where he emphasized a world intelligible and structured according to natural laws.
	questions_and_answers:
	- question: \|
	What are the key components of Aristotle's geocentric model of the universe?
	answer: \|

	FROM alpine:latest
	# Install basic packages
	RUN apk add --no-cache bash

	RUN --mount=type=secret,id=ARTIFACTORY_ACCESS_TOKEN \
	echo "Token is: $ARTIFACTORY_ACCESS_TOKEN"

	# Set the default command to run when starting the container
	CMD ["bash"]

	time="2024-12-06T05:58:13Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"
	time="2024-12-06T06:00:13Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
	time="2024-12-06T06:00:15Z" level=info msg="Setting new image to opensourceops/my-webapp:1.0.10" alias=my-webapp application=my-webapp image_name=opensourceops/my-webapp image_tag=1.0.9 registry=
	time="2024-12-06T06:00:15Z" level=info msg="Successfully updated image 'opensourceops/my-webapp:1.0.9' to 'opensourceops/my-webapp:1.0.10', but pending spec update (dry run=false)" alias=my-webapp application=my-webapp image_name=opensourceops/my-webapp image_tag=1.0.9 registry=
	time="2024-12-06T06:00:15Z" level=info msg="Committing 1 parameter update(s) for application my-webapp" application=my-webapp
	time="2024-12-06T06:00:15Z" level=info msg="Starting configmap/secret informers"
	time="2024-12-06T06:00:15Z" level=info msg="Configmap/secret informer sync