Photo by Mia Golic on Unsplash
This article will be helpful for anyone interested in setting up a local Kubernetes dev/test environment in a reproducible and easy way. Source code for this blog is available in this companion repository.
Photo by Mia Golic on Unsplash
This article will be helpful for anyone interested in setting up a local Kubernetes dev/test environment in a reproducible and easy way. Source code for this blog is available in this companion repository.
Photo by Markus Winkler on Unsplash
Using runbooks can streamline and improve working with Kubernetes by automating repeatable tasks. Bonus point, we can do this using only open source tools.
This article is will be helpful to anyone interested in modern complex software development.
In this article we are going to learn how to automate the provisioning of cloud resources via Crossplane and combine it with GitOps practices.
You will most benefit from this blog if you are a Platform or DevOps Engineer, Infrastructure Architect or Operations Specialist.
If you are new to GitOps, read more about it in my blog GitOps with Kubernetes
Let's set the stage by imagining following context. We are working as a part of a Platform Team in a large organization. Our goal is to help Development Teams to onboard get up to speed with using our Cloud Infrastructure. Here are a few base requirements:
The requirements lead us to an initial architecture proposal with following high level solution strategy.
In real world scenario we would manage Crossplane also using Flux, but for demo purposes we are focusing only on the application level.
The developer experience should be similar to this:
Knowing the requirements and initial architecture, we can start selecting the tools. For our example, the tools we will use are Flux and Crossplane.
We are going to use Flux as a GitOps engine, but the same could be achieved with ArgoCD or Rancher Fleet.
Let's look at the architecture and use cases that both tools support.
Flux exposes several components in the form of Kubernetes CRDs and controllers that help with expressing a workflow with GitOps model. Short description of 3 major components. All those components have their corresponding CRDs.
source: https://github.com/fluxcd/flux2
Source Controller Main role is to provide standardized API to manage sources of the Kubernetes deployments; Git and Helm repositories.
```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 1m
url: https://github.com/stefanprodan/podinfo
```
Kustomize Controller This is a CD part of the workflow. Where source controllers specify sources for data, this controller specifies what artifacts to run from a repository.
> This controller can work with kustomization files, but also plain Kubernetes manifests
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
name: webapp
namespace: apps
spec:
interval: 5m
path: "./deploy"
sourceRef:
kind: GitRepository
name: webapp
namespace: shared
```
Helm Controller This operator helps managing Helm chart releases containing Kubernetes manifests and deploy them onto a cluster.
```yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: backend
namespace: default
spec:
interval: 5m
chart:
spec:
chart: podinfo
version: ">=4.0.0 <5.0.0"
sourceRef:
kind: HelmRepository
name: podinfo
namespace: default
interval: 1m
upgrade:
remediation:
remediateLastFailure: true
test:
enable: true
values:
service:
grpcService: backend
resources:
requests:
cpu: 100m
memory: 64Mi
```
Let’s look how the Crossplane component model looks like. A word of warning, if you are new to Kubernetes this might be overwhelming, but there is value in making an effort to understand it. The below diagram shows the Crossplane component model and its basic interactions.
Source: Author based on Crossplane.io
Learn more about Crossplane in my blog "Infrastructure as Code: the next big shift is here"
If you want to follow along with the demo, clone this repository, it contains all the scripts to run the demo code.
In this demo, we are going to show how to use Flux and Crossplane to provision an EC2 instance directly from a new GitHub repository. This simulates a new team on boarding to our Platform.
To follow along, you will need AWS CLI configured on your local machine.
Once you obtain credentials, configure default profile for AWS CLI following this tutorial.
Locally installed you will need:
Run make
in the root folder of the project, this will:
If you are running on on Mac, use
make setup_mac
instead ofmake
.
Following tools need to be installed manually
IMPORTANT: The demo code will create a small EC2 Instance in eu-centra-1 region. The instance and underlying infrastructure will be removed as part of the demo, but please make sure all the resources were successfully removed and in case of any disruptions in the demo flow, be ready to remove the resources manually.
make
, this will install Crossplane with AWS provider and configure secret to access selected AWS accountFlux CLI was installed as put of the Makefile scrip, but optionally you can configure shell completion for the CLI
. <(flux completion zsh)
Refer to the Flux documentation page for more installation options
export GITHUB_TOKEN=<token copied form GitHub>
export GITHUB_USER=<your user name>
Flux will look for GITHUB_USER and GITHUB_TOKEN variables and once found will create a private repository on GitHub where Flux infrastructure will be tracked.
flux bootstrap github \
--owner=${GITHUB_USER} \
--repository=flux-infra \
--path=clusters/crossplane-cluster \
--personal
Now we will install a Crossplane Composition that defines what cloud resources to crate when someone asks for EC2 claim.
setup Crossplane composition and definition for creating EC2 instances
kubectl crossplane install configuration piotrzan/crossplane-ec2-instance:v1
fork repository with the EC2 claims
gh repo fork https://github.com/Piotr1215/crossplane-ec2
answer YES when prompted whether to clone the repository
clone the flux infra repository created in your personal repos
git clone git@github.com:${GITHUB_USER}/flux-infra.git
cd flux-infra
add source repository to tell Flux what to observe and synchronize
Flux will register this repository and every 30 seconds check for changes.
execute below command in the flux-infra repository, it will add a Git Source
flux create source git crossplane-demo \
--url=https://github.com/${GITHUB_USER}/crossplane-ec2.git \
--branch=master \
--interval=30s \
--export > clusters/crossplane-cluster/demo-source.yaml
the previous command created a file in clusters/crossplane-cluster sub folder, commit the file
git add .
git commit -m "Adding Source Repository"
git push
execute kubectl get gitrepositories.source.toolkit.fluxcd.io -A
to see active Git Repositories sources in Flux
setup watch on the AWS managed resources, for now there should be none
watch kubectl get managed
create Flux Kustomization to watch for specific folder in the repository with the Crossplane EC2 claim
flux create kustomization crossplane-demo \
--target-namespace=default \
--source=crossplane-demo \
--path="./ec2-claim" \
--prune=true \
--interval=1m \
--export > clusters/crossplane-cluster/crossplane-demo.yaml
git add .
git commit -m "Adding EC2 Instance"
git push
after a minute or so you should see a new EC2 Instance being synchronized with Crossplane and resources in AWS
Let's take a step back and make sure we understand all the resources and repositories used.
The first repository we have created is what Flux uses to manage itself on the cluster as well as other repositories.
In order to tell Flux about a repository with Crossplane EC2 claims, we have created a GitSource
YAML file that points to HTTPS address of the repository with the EC2 claims.
The EC2 claims repository contains a folder where plain Kubernetes manifest files are located. In order to tell Flux what files to observe, we have created a Kustomization
and linked it with GitSource
via its name. Kustomization
points to the folder containing K8s manifests.
rm ec2-claim/claim-aws.yaml
git add .
git commit -m "EC2 instance removed"
the ec2-claim folder must be present in the repo after the claim yaml is removed, otherwise Flux cannot reconcile
In case you cannot use the repository, it's possible to cleanup the resources by deleting them from flux.
flux delete kustomization crossplane-demo
will remove all the resources from the cluster and AWSkubectl delete VirtualMachineInstance sample-ec2
watch kubectl get managed
output doesn't contain any AWS resourcesmake cleanup
flux-infra
repositoryGitOps with Flux or Argo CD and Crossplane offers a very powerful and flexible model for Platform builders. In this demo, we have focused on applications side with Kubernetes clusters deployed some other way, either with Crossplane or Fleet or Cluster API etc.
What we achieved on top of using Crossplane’s Resource Model is the fact that we do not interact with kubectl directly any longer to manage resources, but ratter delegate this activity to Flux. Crossplane still runs on the cluster and reconciles all resources. In other words, we've moved the API surface from kubectl to Git.
This documentation assumes basic knowledge of Kubernetes and kubectl. To learn or refresh on container orchestration related concepts, please refer to the official documentation:
In 2017, a cloud-native company Weaveworks release a blog post called “GitOps — Operations by Pull Request”. The post introduces the term GitOps defining it as using Git as a source of truth to operate almost anything. Since then GitOps movement has been growing and gaining in popularity.
Software development tooling and processes have evolved rapidly in last decade to meet growing needs of developers. On top of mastering, often a few, programing languages and paradigms, software developers must learn to navigate increasingly complex landscape of tools and processes.
Working with local Kubernetes cluster such as minikube, k3s, microk8s or others is great for testing new features, experimenting and running POCs. Once you are ready with a cool new functionality or just want to share quickly results of your work with colleagues or customers, well you have to push everything to an online cluster. It might not be an issue if you have good CI/CD pipeline setup, but most of the time it’s simply too much effort for a simple one-off demo.
Saying that Kubernetes is becoming mainstream would be an understatement. In fact, it has influenced how modern distributed systems are designed and operated. By abstracting away infrastructure concerns we are able to leverage Kubernetes as a “platform to build platforms” or “cloud operating system” with lots of obvious benefits, but also development and operational challenges.