gak.dev
Loading...
Loading...
I just deployed this blog! It took half a day to make the site in Leptos and around two days to deploy.
This post is a trail of my yak shaving journey to get this blog deployed. Maybe it'll help someone someday.
I run some VMs at home and thought to start deploying to Kubernetes instead of more VMs. Less overall RAM usage and has nice and easy rolling deployments. I've deployed and maintained kubernetes in a day job before so should be super easy and quick, right!?
Choose a k8s. I heard k3s is pretty lightweight. It says it right on their website. Let's do that. Just a single node for now.
Oooh found k3OS which supplies an ISO. Even better. No need to run a billion shell commands. Boot from ISO. Only few questions asked. Brilliant.
Need to update the k3OS config.yaml to supply my SSH public key, but can't SSH because password authentication is disabled. 🐣
Log in via a VM remote viewer. Turn on password auth, restart SSH. SSH in and update config.yaml with public key. Turn off password authentication. Restart more things. Works!
Copy over the Kubernetes config to ~/.kube/config, replace the IP address so I can access the cluster via kubectl.
Don't forget to make that file have restrictive permissions so other users can't see the secrets inside.
Get distracted after running kubectl too many times and make some aliases:
alias k="kubectl"
alias kg="k get"
alias kgy="k get -o yaml"
alias kga="kg all"
alias kl="k logs"
alias klf="k logs -f"
alias kd="k describe"
alias ke="k edit"
alias kpf="k port-forward"
alias kn="kubectl config set-context --current --namespace"
Start writing a small helper script to automatically extract and decode secrets...
kubectl get secret [secret-name] -o json | jq -r .field1.field2 | base64 -d
...but decide to actually continue this deployment instead.
This would be glorious with shell autocomplete. Imagine typing ksecret [tab][choose a secret in your namespace] [tab][either choose a key in the secret or just show the secret if it's the only one]!
Learn that you use Containerfile these days instead of Dockerfile. Syntax seems identical in my single small attempt. Let's do a two stage build to reduce cruft and source code:
FROM docker.io/rust:1-bullseye as builder
RUN rustup install nightly
RUN rustup default nightly
RUN rustup target add wasm32-unknown-unknown
RUN cargo install cargo-leptos
COPY . /app
WORKDIR /app
RUN cargo leptos build -r
FROM docker.io/debian:bullseye-slim
RUN mkdir /app
WORKDIR /app
COPY --from=builder /app/target/server/release/start-axum /app
COPY --from=builder /app/target/site/ /app/site/
COPY --from=builder /app/blog/ /app/blog/
ENV LEPTOS_OUTPUT_NAME "start-axum"
ENV LEPTOS_SITE_ROOT "site"
ENV LEPTOS_SITE_PKG_DIR "pkg"
ENV LEPTOS_SITE_ADDR "0.0.0.0:3000"
CMD ["/app/start-axum"]
I'm not sure what happened here. I had issues building this on my arm64 computer. Segfaults when building a Rust package.
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Gosh.
I soon also realised I need to target amd64 anyway because of the VM host, so I try some cross platform podman building.
Tried things like --platform linux/amd64 and SSHing in the podman machine and ran sudo -i rpm-ostree install qemu-user-static. Reboot the machine a lot. This didn't work. None of this worked. Someone said it's a qemu bug not fixed yet but the post is a year old. I'm sure there's a way, right?! I give up for now.
At this point just want to get this building without doing containers inside containers so let us use an existing VM on the target host arch.
No podman deb for this old LTS Ubuntu. The ppa doesn't work. No snap. Might as well do do-release-update. Answer lots of questions. Give it a reboot.
apt install podman # works
Create an entry in the justfile to build everything in one go:
sha := `git rev-parse --short HEAD`
build_remote_and_push:
rsync -av --exclude target --exclude .git . buildbox:~/gak-dev
ssh buildbox "cd gak-dev && podman build -t gak-dev ."
ssh buildbox "podman save gak-dev | podman load"
podman push gak-dev:latest registry.cooldom.com/gak-dev:{{sha}}
podman runs out of space during the build. Default size for the podman machine is 10GB. I'm hitting 10GB already with one image?
Run dust to see what's going on. Yup it's a mix of the podman image and existing build artifacts from other builds. Delete lots of things. The default VM disk size is small too.
Let's increase the VM disk size! Tell Proxmox this information.
Use fdisk to expand partition to match the disk. Change not detected, reboot again.
pvresize /dev/sda3
lvextend -l +100%FREE /dev/ubuntu-vg/root
resize2fs /dev/sda3
podman machine rm podman-machine-default
podman machine init --cpus 16 --disk-size 32
I'm not a fan of this process. I should make a script to do the resize for me. One day. 😶🌫️
Let's deploy! No wait. We need a container registry.
Want to self host on-prem. What's a good new registry that I can helm with? Randomly find zot. Looked good. Nice UI. Lots of stars. Recent commits. golang. Lots of boxes ✅.
Run the zot helm chart.
Wait! I need to install helm. Lets install helm with nix. Fix up my nix config. Rebuild.
That's odd. An error that my architecture isn't compatible? Linux only. Must be a mistake. I'm 1000% sure it is cross platform. Investigate further. Force a build? Nah better ask on the nix forums.
Someone replied really quickly saying helm only runs on Linux, but maybe I'm after kubernetes-helm?
🤦 Time to delete all my social accounts out of embarrassment. I should have checked the description. Wait, no. Just say thanks and carry on. There's a deployment to finish.
This other helm is "A free polyphonic synth with lots of modulation" with the last commit in 2018. Alright.
OK kubernetes-helm installed. Back to zot. Helm chart installed no problem.
Wait. It needs credentials otherwise someone naughty on my network can hack all the things. Could not find documentation on how to set up users when using helm to install. I can't work it out without peering through the dark corners of the source code. I dont need this. It feels like weeks so far. Nuke the helm. Nuke the namespace. Gotta be sure.
Let us install the classic Docker Registry via helm. The chart is slightly old (a year since the last release), but I'm sure it's fine.
Set up a values.yaml for the chart so I can configure and upgrade it later with the same settings. Include some htpasswd lines for credentials.
podman login: 403 credentials wrong. Strange.
Debug it a bit. Search the internet. Ask GPT. Turns out I needed to use htpasswd with bcrypt not the default crypt. I did see a note but forgot to think about it.
Update values.yaml with new htpasswd. Run again.
helm upgrade --install -f values.yaml twuni/docker-registry
OK, auth works!
Let's deploy! No wait. We need to connect to the registry outside of kubernetes via a LoadBalancer.
Time to install the ol' trusty, well known nginx controller helm chart.
That's odd, <pending> on the EXTERNAL-IP field.
... ?
There must be another controller running! Let's see:
# kg --all-namespaces service | grep Load
kube-system traefik LoadBalancer 10.43.109.201 192.168.8.101 80:32018/TCP,443:32404/TCP
nginx nginx-controller LoadBalancer 10.43.1.90 <pending> 80:30049/TCP,443:31344/TCP
k3s ships with something called traefik! Alright! Let's go with that. Uninstall the nginx controller. Nuke it all from orbit.
Create a new alias alias kall="kg --all-namespaces" because typing is hard.
Let's deploy! No wait. We want (need) TLS for the registry because security.
Already have a wildcard certificate because I run many subdomains internally only and dont want to expose on the net, and I'm too lazy make it magically work with ACME.
Create the tls secret and shove the reference in the registry ingress yaml. No problemo.
Let's deploy! No wait. We should probably use a persistent volume for storing registry images.
I've used Longhorn in the long past ago, with moderate success. Let us do that.
The docs are pretty good. Don't even need helm!
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.0/deploy/longhorn.yaml
Add a separate ingress, again with a tls secret.
Check out the internal service UI using port-forward. No authentication for now. Will fix it later because this is taking way too long.
Great! Now reconfigure docker registry to use the PV instead of local disk.
Hmm that's odd... there are two default PVs... local disk and longhorn... looks like there's a "default" attribute that's true for both. That doesn't seem right.
# kg storageclass
NAME PROVISIONER VOLUMEBINDINGMODE
longhorn (default) driver.longhorn.io Immediate
local-path (default) rancher.io/local-path WaitForFirstConsumer
(Snippet cropped so it doesn't do crazy unreadable wrapping)
Fix this silly business. It super silly because the value is-default-class key identified thing needs to be "false" as a string, not a bool!
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
Let's Encrypt for https goodness.
Install cert-manager. Go through the guide. Bit hard to concentrate on the different steps. Be nicer if they just made a single yaml example or a helm chart. Maybe they did, I just didn't look for it. I'm so close to the end.
Only a few hours of debugging to finally get the cert working perfectly. I have a strange network setup so it caused more problems than it should have. I also forgot why it took me so long, so no explanation here.
I was told somewhere I need to set up a tls secret with empty entries for a chicken and egg problem. I forgot if I actually needed to do it or not.
Let's deploy! Finally!
Create and apply some exciting yaml files: namespace, service, ingress, registry secret, letsencrypt issuer. Pretty standard stuff.
The odd one out is the Leptos deployment which needs some env vars:
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: gak-dev
name: gak-dev-deployment
spec:
replicas: 2
selector:
matchLabels:
app: gak-dev
template:
metadata:
labels:
app: gak-dev
spec:
containers:
- name: gak-dev-container
image: registry.cooldom.com/gak-dev
workingDir: /app
command: ["/app/start-axum"]
env:
- name: LEPTOS_OUTPUT_NAME
value: "start-axum"
- name: LEPTOS_SITE_ROOT
value: "site"
- name: LEPTOS_SITE_PKG_DIR
value: "pkg"
- name: LEPTOS_SITE_ADDR
value: "0.0.0.0:3000"
ports:
- containerPort: 3000
livenessProbe:
httpGet:
path: /blog/2023-07-14-hello-world
port: 3000
initialDelaySeconds: 60
periodSeconds: 10
imagePullSecrets:
- name: regcred
Chuck in the justfile to set the deployment to the latest image and monitor the deployment.
rollout:
kubectl -n gak-dev set image deployment/gak-dev-deployment gak-dev-container=registry.cooldom.com/gak-dev:{{sha}}
kubectl -n gak-dev rollout status deployment/gak-dev-deployment
Finally https://gak.dev/ is alive!
The moral of the story is don't deploy Kubernetes yourself unless you find the above "fun".