Monday, 15. April 2024Week 16
Saturday, 13. April 2024Week 15
Thursday, 11. April 2024Week 15

NTP IPs

This post from Rachel, reminded me of my own struggle with a Raspberry Pi and time (and yes, I did run into the same DNSSEC problem).
I first tried a RTC shield, but this didn't fully solve all problems.

My workaround in the end is to use two fixed IP addresses in ntp.conf in additon to the usual pool servers.
Thanks ${previous employer} for not changing the IPs of your public NTP servers so far :-)

Wednesday, 10. April 2024Week 15

SixSpotting still going on

While browsing through the archives I stumbled upon the SixSpotting post from 2014.
Turns out the game is still working and after some failed attempts I managed to remember my account name and log in.
I guess it must look quite strange to see a burst of new checkins after almost 10 years 😄

My last 5 SixSpotting checkins

Sunday, 7. April 2024Week 14

humans.txt

After following a couple links from the article linked in the previous blogpost, I ended up reading the Website Component Checklist from Mike Sass.
It provides again a lot of inspiration for things to add to my weblog.
What intrigued me today on the list was humans.txt, which is an initiative for knowing the people behing a website.

Thus I've now added the following humans.txt.

Wednesday, 3. April 2024Week 14
Saturday, 30. March 2024Week 13

Puppet updated!

Yay! Successfully updated my Puppet Server setup from 5.3.7 to 8.4.0 🎉

It was quite a step (5.3.7 was released in January 2019) and as expected 3 major version bumps came with a couple changes.

I opted to re-create the PuppetDB and CA stores from scratch (to avoid having to migrate 5 years of data schema changes, and the CA cert is now also valid for a couple more years again).

To make the old manifests and modules work with the new versions, quite some effort was needed. This included rewriting some no longer maintained modules to use newer stdlib and concat libraries, updating a couple modules from the puppet forge (with the bonus that my puppet server runs airgapped and I had to use the download-tar-copy-extract way to install them) and fixing no longer valid syntax here and there in my custom manifests. Overall I spent about 5 hours on it (and have now a recurring reminder to update puppet more often to make this process less painful).

Helpful as usual were the resources from Vox Pupuli, in particular the Puppet Server and PuppetDB Docker images and the CRAFTY repo which contains a fully self-contained Docker Compose setup very similar to what I'm running.

Some commands that came in handy:

puppet config print ssldir --section agent
Returns the path of the TLS config folder on the client. Useful during a CA change (where you rm -rf the whole folder and then request a new TLS certificate).

puppet agent -t --noop
Dry-run the changes on the client (it does request a new TLS cert though!). Shows a nice diff of the changes it would do to files, helpful to validate that a manifest still behaves the same in the new version.

Monday, 25. March 2024Week 13

Linux Crisis Tools

Brendan Gregg posted the following list of 'crisis tools' which you should install on your Linux servers by default (so they are available when an incident happens).

PackageProvidesNotes
procpsps(1), vmstat(8), uptime(1), top(1)basic stats
util-linuxdmesg(1), lsblk(1), lscpu(1)system log, device info
sysstatiostat(1), mpstat(1), pidstat(1), sar(1)device stats
iproute2ip(8), ss(8), nstat(8), tc(8)preferred net tools
numactlnumastat(8)NUMA stats
tcpdumptcpdump(8)Network sniffer
linux-tools-common
linux-tools-$(uname -r)
perf(1), turbostat(8)profiler and PMU stats
bpfcc-tools (bcc)opensnoop(8), execsnoop(8), runqlat(8), softirqs(8),
hardirqs(8), ext4slower(8), ext4dist(8), biotop(8),
biosnoop(8), biolatency(8), tcptop(8), tcplife(8),
trace(8), argdist(8), funccount(8), profile(8), etc.
canned eBPF tools[1]
bpftracebpftrace, basic versions of opensnoop(8),
execsnoop(8), runqlat(8), biosnoop(8), etc.
eBPF scripting[1]
trace-cmdtrace-cmd(1)Ftrace CLI
nicstatnicstat(1)net device stats
ethtoolethtool(8)net device info
tiptoptiptop(1)PMU/PMC top
cpuidcpuid(1)CPU details
msr-toolsrdmsr(8), wrmsr(8)CPU digging