I just spent x hours trying to upgrade our CI environment to match the AWS Lambda runtime. Rabbit hole galore happened.

In summary:

  • amazonlinux:2023 uses a version of glibc that depends on the syscall clone3 – when running yum update or even curl it failed on (6) getaddrinfo() thread failed to start
  • Docker out of the box doesn’t allow clone3
  • Upgrading from 18 to 20 to 22 of Ubuntu didn’t help
  • Upgrading from 20 to 22 broke openresolv, so tinc-up no longer correctly setup DNS
  • Upgrading from 20 to 22 moved to libssl3, which has a regression in Blowfish, rendering tinc confused as garbage comes in

Workarounds / Solutions

To get Tinc back to a working state, I had to:

  • use libssl3 from Ubuntu 23 (Kinetic) instead of Ubuntu 22 (Jammy) - a fix for this regression has been in master for over a year but hasn’t shipped!
  • enable legacy provider in libssl3’s config
  • modify tinc-up to use the new resolvectl to add domain and dns - legacy /etc/network/interfaces files still work, didn’t have to go all the way to netplan

To get to a place where we can use Amazon Linux 2023 docker images (with the newer glibc depending on clone3), I had to:

  • modify Docker’s daemon.json with a "seccomp-profile": "/etc/docker/wendt-default.json" allowing what is need

Also, see:

  • https://github.com/actions/runner-images/issues/3812
  • https://github.com/moby/moby/issues/42680
  • https://www.tinc-vpn.org/pipermail/tinc/2022-September/005609.html
  • https://launchpad.net/ubuntu/+source/openssl
  • https://github.com/gsliepen/tinc/issues/414
  • https://stackoverflow.com/questions/74959500/getaddrinfo-thread-failed-to-start-while-building-docker-image/76550121#76550121
  • https://askubuntu.com/questions/1263284/apt-update-throws-signature-error-in-ubuntu-20-04-container-on-arm
  • https://github.com/adoptium/containers/issues/215#issuecomment-1142046045 – this summed it all up
  • https://bugzilla.redhat.com/show_bug.cgi?id=1990469
  • https://github.com/amazonlinux/amazon-linux-2023/issues/181
  • https://github.com/amazonlinux/amazon-linux-2023/issues/80
  • https://docs.aws.amazon.com/lambda/latest/dg/images-test.html
This work by Fredrik Wendt is licensed under CC by-sa.