Switch to executor architecture
Does a lot of porting work of the gateway/systemd component to the executor API. This MR also contains the outline for a podman-based executor. It also wraps a lot of small improvements, like reference counting for mounts, and a separate ipam module.
I will put some more testing into it and try to work out some of the kinks with the new IPAM. Documentation should also be ready before we merge it. Afterward, this will likely be merged into main and serve as the new base for an executor system.
@blohmar, I would like to please request your review once this leaves Draft status.
It makes quite a lot of changes to the code structure.
Background
Initially, the successor to the systemd executor was supposed to be a firecracker-based executor. However, this was abbandoned because firecracker appears to not support any directory-based fileshares for VMs.
Afterward, Cloud Hypervisor (ch) was evaluated however it also this eventually also was put in the planned list, because any usage would require most of systemd executor, because each ch process can only run on instance. This could be made workable by, e.g., building on libvirt however libvirt lacks the capability to configure fileshares for ch. I would eventually like to switch to ch with libvirt but this would likely require submitting the required changes upstream to libvirt.
Eventually, I settled on planning to integrate with either docker or podman as container systems. Both will effectively offer the same level of isolation as the currnet systemd executor (cgroup + namespaces).
This was then cut down to podman because, in my personal experience, it has had better tooling for server use-cases. It also features a mature and stable golang API. Docker might eventually also be considered but podman is good enough for now.