RHEL10 and Fedora44 migrations
Did a number of reinstalls for RHEL9 hosts moving to RHEL10 and some Fedora 42 hosts moving to Fedora 44. Most of these are pretty easy, just setup things and run ansible, but there's a few tricky hosts that are not in our main datacenter I've been trying to do.
This includes a donated server that was 12 years old. It's served long and well, but it's too old for RHEL10. Luckily the donating company was happy to provision us a newer/better host.
Coming up soon will be moving the koji builders from f43 to f44.
I'm hoping we can get the bulk of this done before flock.
502's and AI
For a while now we have had sporadic 502 ( thats "Bad Gateway") errors on koji and to a more limited extend on src.fedoraproject.org. They have been super hard to track down and we have tried a number of adjustments based on a number of theories.
This week, I decided to try and really find the cause, and why not also try some of these AI agents that are so good these days. So, I spent a lot of time on monday with claude trying to get somewhere. I found claude to be somewhat useful as a rubber duck and it did point me in some good directions at first. However, it seemed to loose track of context that happened in the very same session, like we determined that apache was not logging any 502's at all, but it kept asking me to enable apache debugging and look at apache logs. no. It also seemed amusingly unaware of anubis ("what is this anubis application?"). It also did help me actually make a patch for anubis to add debugging as I know not much about golang. I was able to add that and get more clarity as to what was happening. On the other side I felt... off after using it much of the day monday. It may have been the way I was using it, but it seemed like it was directing the conversation and it was easy for me to just go along, but actually figuring things out required me to think about how things were setup more and be more pointed in questions. I think if I hadn't known a lot about how things were setup, and just let it drive it would have resulted in no answers and a lot of wasted time/tokens/efforts. So, I am still somewhat of a skeptic. I think there are uses for AI, but it's a tool that isn't good for everything.
The actual problem is that on POST requests (only), sometimes, apache is sending a 200 back from the backend with the results of the request and anubis gets a EOL when reading it. This causes anubis to send the 502 back to the user.
I am not sure why this happens. Some possible theories:
There's a configuration problem with apache and somehow it's tearing down reverseproxy replies before anubis can finish reading them.
There's a bug in apache doing above.
There's a bug in go's proxy support thats causing it to not read some replies correctly.
something else
I did file an anubis bug, but unclear if anubis is really to blame here.
So, since this is only happening on POSTs, and since we already just ALLOW those in anubis (ie, it doesn't challenge on POSTs), I just set things to bypass anubus for POST requests (for koji.fedoraproject.org and src.fedoraproject.org).
This works around the bug/issue for now and users should no longer see 502s.
If you do see one, I am keeping the ticket about this open until monday: https://forge.fedoraproject.org/infra/tickets/issues/12913
kernel security updates whirlwind
There was a series of kernel security issues this week. I helped out pushing them out to stable updates in a timely manner, but also I have:
Added two more builders to the secure-boot channel. Should allow more kernel builds to happen and the ones that are to be faster.
For some reason (likely my fault) there were only a few ppc64le builders available in the secureboot channel. I added tons more. This was causing kernel builds to sometimes sit in buildSRPMfromSCM jobs to make the initial src.rpm. Should be better now
We put mitigations in place for hosts that have local users, but we will likely be doing a update/reboot cycle soon. Week after next perhaps?
s390x maintainer test instance
I decided to poke at the s390x maintainer test instance again. I had managed to get resources from the LinuxONE community cloud a long while back, but they do not offer (or have any plans to offer) Fedora instances.
I tried a number of kexec tricks to get a rhel9 instance to reboot into the fedora 44 installer without much luck. Finally I was able to get a script from Dan Horak that did all the right magic.
So, the instance installed just fine after that and I got it all setup.
s390x-test01.fedorainfracloud.org should be available for packagers to test package builds on now.











You visited Fedora and CentOS at Red Hat Summit 2026
︎




So, let’s see what is included in this new release for the Fedora Atomic Desktop variants (Silverblue, Kinoite, Sway Atomic, Budgie Atomic and COSMIC Atomic).


comments? additions? reactions?
As always, comment on the fediverse: https://fosstodon.org/@nirik/116585359828306662