hold off on packages until dependencies are built #281
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
I've been having problems every now and then when I happen to upgrade my system while builds are running - e.g. currently I seem to have some problem regarding qt6 having some packages upgraded, but not others, causing various programs to just crash immediately.
since it's quite annoying to fix this manually (need to temporarily swap priorities between ALHP and normal arch packages, and manually upgrade packages, which also means I won't have the ALHP variant until the next upgrade of those packages) I would really welcome it if the build process kept co-dependent packages (incl. optdepends) from appearing in repos until all of them are built.
The easiest way to do this would be to ignore dependencies completely and just batch it the same way upstream does (i.e. complete compiling all upstream changes and only then push packages to prod) but that might result in unnecessarily long delays before upgrades trickle down, as well as increased storage requirements (though, I don't think either downside is that bad compared to everyones' systems breaking occasionally)
look closely - the situation appears because you are upgrading packages that are taken from the Arch repositories because they are either ANY packages or packages that failed before or packages that are blacklisted - TLDR: packages that are not present in ALHP - that is all done while the packages present in ALHP are still being built.
ALHP can't hold your packages on your system if they come from the Arch repositories. That's up to you.
how the hell did that happen? 😕 I'm 100% sure I never manually installed the non-ALHP
qt6-base, and I know pacman doesn't use a lower-prio repo if a higher-prio repo has the requested package, even if the lower-prio repo has a newer version. And I can see that ALHP does haveqt6-basebuilt so I have no idea how this happened.Please keep this issue open and I will report back next time something like this happens with more info.
You can uncomment the line
#VerbosePkgListsin your/etc/pacman.conf(to readVerbosePkgLists) to make it easier to spot non-ALHP updates.It can happen that another Arch - package pulls in an Arch package as a dependency although it does have a corresponding package in ALHP but that one does not have the required version yet - but that would be a modification in pacman or the way that Arch handles dependencies, not something that ALHP can change.
That being said, it might be possible to write a pacman hook that checks if packages are still being built in ALHP when trying to update - but I won't have time to help with that, not sure how @anonfunc sees that
this could theoretically be fixed by serving non-ALHP packages like a normal arch mirror instead of relying on pacman to fall back on regular arch mirrors. This way you could remove upstream arch from the list of repos and eliminate the chance of this happening.
Yea I though about that, but that would mean we would need to freeze the mirror packages to the same build-cycles the buildbot runs at. certainly doable, but not sure I have time for that rn. But I'll have another look.
This is more or less already possible by utilizing https://somegit.dev/ALHP/ALHP.utils, you would just need to wire that up into a hook. But this also produces false-positives, since not every update may behave breaking in that sense.
just had another issue with unevenly upgraded packages. python got upgraded in ALHP, but a bunch of things that depended on it were still queued or building (in fact, still are) - basically half my system stopped working after I upgraded a few hours ago.
e.g.:
kittyis currently at0.45.0-3.1(currently queued for-4.1), but only0.45.0-4works with AHLP's currentpython=3.14.2-2.1.Yes, this is explained in the the FAQ here.
Yes, it is explained in the FAQ. But this issue is about preventing this - ideally, dependent packages would built, put into some sort of staging and then published at the same time, when all packages in the dependency chain are built.
Presumably arch's official repos follow similar pattern to this to avoid breakages?
please read the comments above.
What you want would mean that ALHP would need to provide all packages, not only the rebuilt packages.
This would mean a lot more storage space and traffic.
As ALHP is a community project and @anonfunc already invests a lot of time and money with no return, it is up to the users to handle the mismatch between the ALHP and Arch repo's during bigger rebuilds.
Just an FYI: Cachyos runs into the same problems with every big rebuild because they also use many packages directly from the Arch repo's.
I think the impact will be a lot smaller once we built testing as well. I have already prepared the changes locally, just need to test them. Probably with a snapshot + rollback approach.
@Riedler:
@BS86:
https://man.archlinux.org/man/pacman.conf.5.en#REPOSITORY_SECTIONS:
Note,
man 5 pacman.confsays nothing about dependencies, a package version from the first repository must always be used. I.e. if ALHP contains some package, then it will be installed, no matter of dependencies, dependencies of dependencies, version in official repository, etc. @BS86, if I understand you correctly, you disagree with that. Could you provide some links?If version requirement is set strictly and explicitly, e.g. with
=in thedependsarray of somePKGBUILDfile (no matter in what repository thatPKGBUILDis), and an ALHP package does not meet that requirement, then no magic will happen: pacman will complain and stop.But if something requires specific version of package implicitly, then, yes, welcome to the partial upgrade. To my understanding, usually it may happen with libraries, that is why "Application X fails to start because it links to an old/newer lib" is specially marked in the mentioned FAQ.
When building is in progress (which might take very long time—hours, or even days), ALHP may be internally inconsistent, so update on your own risk.
I hope I summarized it right. But I'm open to learn something new:D
I understand that much of what we've said is documented behaviour.
yes, attempting to resolve the internal inconsistency in ALHP is what this issue is about.
The crux of the issue which you haven't mentioned is that ALHP currently doesn't build all arch packages, so even if ALHP is internally consistent by holding packages back until all of them are built, they may dep-conflict with packages that are updated in upstream before ALHP finishes compiling the current batch of packages. Basically, as long as you don't serve literally all packages, there is a risk of partial upgrades.
I left out some details for brevity. future readers may want to read the rest of the convo for full context.
Ok, I see. What would you recommend from your experience?
To be clear, I'm trying to coin some tips for ALHP in ArchWiki. Imagine you are a newcomer:)
No clue, I'm rawdogging it atm. didn't have time to write a hook yet (the one @anonfunc proposed earlier in the thread) so I just manually fix my system whenever it breaks.
I think the best thing to do if you've got the time for it is to write a hook for pacman to block upgrades (or at least warn the user with a y/n continue) while ALHP is busy building something, and then recommend using that in the arch wiki. see #281 (comment)