From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id C33FF1382C5 for ; Wed, 10 Feb 2021 23:20:12 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id B1CB6E09D9; Wed, 10 Feb 2021 23:20:08 +0000 (UTC) Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 87590E0909 for ; Wed, 10 Feb 2021 23:20:08 +0000 (UTC) Received: by mail-ej1-x635.google.com with SMTP id y9so7081738ejp.10 for ; Wed, 10 Feb 2021 15:20:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=asokolov-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=UYvqf7HvY5Mx1l4KXs+GqYYESylSEXtjfQAoAHChfG4=; b=yrW5CxmGIx6Jl7qcqE051oIahOF3HVCJfroCWhqwzLJlJIRkYls5npcKjAMtuy2fGF Br9njUTJEg3/NBqk9B2G4JsQr7ol9PtDdHJyEX8wJAVwDO3WN2EqHqKp3ihXAfx8sSse aeVtFRgcSD9q3jG+dWTu5HlgOdvgAJ38+zP1cmTjeMZ1YTbkvTkrE6DBXYmaPihpR4Ko +Nvna5KSw7InAw6h74JF+JTq7CKXBraUajeYzV0WSS4W8RqmkgqDWM1O9psIv9gyj+R4 HFgaLK9UrphJ5MCq3lc5uXQie1JkAmzEp75hu5aqi/4LI7ZGeyt50Vmb/p+ZfYCetWcR vKaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=UYvqf7HvY5Mx1l4KXs+GqYYESylSEXtjfQAoAHChfG4=; b=KRE31gVx34qhew9jjF9+nsyMhLPQuM+lTvuou/th4p9se29k3+iaHUbraZEckziDYc 3V1BATGveuogX2/mDGHb7cLIR5pOl6+0hGL9U0VjefOQdCiKXEEACb2npiDJdGlgsNB6 E0sINzIbuEhIZb0N+9r0KHqug1PgE5CB3A4peeC21lE6HjkK1PE6m/GnGgYM3i1kC3DO f3PiKGPSJSS9oJA4U5KzheH9fVWTvF9Y0TiNK9B6u+VXKX+jq3is7Z4ermqVMFjclu8O YzPhnftBefWS8T36A9kbSoEqNxMPTLwonTn4zkuGtw2pg4vnW3WHaPNaWodw68mNvu7X rH4g== X-Gm-Message-State: AOAM531IgYyg9dYAj6IWAUQSDvpbu6sWRClJa64QL1xTqghPX7bUxh9o NrpcacQXjcPoUPLdIe+aLrGEa22OyO5vC9baHXpKCzXl+cmHyQ== X-Google-Smtp-Source: ABdhPJy+7BmxhYXTouUshxRTxtSXHyuCLsItxVmAjBOPzWBGH3O1gT104poa6KRSnAqeQpSLjQxqrAF8Bnczw6ng+YY= X-Received: by 2002:a17:906:fcb5:: with SMTP id qw21mr5321800ejb.391.1612999207075; Wed, 10 Feb 2021 15:20:07 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <24819743.1r3eYUQgxm@farino> In-Reply-To: From: Alexey Sokolov Date: Wed, 10 Feb 2021 23:19:51 +0000 Message-ID: Subject: Re: [gentoo-dev] New project: binhost To: gentoo-dev@lists.gentoo.org Cc: binhost@gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 74bd4048-bfea-47ee-b809-e823f2ee92d9 X-Archives-Hash: 958438b740cdb43c142aed0a6e769b7f =D1=81=D1=80, 10 =D1=84=D0=B5=D0=B2=D1=80. 2021 =D0=B3. =D0=B2 19:12, Rich = Freeman : > > On Wed, Feb 10, 2021 at 12:57 PM Andreas K. H=C3=BCttel wrote: > > > > * what portage features are still needed or need improvements (e.g. bin= pkg > > signing and verification) > > * how should hosting look like > > Some ideas for portage enhancements: > > 1. Ability to fetch binary packages from some kind of repo. > 2. Ability to have multiple binary packages co-exist in a repo (local > or remote) with different build attributes (arch, USE, CFLAGS, > DEPENDS, whatever). > 3. Ability to pick the most appropriate binary packages to use based > on user preferences (with a mix of hard and soft preferences). > > One idea I've had around how #2-3 might be implemented is: > 1. Binary packages already contain data on how they were built (USE > flags, dependencies, etc). Place this in a file using a deterministic > sorting/etc order so that two builds with the same settings will have > the same results. > 2. Generate a hash of the file contents - this can go in the filename > so that the file can co-exist with other files, and be located > assuming you have a full matching set of metadata. > 3. Start dropping attributes from the file based on a list of > priorities and generate additional hashes. Create symlinked files to > the original file using these hashes (overwriting or not existing > symlinks based on policy). This allows the binary package to be found > using either an exact set of attributes or a subset of higher-priority > attributes. This is analogous to shared object symlinking. > 4. The package manager will look for a binary package first using the > user's full config, and then by dropping optional elements of the > config (so maybe it does the search without CFLAGs, then without USE > flags). Eventually it aborts based on user prefs (maybe the user only > wants an exact match, or is willing to accept alternate CFLAGs but not > USE flags, or maybe anything for the arch is selected. A related idea: if user could specify which USE-flags are mandatory to be set, which USE flags are mandatory to be not set, and which can be either way, it's easier to find the matching binary package with less constraints, where only some flags match. > 5. As always the final selected binary package still gets evaluated > like any other binary package to ensure it is usable. > > Such a system can identify whether a potentially usable file exists > using only filename, cutting down on fetching. In the interests of > avoiding useless fetches we would only carry step 3 reasonably far - > packages would have to match based on architecture and any dynamic > linking requirements. So we wouldn't generate hashes that didn't > include at least those minimums, and the package manager wouldn't > search for them. > > Obviously you could do more (if you have 5 combinations of use flags, > look for the set that matches most closely). That couldn't be done > using hashes alone in an efficient way. You could have a small > manifest file alongside the binary package that could be fetched > separately if the package manager wants to narrow things down and > fetch a few of those to narrow it down further. > > Or you could skip the hash searching and just fetch all the manifests > for a particular package/arch and just search all of those, but that > is more data to transfer just to do a query. A metadata cache of some > kind of might be another solution. Content hashes would probably > still be useful just to allow co-existence of alternate builds. > > -- > Rich >