[gentoo-dev] mirror storage growth rate

[gentoo-dev] mirror storage growth rate

[Jaco Kroon]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Hi All,

I was messing with some storage related caching on some of our hosts 
this morning when I wondered about how much storage the gentoo mirrors 
were consuming.  I'm not too worried about the current storage, but I am 
noticing that the storage requirements are creeping quite a bit (as per 
attached), and if that growth rate continues it may become a problem 
*eventually*.

Can this growth be explained?

Is it expected to continue at this rate?

Kind regards,
Jaco

[-- Attachment #2: gentoo_mirror_storage.png --]
[-- Type: image/png, Size: 3244 bytes --]

Re: [gentoo-dev] mirror storage growth rate

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On Fri, 2024-03-15 at 10:06 +0200, Jaco Kroon wrote:
> Hi All,
> 
> I was messing with some storage related caching on some of our hosts 
> this morning when I wondered about how much storage the gentoo mirrors 
> were consuming.  I'm not too worried about the current storage, but I am 
> noticing that the storage requirements are creeping quite a bit (as per 
> attached), and if that growth rate continues it may become a problem 
> *eventually*.
> 
> Can this growth be explained?
> 

I guess the simplest explanation is that software is growing larger,
and in the end we should be expecting to adding new packages faster than
removing dead ones.  Add to that the grotesque inefficiency of modern
programming languages such as Go and Rust.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] mirror storage growth rate

[Hoël Bézier]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

>I guess the simplest explanation is that software is growing larger,
>and in the end we should be expecting to adding new packages faster than
>removing dead ones.  Add to that the grotesque inefficiency of modern
>programming languages such as Go and Rust.

Wouldn’t initiatives like rust-dev[0] help with that? I know that Debian is 
also packaging Rust this way[1].

[0]: https://wiki.gentoo.org/wiki/Project:Rust/rust-dev
[1]: https://wiki.debian.org/Rust

Hoël

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] mirror storage growth rate

[Maciej Barć]

[-- Attachment #1.1.1: Type: text/plain, Size: 1506 bytes --]

> Wouldn’t initiatives like rust-dev[0] help with that? I know that Debian 
> is also packaging Rust this way[1].

I think this was tried long time ago in rust-overlay and failed at the 
end because the dependency graph was incredibly big. In fact you can see 
it on the wiki, this is larger than _the bigger_ Haskell packages.

> I guess the simplest explanation is that software is growing larger,

This is not only the case of Rust, but Go, JAVA and .NET and maybe some 
other projects. Self-bootstrap anyone? :)

> Can this growth be explained?
> Is it expected to continue at this rate? 

Graph is just showing the overall growth, if we associate distfiles to 
packages we will get the answers.

W dniu 15.03.2024 o 16:40, Hoël Bézier pisze:
>> I guess the simplest explanation is that software is growing larger,
>> and in the end we should be expecting to adding new packages faster than
>> removing dead ones.  Add to that the grotesque inefficiency of modern
>> programming languages such as Go and Rust.
> 
> Wouldn’t initiatives like rust-dev[0] help with that? I know that Debian 
> is also packaging Rust this way[1].
> 
> [0]: https://wiki.gentoo.org/wiki/Project:Rust/rust-dev
> [1]: https://wiki.debian.org/Rust
> 
> Hoël

-- 
Have a great day!

~ Maciej XGQT Barć

xgqt@gentoo.org
Gentoo Linux developer
(dotnet, emacs, math, ml, nim, scheme, sci)
https://wiki.gentoo.org/wiki/User:Xgqt
9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 16315 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] mirror storage growth rate

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

Hi Jaco, 

* we have more stages
* the binary packages have to go somewhere
* and, temporarily, things are duplicated due to the 17.x / 23.0 profile transition

The third point will eventually go away. However, I'm not sure how much it actually
contributes.

https://www.akhuettel.de/~huettel/plots/mirrors.php

If you look at the plots, the distfiles part is surprisingly large.
Binary packages (17.x and 23.0) and 17.x stages are under "releases".
The 23.0 stages for testing are under "experimental".

Lastly, I'm still working on an automated cleanup for outdated "small arches" binary
packages (i.e. not arm64 and amd64, these are cleaned automatically already).
This just wasn't a priority so far.

Hope this helps.
-a

Am Freitag, 15. März 2024, 09:06:36 CET schrieb Jaco Kroon:
> Hi All,
> 
> I was messing with some storage related caching on some of our hosts 
> this morning when I wondered about how much storage the gentoo mirrors 
> were consuming.  I'm not too worried about the current storage, but I am 
> noticing that the storage requirements are creeping quite a bit (as per 
> attached), and if that growth rate continues it may become a problem 
> *eventually*.
> 
> Can this growth be explained?
> 
> Is it expected to continue at this rate?
> 
> Kind regards,
> Jaco
> 


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]