From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 63B67158095 for ; Sun, 28 Aug 2022 21:13:13 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A1571E0968; Sun, 28 Aug 2022 21:13:08 +0000 (UTC) Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 5D9B5E095A for ; Sun, 28 Aug 2022 21:13:08 +0000 (UTC) Received: by mail-pg1-f171.google.com with SMTP id q63so6028147pga.9 for ; Sun, 28 Aug 2022 14:13:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc; bh=Te6OeRQFslfrYBkTHlOPU4vA2h3kHu+w3subaPJSPMM=; b=5PnldXXDBJLC2Cuywxkh0zt666jKtAe4olGwr+RdVRXsw2awnK8oRtLI0LVLGurK+P yeTGGeHQrzc9Uwih5chskfHzqkJTXwLaTIsSQ0ci3QHjj2iK9yawUtxkpsHVMNy+ovHo 2YtU+Z0nVTc/9YOiZMZ5rkKRw/bbakt/z5hvmwtXucbI6O4h/eFGY8vmRboXafXLPaM2 2g3vuC9nx5VJRoHSZgQjX3bwoj6OCjzbMf/FNieEtD8gSdWbhNxU0n+PSJ+FAPmLNZGT gtn3CvmhSI0uUPjSvAeRLfZBobUW7CQLnZcTD1pfNaBFkVftjtFL8mmc/4viTwBKB/bN gFOA== X-Gm-Message-State: ACgBeo3snDEjhPYf2nINBuDbEv8RpNA9rypU8i6f7i6FgXZxCjHqN8K4 UoRbuasrUnNaSJSOYILuR/2uhb63VhXOUzNmYrwnYkCUz8Q= X-Google-Smtp-Source: AA6agR4h0uFjPqRdhtLxBl7Rp1qDOrr7f54sqvOdjYr5iEi4mANVyBWBLbCh5hWqh+11VMC0vFoecPpGwJHNpPM22N0= X-Received: by 2002:a63:5916:0:b0:41d:2c8c:7492 with SMTP id n22-20020a635916000000b0041d2c8c7492mr11253394pgb.81.1661721186758; Sun, 28 Aug 2022 14:13:06 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <464b2b22-6c7c-6dd1-c0b0-b4905222943f@gmail.com> <2895bfbc-5795-5729-69c5-1fd15874d35d@youngman.org.uk> In-Reply-To: <2895bfbc-5795-5729-69c5-1fd15874d35d@youngman.org.uk> From: Rich Freeman Date: Sun, 28 Aug 2022 17:12:57 -0400 Message-ID: Subject: Re: [gentoo-user] Limiting amount of memory a program can use. To: gentoo-user@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 34661dba-277a-43f8-82b3-75865fb42928 X-Archives-Hash: a12da99b9e5b7232592801d42371f328 On Sun, Aug 28, 2022 at 11:32 AM Wols Lists wrote: > > On 28/08/2022 15:21, Rich Freeman wrote: > > Something I wish linux supported was discardable memory, for > > caches/etc. A program should be able to allocate memory while passing > > a hint to the kernel saying that the memory is discardable. > > Linux DOES have that ... > > I'm not sure how to use it, but you can pass a flag to open(), which > says "do not cache this file". That isn't what I was proposing. That has been around for a while. However, it only impacts caching/buffering of open files, which is kernel memory, and not memory held by a process itself. You wouldn't want to limit kernel file caching of a torrent file, since those files are pretty ideal candidates for caching (since the client is likely to send data not long after receiving it). There isn't really any benefit to this either in this case as the kernel already automatically drops file cache memory as needed under memory pressure. I was referring to application caching, which is internal storage used by an application that may or may not have anything to do with open files. A browser cache is a good example of something like this (though that extends to disk caching often and those files would also get cached by the kernel). The kernel can't simply drop this memory when it needs memory since there is no way for it to tell which memory an application owns is used for this purpose, and no way currently to handle the resulting segfaults when the application goes to read that memory. Reading memory is a CPU instruction, not a system call, so it would trigger an exception at the CPU level which must be handled. I'm sure something could be engineered but it is more complex and I'm guessing nobody has seen the need for it yet. Ideally you'd also have ways for the OS to hint how much memory applications ought to consume in this way. After all, you might have two applications running at the same time, which both want to use lots of extra RAM for caching if it is available, but neither has any way of knowing that this situation exists and it is undesirable if all the memory goes to the first application that asks, and also undesirable if after everything is done grabbing memory if there is a bunch of it sitting around doing nothing. TL;DR: cache is not necessarily the same as kernel file cache. -- Rich