[gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[Franz Fellner]

Hey all,

I have issues with some prgrams eating too much memory. This seems to be related to glibc not trimming as necessary which results in way too much memory still occupied by the program after free()ing memory.
I can't use gcc (specifically g++) with quite some apps now because it starts collecting memory (+swap) until everything falls apart, and I finally came to the conclusion also gcc might suffer from bad trimming behaviour.
As glibc is the package that implements free I want to have a closer look at it. The first idea is to get rid of Gentoo patches which are controlled by USE="vanilla". Playing around with glibc might destroy my system. Downgrades are already unsupported. So my question:

Can I safely switch from -vanilla to +vanilla in glibc?

This is how glibc currently is installed:
sys-libs/glibc-2.23-r2(multilib rpc -audit -caps -debug -gd -hardened -nscd -profile -selinux -suid -systemtap -vanilla CROSSCOMPILE_OPTS="-headers-only")

Thx for your help
Franz

[gentoo-user] Re: How safe is it to change vanilla-USE-Flag in glibc?

[James]

Franz Fellner <alpine.art.de <at> gmail.com> writes:


> I have issues with some prgrams eating too much memory. This seems to be
related to glibc not trimming as
> necessary which results in way too much memory still occupied by the
program after free()ing memory.

Perhaps you should file a bug, providing some evidence if other distros are
affected (this suggests it might be a gcc issue) or other distros are not
affected (this suggests it might be a gentoo pathch issue)?

> Franz


hth,
James

Re: [gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[Michael Orlitzky]

On 07/06/2016 03:17 AM, Franz Fellner wrote:
> Hey all,
> 
> I have issues with some prgrams eating too much memory. This seems to be related to glibc not trimming as necessary which results in way too much memory still occupied by the program after free()ing memory.
> I can't use gcc (specifically g++) with quite some apps now because it starts collecting memory (+swap) until everything falls apart, and I finally came to the conclusion also gcc might suffer from bad trimming behaviour.
> As glibc is the package that implements free I want to have a closer look at it. The first idea is to get rid of Gentoo patches which are controlled by USE="vanilla". Playing around with glibc might destroy my system. Downgrades are already unsupported. So my question:
> 
> Can I safely switch from -vanilla to +vanilla in glibc?
> 

It looks to me like USE=vanilla controls only whether or not bundled
timezone data is used. If that's the case (double-check!), it's probably
safe to unmerge timezone-data and re-emerge glibc with USE=vanilla.

To be safe, you can bundle up your existing glibc with quickpkg. Then if
something goes wrong, you can always boot to a liveCD and unpack the old
version.

Re: [gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[Franz Fellner]

On Wed, 06 Jul 2016 11:19:44 -0400, Michael Orlitzky <mjo@gentoo.org> wrote:
> On 07/06/2016 03:17 AM, Franz Fellner wrote:
> > Hey all,
> > 
> > I have issues with some prgrams eating too much memory. This seems to be related to glibc not trimming as necessary which results in way too much memory still occupied by the program after free()ing memory.
> > I can't use gcc (specifically g++) with quite some apps now because it starts collecting memory (+swap) until everything falls apart, and I finally came to the conclusion also gcc might suffer from bad trimming behaviour.
> > As glibc is the package that implements free I want to have a closer look at it. The first idea is to get rid of Gentoo patches which are controlled by USE="vanilla". Playing around with glibc might destroy my system. Downgrades are already unsupported. So my question:
> > 
> > Can I safely switch from -vanilla to +vanilla in glibc?
> > 
> 
> It looks to me like USE=vanilla controls only whether or not bundled
> timezone data is used. If that's the case (double-check!), it's probably
> safe to unmerge timezone-data and re-emerge glibc with USE=vanilla.

There is a huge glibc-2.23-patches-4.tar.bz2 in my DISTDIR and the patches get applied.

> To be safe, you can bundle up your existing glibc with quickpkg. Then if
> something goes wrong, you can always boot to a liveCD and unpack the old
> version.

Yes, I know that works ;) But I don't have any livecd around (and most likely not even an empty disc, at least it's not therewhere it should be) so it is extra work I could avoid if a DEV gives me the OK regarding USE=vanilla.

Thx
Franz

Re: [gentoo-user] Re: How safe is it to change vanilla-USE-Flag in glibc?

[Franz Fellner]

On Wed, 06 Jul 2016 15:06:20 +0000, James <wireless@tampabay.rr.com> wrote:
> Franz Fellner <alpine.art.de <at> gmail.com> writes:
> 
> 
> > I have issues with some prgrams eating too much memory. This seems to be
> > related to glibc not trimming as
> > necessary which results in way too much memory still occupied by the
> > program after free()ing memory.
> 
> Perhaps you should file a bug, providing some evidence if other distros are
> affected (this suggests it might be a gcc issue) or other distros are not
> affected (this suggests it might be a gentoo pathch issue)?

I mentioned my issues several times with the answer "can't be, the issue is on your side".
When I filed a bug against tmux regarding its memory consumption I got told "glibc is known to do bad things" and I was given this patch for tmux which instantenously solved my issue:

diff --git a/grid.c b/grid.c
index ef7c374..96385f4 100644
--- a/grid.c
+++ b/grid.c
@@ -113,6 +113,8 @@ grid_destroy(struct grid *gd)
 	free(gd->linedata);
 
 	free(gd);
+
+	malloc_trim(0);
 }
 
 /* Compare grids. */
@@ -326,6 +328,8 @@ grid_clear_lines(struct grid *gd, u_int py, u_int ny)
 		free(gl->celldata);
 		memset(gl, 0, sizeof *gl);
 	}
+
+	malloc_trim(0);
 }
 
 /* Move a group of lines. */

I have some other applications that are unusable to a certain degree. Having to patch every single application isn't possible, I want to get to the root of the issue :) And USE=vanilla is my first attempt.

Thx
Franz

Re: [gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[Michael Orlitzky]

On 07/06/2016 01:02 PM, Franz Fellner wrote:
>>
>> It looks to me like USE=vanilla controls only whether or not bundled
>> timezone data is used. If that's the case (double-check!), it's probably
>> safe to unmerge timezone-data and re-emerge glibc with USE=vanilla.
> 
> There is a huge glibc-2.23-patches-4.tar.bz2 in my DISTDIR and the patches get applied.
> 

Yeah, but they get applied even with USE=vanilla. You can check by
beginning the emerge, and then hitting Ctrl-C while it's compiling.
Scroll up and you should see a bunch of "Applying..." lines.

Re: [gentoo-user] How safe is it to change vanilla-USE-Flag in glibc?

[Michael Orlitzky]

On 07/06/2016 01:11 PM, Michael Orlitzky wrote:
>>
>> There is a huge glibc-2.23-patches-4.tar.bz2 in my DISTDIR and the patches get applied.
>>
> 
> Yeah, but they get applied even with USE=vanilla. You can check by
> beginning the emerge, and then hitting Ctrl-C while it's compiling.
> Scroll up and you should see a bunch of "Applying..." lines.
> 

Nevermind, you get different sets of patches with USE=vanilla. The glibc
ebuild is using that illegal eblit garbage. You'll have to get an
opinion from someone who knows what those patches do.

Re: [gentoo-user] Re: How safe is it to change vanilla-USE-Flag in glibc?

[Fernando Rodriguez]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/06/2016 01:10 PM, Franz Fellner wrote:
> On Wed, 06 Jul 2016 15:06:20 +0000, James <wireless@tampabay.rr.com> wrote:
>> Franz Fellner <alpine.art.de <at> gmail.com> writes:
>>
>>
>>> I have issues with some prgrams eating too much memory. This seems to be
>>> related to glibc not trimming as
>>> necessary which results in way too much memory still occupied by the
>>> program after free()ing memory.
>>
>> Perhaps you should file a bug, providing some evidence if other distros are
>> affected (this suggests it might be a gcc issue) or other distros are not
>> affected (this suggests it might be a gentoo pathch issue)?
> 
> I mentioned my issues several times with the answer "can't be, the issue is on your side".
> When I filed a bug against tmux regarding its memory consumption I got told "glibc is known to do bad things" and I was given this patch for tmux which instantenously solved my issue:
> 
> diff --git a/grid.c b/grid.c
> index ef7c374..96385f4 100644
> --- a/grid.c
> +++ b/grid.c
> @@ -113,6 +113,8 @@ grid_destroy(struct grid *gd)
>  	free(gd->linedata);
>  
>  	
> +
> +	malloc_trim(0);
>  }
>  
>  /* Compare grids. */
> @@ -326,6 +328,8 @@ grid_clear_lines(struct grid *gd, u_int py, u_int ny)
>  		free(gl->celldata);
>  		memset(gl, 0, sizeof *gl);
>  	}
> +
> +	malloc_trim(0);
>  }
>  
>  /* Move a group of lines. */
> 
> I have some other applications that are unusable to a certain degree. Having to patch every single application isn't possible, I want to get to the root of the issue :) And USE=vanilla is my first attempt.
> 
> Thx
> Franz
> 

Check out the mallopt(3) man page. You can fine-tune glibc heap through environment variables without patches. Specifically use MALLOC_TRIM_THRESHOLD_ to tell glibc when to trim and MALLOC_MMAP_THRESHOLD_ to tell it to use mmap() to allocate large chunks (so they're returned to the kernel when free()d).

Just setting the trim treshold too low or calling malloc_trim(0) may be a bad idea. In some cases it will make your process numbers look better but hurt performance because malloc() will have to request memory from the kernel more often and on a system with swap it will just be swapped out if needed so it shouldn't be a problem unless it's a lot. And if it is a lot I think you should be looking somewhere else for the source of the problem.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJXfVIcAAoJEPbOFX/5UlwcwT4P/RDvLd6zsvNulN7AYCvPYFr/
2+BZmSSDeKJh5kbdGUzWb8FnqucAeFIBHYdpviEG7umDpZT6RN8MiFeDnK8W8Ww9
jLQyPgSXxTlfFNMGyxnJ/+EphSxPQ0LNjk98etHOKZNx6lM9NSN4kESRpk0LzSEs
Qqp5WWipZU0kMqpfXZDr9U92MFLuSSe/LMRxuovIkz/tqlDHA1HTScob5CCAGmBx
G4m1vxM7dPyv9YDAOELLR2y8CuoJzbiwhxOBAUBFJk3hF9Z1wDW0/8DbcogSwZkd
T4mEQB28zEawjs0aguBXSKYemN3Nik1d1sPokZQ7SKCgVpIT0bHDV/cuoccTsWr0
eliobWgCzipXa15hvccU+xqSD25tZYX3f47T7/IMiwZW4CMFZZtY4X9BX/KTn7/C
Q29M3ZM4NkbmSU27FtpKUQNl4/KrvMypmNBGcqb1BH74Y9Pay0Y9wHb+qeu8eITQ
A07wgCJTtCm0s3dtyU7tu5SU6R2CSbJurJ9pvAHC1qNhQfCQYy3yxOEh0CCrCXC1
5nH94W1W1wjthJzYf728lvFloEpI7uT0RngpTJIR5Fg6Ku5ZA14rv8qANEbLEnTe
hFV32a0cSbp3jMnUyvdYyxx89aOhJ08AU2Wf1L5DF5bMG864E5oH3AZc5a3OeNNu
jJ+zTzNMF2lZ2ePZYcRg
=ALKt
-----END PGP SIGNATURE-----