[gentoo-user] Portage metadata cache backend: sqlite or not?

[gentoo-user] Portage metadata cache backend: sqlite or not?

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

Hi list

For the longest time I've had portage configured to use the sqlite metadata
cache backend as per an old HOWTO [0], however, I thought that it would be a
good idea to revisit that decision.

Now apparently, this was supposed to speed up portage, although even that
depends.  For instance, [0] says that the metadata_overlay backend is faster on
fast storage; since all of portage is on an SSD, that ought to be the case for
me. However, [0] is pretty outdated, so I don't really know, and don't have any
comparison.

In addition to that, I don't explicitly make use of the sqlite metadata cache,
that is, I don't (consciously) use any software that accesses those DBs, except
for eix (except for overlays, where one would need to run "emerge --regen"
first, which is *ssssslllllloooowwwww*), which can make use of them if
CACHE_METHOD is set appropriately; this speeds up eix-update considerably.

Does anybody here have experience with this, or a recommendation?  I tried
switching to the default cache method temporarily to see how things perform,
and "emerge @world -uDNva" slowed down by about 30 seconds, so preliminary
results point to sticking with sqlite (although it could at least partly be a
btrfs performance regression in Linux 3.15, since there have been several reports of
those, and several fixes slated for 3.16). Anyway, I'm also unsure of unintended
consequences, or other settings I might have to change, too.

Also, does anybody have any performance data and/or experience on using btrfs
with compression in this context?

[0] http://www.gentoo-wiki.info/TIP_speed_up_portage_with_sqlite

Greetings,
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-user] Re: Portage metadata cache backend: sqlite or not?

[James]

Marc Joliet <marcec <at> gmx.de> writes:

> 
> Hi list
> 
> For the longest time I've had portage configured to use the sqlite 
> metadata cache backend as per an old HOWTO [0], however, I thought that 
> it would be a > good idea to revisit that decision.

> Does anybody here have experience with this, or a recommendation?  

I cannot directly speak to your issues. But, a while back when I 
was researching some dB tools, I discovered a add-on for browsers that 
allows you to view sqllite db:

 SQLite Manager 0.8.1 

and this interesting tool:

http://sqlitestudio.pl/

I hope this helps you diagnose your problems.


James

Re: [gentoo-user] Portage metadata cache backend: sqlite or not?

[Nilesh Govindrajan]

On Tue, Aug 12, 2014 at 11:40 PM, Marc Joliet <marcec@gmx.de> wrote:
> Hi list
>
> For the longest time I've had portage configured to use the sqlite metadata
> cache backend as per an old HOWTO [0], however, I thought that it would be a
> good idea to revisit that decision.
>
> Now apparently, this was supposed to speed up portage, although even that
> depends.  For instance, [0] says that the metadata_overlay backend is faster on
> fast storage; since all of portage is on an SSD, that ought to be the case for
> me. However, [0] is pretty outdated, so I don't really know, and don't have any
> comparison.
>
> In addition to that, I don't explicitly make use of the sqlite metadata cache,
> that is, I don't (consciously) use any software that accesses those DBs, except
> for eix (except for overlays, where one would need to run "emerge --regen"
> first, which is *ssssslllllloooowwwww*), which can make use of them if
> CACHE_METHOD is set appropriately; this speeds up eix-update considerably.
>
> Does anybody here have experience with this, or a recommendation?  I tried
> switching to the default cache method temporarily to see how things perform,
> and "emerge @world -uDNva" slowed down by about 30 seconds, so preliminary
> results point to sticking with sqlite (although it could at least partly be a
> btrfs performance regression in Linux 3.15, since there have been several reports of
> those, and several fixes slated for 3.16). Anyway, I'm also unsure of unintended
> consequences, or other settings I might have to change, too.
>
> Also, does anybody have any performance data and/or experience on using btrfs
> with compression in this context?
>
> [0] http://www.gentoo-wiki.info/TIP_speed_up_portage_with_sqlite
>
> Greetings,
> --
> Marc Joliet
> --
> "People who think they know everything really annoy those of us who know we
> don't" - Bjarne Stroustrup

Having tried this feature, I'd advise against it. It takes long time
to generate metadata after sync and not really that advantageous. Also
eix has it's own issues in this mode.

Re: [gentoo-user] Portage metadata cache backend: sqlite or not?

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Oh wow, I completely forgot about this open thread.  In my defense, I had to
move out two (or was it three?) days after I started the thread, and didn't have
internet again until over a month later.

Am Wed, 13 Aug 2014 07:30:01 +0530
schrieb Nilesh Govindrajan <me@nileshgr.com>:

[...]
> Having tried this feature, I'd advise against it. It takes long time
> to generate metadata after sync and not really that advantageous. Also
> eix has it's own issues in this mode.

Do I understand correctly: you recommend switching away from the sqlite
backend?  Also, can you please elaborate on the issues with eix?  I haven't
noticed anything so far, but some bugs are easy to miss.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[DONE] Re: [gentoo-user] Portage metadata cache backend: sqlite or not?

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

Am Mon, 4 May 2015 21:33:53 +0200
schrieb Marc Joliet <marcec@gmx.de>:

> Oh wow, I completely forgot about this open thread.  In my defense, I had to
> move out two (or was it three?) days after I started the thread, and didn't have
> internet again until over a month later.
> 
> Am Wed, 13 Aug 2014 07:30:01 +0530
> schrieb Nilesh Govindrajan <me@nileshgr.com>:
> 
> [...]
> > Having tried this feature, I'd advise against it. It takes long time
> > to generate metadata after sync and not really that advantageous. Also
> > eix has it's own issues in this mode.
> 
> Do I understand correctly: you recommend switching away from the sqlite
> backend?  Also, can you please elaborate on the issues with eix?  I haven't
> noticed anything so far, but some bugs are easy to miss.

Well, there's no need to leave this thread dangling, so I went ahead and moved
away from sqlite, and have not noticed any regressions (or improvements, other
than faster syncing, because sqlite needs the metadata-transfer feature).

So I consider this thread done.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]