[gentoo-user] Massive kmail breakage with mariadb-10.4.6

[gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

Hello lists,

Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I came to 
look at my emails today, kmail couldn't do anything. To start with, it was 
stuck in an endless, looping attempt to display the current email. Then it 
couldn't archive the inbox, then it couldn't import from an archive. I 
couldn't even move an email from one folder to another. Akonadictl fsck and 
vacuum seemed to run but didn't improve anything. Even reverting to an old 
copy of my home directory didn't help.

Eventually I noticed that mariadb had been upgraded, so I masked the latest 
version and reverted to the older one. Lo! and Behold! Kmail sprang back into 
life.

I've sent this to both the relevant lists; I hope that's not a heinous 
transgression.

-- 
Gentoo testing system
gcc 9.1.0, sys-kernel/gentoo-sources 5.1.12
QT 5.12.3, KDE frameworks 5.59.0, KDE plasma 5.16.1
KDE apps 19.04.2oo incl KMail 19.04.1-r1 (5.11.2), akonadi 19.04.0
x11-drivers/xf86-video-amdgpu 19.0.0
dev-libs/amdgpu-pro-opencl 18.20.684755

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]

On Saturday, 22 June 2019 10:07:46 BST Peter Humphrey wrote:
> Hello lists,
> 
> Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I came
> to look at my emails today, kmail couldn't do anything. To start with, it
> was stuck in an endless, looping attempt to display the current email. Then
> it couldn't archive the inbox, then it couldn't import from an archive. I
> couldn't even move an email from one folder to another. Akonadictl fsck and
> vacuum seemed to run but didn't improve anything. Even reverting to an old
> copy of my home directory didn't help.
> 
> Eventually I noticed that mariadb had been upgraded, so I masked the latest
> version and reverted to the older one. Lo! and Behold! Kmail sprang back
> into life.
> 
> I've sent this to both the relevant lists; I hope that's not a heinous
> transgression.

Perhaps mariadb-10.4.6 has had some major code changes which make it 
incompatible with the 10.3.16 database structures, although you wouldn't 
expect the level of breakage you're describing happening on a minor version 
update.

Usually this incantation works:

pkill kmail
akonadictl stop
akonadictl start
akonadictl fsck
akonadictl vacuum

with a reasonable waiting time between each step to make sure the command has 
finished processing the db files.  If the db itself has been corrupted badly 
(as it can happen on a ropey fs) then deleting and recreating the affected 
akonadi resource may be a better option.

Did you try removing the akonadi resource, then recreating it and waiting for 
mariadb to re-index the contents of your emails/contacts/notes/what-ever?

When things go awry on a system using btrfs and no mirror, this is my nuclear 
option to restore a working kdepim.

PS. One of these days I will try to educate myself better on recovering from 
btrfs corruption (or more likely reformat the fs to ext4 and forget about it). 
;-)

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On Sat, 22 Jun 2019 10:07:46 +0100, Peter Humphrey wrote:

> Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I
> came to look at my emails today, kmail couldn't do anything. To start
> with, it was stuck in an endless, looping attempt to display the
> current email. Then it couldn't archive the inbox, then it couldn't
> import from an archive. I couldn't even move an email from one folder
> to another. Akonadictl fsck and vacuum seemed to run but didn't improve
> anything. Even reverting to an old copy of my home directory didn't
> help.

Did you restart msyqld and run mysql_upgrade?


-- 
Neil Bothwick

Okay, I pulled the pin. Now what? Hey, where are you going?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Mick]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

On Saturday, 22 June 2019 12:33:29 BST Neil Bothwick wrote:
> On Sat, 22 Jun 2019 10:07:46 +0100, Peter Humphrey wrote:
> > Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I
> > came to look at my emails today, kmail couldn't do anything. To start
> > with, it was stuck in an endless, looping attempt to display the
> > current email. Then it couldn't archive the inbox, then it couldn't
> > import from an archive. I couldn't even move an email from one folder
> > to another. Akonadictl fsck and vacuum seemed to run but didn't improve
> > anything. Even reverting to an old copy of my home directory didn't
> > help.
> 
> Did you restart msyqld and run mysql_upgrade?

AFAIK when akonadi (re)starts it always runs mysql_upgrade.

-- 
Regards,

Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Saturday, 22 June 2019 12:33:29 BST Neil Bothwick wrote:
> On Sat, 22 Jun 2019 10:07:46 +0100, Peter Humphrey wrote:
> > Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I
> > came to look at my emails today, kmail couldn't do anything. To start
> > with, it was stuck in an endless, looping attempt to display the
> > current email. Then it couldn't archive the inbox, then it couldn't
> > import from an archive. I couldn't even move an email from one folder
> > to another. Akonadictl fsck and vacuum seemed to run but didn't improve
> > anything. Even reverting to an old copy of my home directory didn't
> > help.
> 
> Did you restart msyqld and run mysql_upgrade?

I ran lib_users -s after the update yesterday, found a lot of old libs in use 
so rebooted. I also rebooted more than once this morning.

To answer Mick, I tried akonadi fsck and vacuum; they reported success but 
didn't affect what I was seeing in kmail.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Saturday, 22 June 2019 15:36:49 BST Peter Humphrey wrote:
> On Saturday, 22 June 2019 12:33:29 BST Neil Bothwick wrote:
> > On Sat, 22 Jun 2019 10:07:46 +0100, Peter Humphrey wrote:
> > > Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I
> > > came to look at my emails today, kmail couldn't do anything. To start
> > > with, it was stuck in an endless, looping attempt to display the
> > > current email. Then it couldn't archive the inbox, then it couldn't
> > > import from an archive. I couldn't even move an email from one folder
> > > to another. Akonadictl fsck and vacuum seemed to run but didn't improve
> > > anything. Even reverting to an old copy of my home directory didn't
> > > help.
> > 
> > Did you restart msyqld and run mysql_upgrade?
> 
> I ran lib_users -s after the update yesterday, found a lot of old libs in
> use so rebooted. I also rebooted more than once this morning.
> 
> To answer Mick, I tried akonadi fsck and vacuum; they reported success but
> didn't affect what I was seeing in kmail.

I've just tried upgrading mariadb again while watching it, and got similar 
results. I did notice that an error notice came up about being unable to store 
a message received via POP3, which is my main incoming source. I can't quote 
exactly because the notice disappeared too soon.

Back to 10.3.16 for now.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Saturday, 22 June 2019 10:07:46 BST Peter Humphrey wrote:
> Hello lists,
> 
> Yesterday's routine update took mariadb from 10.3.16 to 10.4.6. When I came
> to look at my emails today, kmail couldn't do anything. To start with, it
> was stuck in an endless, looping attempt to display the current email. Then
> it couldn't archive the inbox, then it couldn't import from an archive. I
> couldn't even move an email from one folder to another. Akonadictl fsck and
> vacuum seemed to run but didn't improve anything. Even reverting to an old
> copy of my home directory didn't help.
> 
> Eventually I noticed that mariadb had been upgraded, so I masked the latest
> version and reverted to the older one. Lo! and Behold! Kmail sprang back
> into life.

Today's update included dev-db/mysql-connector-c-6.1.11-r1 to -r2, so I tried 
mariadb 10.4.6 again. No improvement.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Wols Lists]

On 09/07/19 11:38, Peter Humphrey wrote:
>> Eventually I noticed that mariadb had been upgraded, so I masked the latest
>> > version and reverted to the older one. Lo! and Behold! Kmail sprang back
>> > into life.

> Today's update included dev-db/mysql-connector-c-6.1.11-r1 to -r2, so I tried 
> mariadb 10.4.6 again. No improvement.

Do you need to update some mariadb disk format?

I gather they changed something ages ago and you had to fix the database
files. Broke my fetchmail/postfix setup and I couldn't work out how to
fix it. Fortunately I wasn't using it as a mail store. That's not been
working for a few years now.

Could be the new mariadb doesn't recognise the old database files.

Cheers,
Wol

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Tuesday, 9 July 2019 19:45:25 BST Wols Lists wrote:
> On 09/07/19 11:38, Peter Humphrey wrote:
> >> Eventually I noticed that mariadb had been upgraded, so I masked the
> >> latest
> >> 
> >> > version and reverted to the older one. Lo! and Behold! Kmail sprang
> >> > back
> >> > into life.
> > 
> > Today's update included dev-db/mysql-connector-c-6.1.11-r1 to -r2, so I
> > tried mariadb 10.4.6 again. No improvement.
> 
> Do you need to update some mariadb disk format?
> 
> I gather they changed something ages ago and you had to fix the database
> files. Broke my fetchmail/postfix setup and I couldn't work out how to
> fix it. Fortunately I wasn't using it as a mail store. That's not been
> working for a few years now.
> 
> Could be the new mariadb doesn't recognise the old database files.
> 
> Cheers,
> Wol

From what I recall when akonadictl starts its mysql database it runs an update 
by default to sort out any changes to keys priveleges and what not, but this 
version update may require a more manual intervention?  I would have thought 
there would be some e-news to this effect though.
-- 
Regards,

Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

> I've just tried upgrading mariadb again while watching it, and got similar
> results. I did notice that an error notice came up about being unable to
> store
> a message received via POP3, which is my main incoming source. I can't
> quote
> exactly because the notice disappeared too soon.
>
> Back to 10.3.16 for now.
>

Just to confirm, when you say upgrade you mean something like;
emerge -u mariadb
systemctl restart mariadb
mysql_upgrade -u root -p

Right?

[-- Attachment #2: Type: text/html, Size: 844 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Wednesday, 10 July 2019 00:06:43 BST Adam Carter wrote:
> > I've just tried upgrading mariadb again while watching it, and got similar
> > results. I did notice that an error notice came up about being unable to
> > store
> > a message received via POP3, which is my main incoming source. I can't
> > quote
> > exactly because the notice disappeared too soon.
> > 
> > Back to 10.3.16 for now.
> 
> Just to confirm, when you say upgrade you mean something like;
> emerge -u mariadb
> systemctl restart mariadb
> mysql_upgrade -u root -p

Akonadi runs an instance of mariadb for its own use, without reference to what 
else might be running on the machine.

I've never had to run mysql_upgrade before, and I can't find a way to do it 
manually because of this in .local/share/akonadi/mysql.conf:

# Do not listen for TCP/IP connections at all
skip_networking

Maybe I could comment that out temporarily, but I don't know what else might 
be affected. Otherwise it looks like creating a new user for myself and 
importing the message archive.

Another strange thing is that I sent the root message of this thread to the 
kdepim-users list as well, and referred to it later, but there's been not a 
squeak from anyone there.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Wednesday, 10 July 2019 10:06:01 BST Peter Humphrey wrote:
> On Wednesday, 10 July 2019 00:06:43 BST Adam Carter wrote:
> > > I've just tried upgrading mariadb again while watching it, and got
> > > similar
> > > results. I did notice that an error notice came up about being unable to
> > > store
> > > a message received via POP3, which is my main incoming source. I can't
> > > quote
> > > exactly because the notice disappeared too soon.
> > > 
> > > Back to 10.3.16 for now.
> > 
> > Just to confirm, when you say upgrade you mean something like;
> > emerge -u mariadb
> > systemctl restart mariadb
> > mysql_upgrade -u root -p
> 
> Akonadi runs an instance of mariadb for its own use, without reference to
> what else might be running on the machine.
> 
> I've never had to run mysql_upgrade before, and I can't find a way to do it
> manually because of this in .local/share/akonadi/mysql.conf:
> 
> # Do not listen for TCP/IP connections at all
> skip_networking
> 
> Maybe I could comment that out temporarily, but I don't know what else might
> be affected. Otherwise it looks like creating a new user for myself and
> importing the message archive.

Well, I tried that but when I started kmail to set it up again, it crashed 
after telling me it had failed to get a lock. On what, it didn't say.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Mick]

[-- Attachment #1: Type: text/plain, Size: 4334 bytes --]

On Wednesday, 10 July 2019 14:31:19 BST Peter Humphrey wrote:
> On Wednesday, 10 July 2019 10:06:01 BST Peter Humphrey wrote:
> > On Wednesday, 10 July 2019 00:06:43 BST Adam Carter wrote:
> > > > I've just tried upgrading mariadb again while watching it, and got
> > > > similar
> > > > results. I did notice that an error notice came up about being unable
> > > > to
> > > > store
> > > > a message received via POP3, which is my main incoming source. I can't
> > > > quote
> > > > exactly because the notice disappeared too soon.
> > > > 
> > > > Back to 10.3.16 for now.
> > > 
> > > Just to confirm, when you say upgrade you mean something like;
> > > emerge -u mariadb
> > > systemctl restart mariadb
> > > mysql_upgrade -u root -p
> > 
> > Akonadi runs an instance of mariadb for its own use, without reference to
> > what else might be running on the machine.
> > 
> > I've never had to run mysql_upgrade before, and I can't find a way to do
> > it
> > manually because of this in .local/share/akonadi/mysql.conf:
> > 
> > # Do not listen for TCP/IP connections at all
> > skip_networking
> > 
> > Maybe I could comment that out temporarily, but I don't know what else
> > might be affected. Otherwise it looks like creating a new user for myself
> > and importing the message archive.
> 
> Well, I tried that but when I started kmail to set it up again, it crashed
> after telling me it had failed to get a lock. On what, it didn't say.

/usr/bin/akonadi_control launches /usr/bin/akonadiserver, which lunches /usr/
sbin/mysqld:

/usr/bin/akonadi_control
|
 \_ /usr/bin/akonadiserver
    |
     \_/usr/sbin/mysqld

They're all running as the user who launches kmail, i.e. yourself.  Also, 
unless you have tweaked access to the database to allow TCP, it will only use 
a local Unix socket.  Have a look in your /tmp fs for this socket name.  If 
your kdepim user is logged in as user 'peter', I'm guessing you'll see 
something like this, as long as akonadiserver is running:

/tmp/akonadi-peter.XXXXXX/mysql.socket

You could try running mysql_upgrade on this, but the command will request 
access to default mysql database tables and its socket under /var/run/mysqld/, 
which I assume you won't be running unnecessarily just for a Plasma/KDE 
desktop.  Consequently the incantation will fail.

Instead, you could try running the individual commands mysql_upgrade runs 
yourself, only on the akonadi tables.  Here's my attempt:

$ /usr/bin/mysqlcheck -u michael --all-databases --check-upgrade --auto-repair 
--socket=/tmp/akonadi-michael.ZtUWTD/mysql.socket
akonadi.collectionattributetable                   OK
akonadi.collectionmimetyperelation                 OK
akonadi.collectionpimitemrelation                  OK
akonadi.collectiontable                            OK
akonadi.flagtable                                  OK
akonadi.mimetypetable                              OK
akonadi.parttable                                  OK
akonadi.parttypetable                              OK
akonadi.pimitemflagrelation                        OK
akonadi.pimitemtable                               OK
akonadi.pimitemtagrelation                         OK
akonadi.relationtable                              OK
akonadi.relationtypetable                          OK
akonadi.resourcetable                              OK
akonadi.schemaversiontable                         OK
akonadi.tagattributetable                          OK
akonadi.tagremoteidresourcerelationtable           OK
akonadi.tagtable                                   OK
akonadi.tagtypetable                               OK

Or, you could connect to the above socket with /usr/bin/mysql and run SQL 
commands from within, after you select each akonadi database/table in turn.

Normally, I don't think any of the above is required.  From what I recall 
akonadiserver runs mysql_upgrade each and every time akonadiserver is 
launched.  However, if akonadi can't run the kdepim mysql following a database 
version update, then you'll need to look deeper into it.

If akonadiserver does not start/crashes, it may be more effective to look at 
the mysql.err logs under  ~/.local/share/akonadi/db_data/.

You could also launch akonadiconsole, switch on debugging and see if it offers 
anything more informative when you try to restart akonadi.

HTH.

-- 
Regards,

Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Wednesday, 10 July 2019 18:40:04 BST Mick wrote:
> On Wednesday, 10 July 2019 14:31:19 BST Peter Humphrey wrote:
> > On Wednesday, 10 July 2019 10:06:01 BST Peter Humphrey wrote:
> > > On Wednesday, 10 July 2019 00:06:43 BST Adam Carter wrote:
> > > > > I've just tried upgrading mariadb again while watching it, and got
> > > > > similar
> > > > > results. I did notice that an error notice came up about being
> > > > > unable
> > > > > to
> > > > > store
> > > > > a message received via POP3, which is my main incoming source. I
> > > > > can't
> > > > > quote
> > > > > exactly because the notice disappeared too soon.
> > > > > 
> > > > > Back to 10.3.16 for now.
> > > > 
> > > > Just to confirm, when you say upgrade you mean something like;
> > > > emerge -u mariadb
> > > > systemctl restart mariadb
> > > > mysql_upgrade -u root -p
> > > 
> > > Akonadi runs an instance of mariadb for its own use, without reference
> > > to
> > > what else might be running on the machine.
> > > 
> > > I've never had to run mysql_upgrade before, and I can't find a way to do
> > > it
> > > manually because of this in .local/share/akonadi/mysql.conf:
> > > 
> > > # Do not listen for TCP/IP connections at all
> > > skip_networking
> > > 
> > > Maybe I could comment that out temporarily, but I don't know what else
> > > might be affected. Otherwise it looks like creating a new user for
> > > myself
> > > and importing the message archive.
> > 
> > Well, I tried that but when I started kmail to set it up again, it crashed
> > after telling me it had failed to get a lock. On what, it didn't say.
> 
> /usr/bin/akonadi_control launches /usr/bin/akonadiserver, which lunches
> /usr/ sbin/mysqld:
> 
> /usr/bin/akonadi_control
> 
>  \_ /usr/bin/akonadiserver
> 
>      \_/usr/sbin/mysqld
> 
> They're all running as the user who launches kmail, i.e. yourself.  Also,
> unless you have tweaked access to the database to allow TCP, it will only
> use a local Unix socket.  Have a look in your /tmp fs for this socket name.
>  If your kdepim user is logged in as user 'peter', I'm guessing you'll see
> something like this, as long as akonadiserver is running:
> 
> /tmp/akonadi-peter.XXXXXX/mysql.socket
> 
> You could try running mysql_upgrade on this, but the command will request
> access to default mysql database tables and its socket under
> /var/run/mysqld/, which I assume you won't be running unnecessarily just
> for a Plasma/KDE desktop.  Consequently the incantation will fail.
> 
> Instead, you could try running the individual commands mysql_upgrade runs
> yourself, only on the akonadi tables.  Here's my attempt:
> 
> $ /usr/bin/mysqlcheck -u michael --all-databases --check-upgrade
> --auto-repair --socket=/tmp/akonadi-michael.ZtUWTD/mysql.socket
> akonadi.collectionattributetable                   OK
> akonadi.collectionmimetyperelation                 OK
> akonadi.collectionpimitemrelation                  OK
> akonadi.collectiontable                            OK
> akonadi.flagtable                                  OK
> akonadi.mimetypetable                              OK
> akonadi.parttable                                  OK
> akonadi.parttypetable                              OK
> akonadi.pimitemflagrelation                        OK
> akonadi.pimitemtable                               OK
> akonadi.pimitemtagrelation                         OK
> akonadi.relationtable                              OK
> akonadi.relationtypetable                          OK
> akonadi.resourcetable                              OK
> akonadi.schemaversiontable                         OK
> akonadi.tagattributetable                          OK
> akonadi.tagremoteidresourcerelationtable           OK
> akonadi.tagtable                                   OK
> akonadi.tagtypetable                               OK

$ /usr/bin/mysqlcheck -u prh --all-databases --check-upgrade --auto-repair --
socket=/tmp/akonadi-prh.1l0Gu6/mysql.socket
akonadi.collectionattributetable                   OK
akonadi.collectionmimetyperelation                 OK
akonadi.collectionpimitemrelation                  OK
akonadi.collectiontable                            OK
akonadi.flagtable                                  OK
akonadi.mimetypetable                              OK
akonadi.parttable                                  OK
akonadi.parttypetable                              OK
akonadi.pimitemflagrelation                        OK
akonadi.pimitemtable                               OK
akonadi.pimitemtagrelation                         OK
akonadi.relationtable                              OK
akonadi.relationtypetable                          OK
akonadi.resourcetable                              OK
akonadi.schemaversiontable                         OK
akonadi.tagattributetable                          OK
akonadi.tagremoteidresourcerelationtable           OK
akonadi.tagtable                                   OK
akonadi.tagtypetable                               OK

That's on the older version of mariadb. I'll try it again with my new user and 
the new mariadb.

> Or, you could connect to the above socket with /usr/bin/mysql and run SQL
> commands from within, after you select each akonadi database/table in turn.
> 
> Normally, I don't think any of the above is required.  From what I recall
> akonadiserver runs mysql_upgrade each and every time akonadiserver is
> launched.  However, if akonadi can't run the kdepim mysql following a
> database version update, then you'll need to look deeper into it.
> 
> If akonadiserver does not start/crashes, it may be more effective to look at
> the mysql.err logs under  ~/.local/share/akonadi/db_data/.
> 
> You could also launch akonadiconsole, switch on debugging and see if it
> offers anything more informative when you try to restart akonadi.
> 
> HTH.

Certainly does! Many thanks Mick.

-- 
Regards,
Peter.

Re: [gentoo-user] Massive kmail breakage with mariadb-10.4.6

[Peter Humphrey]

On Wednesday, 10 July 2019 18:40:04 BST Mick wrote:

> Normally, I don't think any of the above is required.  From what I recall
> akonadiserver runs mysql_upgrade each and every time akonadiserver is
> launched.  However, if akonadi can't run the kdepim mysql following a
> database version update, then you'll need to look deeper into it.

It turns out to be a bug in mariadb-10.4.6: 
https://bugs.kde.org/show_bug.cgi?id=409224 and
https://bugs.gentoo.org/688746

Meanwhile, thanks to Mick, I've learned a few things about the akonadi 
database.

-- 
Regards,
Peter.