[gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

Bug: https://bugs.gentoo.org/825234
Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
---
 ...adb-database-restore-maybe-required.en.txt | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt

diff --git a/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
new file mode 100644
index 0000000..c4a698e
--- /dev/null
+++ b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
@@ -0,0 +1,46 @@
+Title: Full MariaDB database restore maybe required
+Author: Thomas Deutschmann <whissi@gentoo.org>
+Posted: 2021-11-23
+Revision: 1
+News-Item-Format: 2.0
+Display-If-Installed: dev-db/mariadb
+Display-If-Installed: sys-cluster/galera
+
+On 2021-11-21, a member of the QA project accidentially de-keyworded
+MariaDB 10.6 to address a file collision, users, who also had latest
+dev-db/mariadb-connector-c installed, experienced (NOTE: The default
+MySQL connector in Gentoo Linux is provided by
+dev-db/mysql-connector-c) [Link 1].
+
+However, downgrades are not supported in MySQL/MariaDB [Link 2].
+
+In case you already fully upgraded to MariaDB 10.6 (which includes
+executing mysql_upgrade command) and forcefully downgraded your
+MariaDB instance afterwards during the time window when keywords were
+removed, you maybe experiencing different problems:
+
+At best, your forcefully downgraded MariaDB instance prevented startup
+so all you have to do is upgrade to MariaDB 10.6 again to resume
+services.
+
+In case previous MariaDB version was able to start, you are encouraged
+to do a full backup as soon as possible using mysqldump command and
+manually restore each database ("logical downgrade") to prevent any
+data corruption.
+
+Depending on used feature set and from which version you upgraded,
+it is maybe required to do a full restore from a previous backup before
+MariaDB 10.6 upgrade to restore services and prevent any data loss or
+future runtime errors.
+
+In case you are using MariaDB in a cluster and/or Galera setup you
+probably have to rebuild the entire cluster in case the upgrade to
+MariaDB 10.6 was already replicated (using pt-table-checksum from
+dev-db/percona-toolkit can help you to validate your cluster).
+
+Keep in mind that due to the forced downgraded, point-in-time recovery
+may not be available to the extent that you are used to.
+
+Link 1: https://bugs.gentoo.org/825234#c8
+
+Link 2: https://mariadb.com/kb/en/downgrading-between-major-versions-of-mariadb/
-- 
2.34.0

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Mike Gilbert]

On Wed, Nov 24, 2021 at 10:21 PM Thomas Deutschmann <whissi@gentoo.org> wrote:
> +On 2021-11-21, a member of the QA project accidentially de-keyworded
> +MariaDB 10.6 to address a file collision, users, who also had latest
> +dev-db/mariadb-connector-c installed, experienced (NOTE: The default
> +MySQL connector in Gentoo Linux is provided by
> +dev-db/mysql-connector-c) [Link 1].

This sentence is very difficult to read. Also, I don't think it is
relevant to call out the mistake by the QA team in a news item
intended for end users. I would rewrite this as:

On 2021-11-21, keywords for dev-db/mariadb-10.6 were removed to
address a file collision with dev-db/mariadb-connector-c. This
unintentionally triggered a version downgrade for users who had
successfully upgraded to dev-db/mariadb-10.6 already.

> +However, downgrades are not supported in MySQL/MariaDB [Link 2].
> +
> +In case you already fully upgraded to MariaDB 10.6 (which includes
> +executing mysql_upgrade command) and forcefully downgraded your
> +MariaDB instance afterwards during the time window when keywords were
> +removed, you maybe experiencing different problems:
> +
> +At best, your forcefully downgraded MariaDB instance prevented startup
> +so all you have to do is upgrade to MariaDB 10.6 again to resume
> +services.

I don't like the phrase "forcefully downgraded" here. This implies
that something happened without the user's consent. emerge would have
informed them of the downgrade before it happened. I would suggest
removing the word "forcefully" from these paragraphs.

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Marek Szuba]

[-- Attachment #1.1: Type: text/plain, Size: 565 bytes --]

On 2021-11-25 04:49, Mike Gilbert wrote:

> Also, I don't think it is relevant to call out the mistake by the QA
> team in a news item intended for end users.

My sentiment exactly. No finger pointing in news items, please.

> I don't like the phrase "forcefully downgraded" here. This implies 
> that something happened without the user's consent. emerge would
> have informed them of the downgrade before it happened. I would
> suggest removing the word "forcefully" from these paragraphs.

Again, I very much second this suggestion.

-- 
Marecki

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

>>>>> On Thu, 25 Nov 2021, Mike Gilbert wrote:

> On Wed, Nov 24, 2021 at 10:21 PM Thomas Deutschmann <whissi@gentoo.org> wrote:
>> +On 2021-11-21, a member of the QA project accidentially de-keyworded
>> +MariaDB 10.6 to address a file collision, users, who also had latest
>> +dev-db/mariadb-connector-c installed, experienced (NOTE: The default
>> +MySQL connector in Gentoo Linux is provided by
>> +dev-db/mysql-connector-c) [Link 1].

> This sentence is very difficult to read. Also, I don't think it is
> relevant to call out the mistake by the QA team in a news item
> intended for end users. I would rewrite this as:

> On 2021-11-21, keywords for dev-db/mariadb-10.6 were removed to
> address a file collision with dev-db/mariadb-connector-c. This
> unintentionally triggered a version downgrade for users who had
> successfully upgraded to dev-db/mariadb-10.6 already.

+1

News item are to provide information to users, and the details about who
did what in the past aren't necessary here. At most, keep the link to
bug 825234 for reference (but not to any specific comment).

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 1132 bytes --]

Hi,

On 2021-11-25 04:49, Mike Gilbert wrote:
> On 2021-11-21, keywords for dev-db/mariadb-10.6 were removed to
> address a file collision with dev-db/mariadb-connector-c. This
> unintentionally triggered a version downgrade for users who had
> successfully upgraded to dev-db/mariadb-10.6 already.

Works for me. However, I would write dev-db/mariadb:10.6. Is that 
acceptable for you?


> I don't like the phrase "forcefully downgraded" here. This implies
> that something happened without the user's consent. emerge would have
> informed them of the downgrade before it happened. I would suggest
> removing the word "forcefully" from these paragraphs.

If you do a normal world upgrade, this is the default portage behavior, 
not? I.e. package manager will downgrade if you don't stop. And 
especially on servers, people tend to use cronjobs/scripts to do that...

And forcefully here refers to the undesirable result (at least that was 
my intention). Something the user doesn't want.


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Mike Gilbert]

On Thu, Nov 25, 2021 at 8:59 AM Thomas Deutschmann <whissi@gentoo.org> wrote:
>
> Hi,
>
> On 2021-11-25 04:49, Mike Gilbert wrote:
> > On 2021-11-21, keywords for dev-db/mariadb-10.6 were removed to
> > address a file collision with dev-db/mariadb-connector-c. This
> > unintentionally triggered a version downgrade for users who had
> > successfully upgraded to dev-db/mariadb-10.6 already.
>
> Works for me. However, I would write dev-db/mariadb:10.6. Is that
> acceptable for you?

Sure.

> > I don't like the phrase "forcefully downgraded" here. This implies
> > that something happened without the user's consent. emerge would have
> > informed them of the downgrade before it happened. I would suggest
> > removing the word "forcefully" from these paragraphs.
>
> If you do a normal world upgrade, this is the default portage behavior,
> not? I.e. package manager will downgrade if you don't stop. And
> especially on servers, people tend to use cronjobs/scripts to do that...

Something happening by default is not the same as forcing it to happen.

Using a cron job or other blind automation to do package upgrades on
any production system is a bad idea. We certainly do not recommend
that to people, nor force them to do it.

> And forcefully here refers to the undesirable result (at least that was
> my intention). Something the user doesn't want.

That's really not what "forcefully" means. It would be fine to use
"unintentional" or "unwanted" in its place.

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Pacho Ramos]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

El jue, 25-11-2021 a las 09:20 -0500, Mike Gilbert escribió
[...]
> I don't like the phrase "forcefully downgraded" here. This implies
> > > that something happened without the user's consent. emerge would have
> > > informed them of the downgrade before it happened. I would suggest
> > > removing the word "forcefully" from these paragraphs.
> > 
> > If you do a normal world upgrade, this is the default portage behavior,
> > not? I.e. package manager will downgrade if you don't stop. And
> > especially on servers, people tend to use cronjobs/scripts to do that...
> 
> Something happening by default is not the same as forcing it to happen.
> 
> Using a cron job or other blind automation to do package upgrades on
> any production system is a bad idea. We certainly do not recommend
> that to people, nor force them to do it.
> 
> 

As a side note, maybe ebuild should add sanity checks (like those from glibc) to
prevent downgrades, otherwise people could still accidentally hit the issue

Thanks

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Piotr Karbowski]

Hi,

On 25/11/2021 16.50, Pacho Ramos wrote:
> As a side note, maybe ebuild should add sanity checks (like those from glibc) to
> prevent downgrades, otherwise people could still accidentally hit the issue

You might have valid use cases to downgrade mysql, if you are okay not 
preserving data. I'd assume it's a common knowledge that downgrade of 
any data store, be that database or likes like elasticsearch, will 
corrupt the data.

To make it actually useful I think we'd need new EAPI feature 
'downgrade-protection', that unless is explicit ignored like 
--ignore-downgrade-protection mysql, it would prevent it from happening.

-- Piotr.

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[David Seifert]

On Thu, 2021-11-25 at 17:21 +0100, Piotr Karbowski wrote:
> Hi,
> 
> On 25/11/2021 16.50, Pacho Ramos wrote:
> > As a side note, maybe ebuild should add sanity checks (like those from
> > glibc) to
> > prevent downgrades, otherwise people could still accidentally hit the
> > issue
> 
> You might have valid use cases to downgrade mysql, if you are okay not
> preserving data. I'd assume it's a common knowledge that downgrade of 
> any data store, be that database or likes like elasticsearch, will 
> corrupt the data.
> 
> To make it actually useful I think we'd need new EAPI feature 
> 'downgrade-protection', that unless is explicit ignored like 
> --ignore-downgrade-protection mysql, it would prevent it from happening.
> 
> -- Piotr.
> 

https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Piotr Karbowski]

> https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643

Would you see something like this on more ebuilds, postgres, mysql, 
elasticsearch, or have proper FEATURE flag for it instead?

It's all cool and giggles until you realize that even such random 
variable is not even prefixed with PORTAGE_ or anything, meaning it 
could be taken out of shell and meant for entirely different thing.

-- Piotr.

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 1124 bytes --]

On 2021-11-25 18:01, Piotr Karbowski wrote:
>> https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643 
>>
> 
> Would you see something like this on more ebuilds, postgres, mysql, 
> elasticsearch, or have proper FEATURE flag for it instead?
> 
> It's all cool and giggles until you realize that even such random 
> variable is not even prefixed with PORTAGE_ or anything, meaning it 
> could be taken out of shell and meant for entirely different thing.

Yeah, sounds like the I_KNOW_WHAT_I_AM_DOING thing which you end up 
having enabled globally for various reasons.

But thank you Pacho. This is indeed a good suggestion and we will have a 
look how we can improve (there is also some work happening upstream: 
dev-db/mysql-8+ for example removed mysql_upgrade command and will take 
care of everything by itself and prevent startup when it detects a 
downgrade and there is some discussion for such a feature for 10.7 
(probably too late) or 10.8).


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]



> On 25 Nov 2021, at 17:07, Thomas Deutschmann <whissi@gentoo.org> wrote:
> 
> On 2021-11-25 18:01, Piotr Karbowski wrote:
>>> https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643
>> Would you see something like this on more ebuilds, postgres, mysql, elasticsearch, or have proper FEATURE flag for it instead?
>> It's all cool and giggles until you realize that even such random variable is not even prefixed with PORTAGE_ or anything, meaning it could be taken out of shell and meant for entirely different thing.
> 
> Yeah, sounds like the I_KNOW_WHAT_I_AM_DOING thing which you end up having enabled globally for various reasons.

Just like updating in a cron job is a not-great idea, setting this globally and not per-package via /etc/portage/env sounds rather cavalier.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

On Fri, Nov 26, 2021 at 12:55:10AM +0000, Sam James wrote:
> 
> 
> > On 25 Nov 2021, at 17:07, Thomas Deutschmann <whissi@gentoo.org> wrote:
> > 
> > On 2021-11-25 18:01, Piotr Karbowski wrote:
> >>> https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643
> >> Would you see something like this on more ebuilds, postgres, mysql, elasticsearch, or have proper FEATURE flag for it instead?
> >> It's all cool and giggles until you realize that even such random variable is not even prefixed with PORTAGE_ or anything, meaning it could be taken out of shell and meant for entirely different thing.
> > 
> > Yeah, sounds like the I_KNOW_WHAT_I_AM_DOING thing which you end up having enabled globally for various reasons.
> 
> Just like updating in a cron job is a not-great idea, setting this globally and not per-package via /etc/portage/env sounds rather cavalier.

Generally I don't think there should be a central catch-all variable for
this, ebuilds should use their own unlikely variable as needed if using
this route until a better solution.

e.g. MARIADB_I_ALLOW_THE_DESTRUCTION_OF_MY_DATABASE=yes

Could take it further and require that =yes be =${PV} so that it only
works once if forgotten.

Then users setting this in not-so-great ways are still unlikely to be
bitten by it.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Mike Gilbert]

On Thu, Nov 25, 2021 at 12:01 PM Piotr Karbowski <slashbeast@gentoo.org> wrote:
>
> > https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643
>
> Would you see something like this on more ebuilds, postgres, mysql,
> elasticsearch, or have proper FEATURE flag for it instead?
>
> It's all cool and giggles until you realize that even such random
> variable is not even prefixed with PORTAGE_ or anything, meaning it
> could be taken out of shell and meant for entirely different thing.

Another problem with such magic variables is that they are ineffective
when installing binpkgs. During pkg_preinst, portage will restore the
environment variables that were present when the package was built,
and ignore anything you have set in the environment since then.

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On Thu, 2021-11-25 at 18:01 +0100, Piotr Karbowski wrote:
> > https://github.com/gentoo/gentoo/blob/master/sys-libs/glibc/glibc-2.34-r2.ebuild#L643
> 
> Would you see something like this on more ebuilds, postgres, mysql, 
> elasticsearch, or have proper FEATURE flag for it instead?

I don't think there's any need for this with PostgreSQL. The major versions
are slotted, similar to MariaDB, but the data itself is also installed into
versioned locations. Migrating usually involves copying the data to the new
location.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]



> On 25 Nov 2021, at 13:59, Thomas Deutschmann <whissi@gentoo.org> wrote:
> If you do a normal world upgrade, this is the default portage behavior, not? I.e. package manager will downgrade if you don't stop. And especially on servers, people tend to use cronjobs/scripts to do that...

I would really not recommend doing this and I say this regularly. We expect users to:
- read elog output;
- read news items first;
- read pkg_pretend / pkg_postinst output which may be critical and needed to be acted on soon (even if it's accompanied by a news item, although it isn't always).

Should we try to improve that situation? Yeah, I think so, but that's really not the case right now.

> And forcefully here refers to the undesirable result (at least that was my intention). Something the user doesn't want.

As we've discussed at length during the "Bitcoin consent" issue (wrt Taproot), users are expected to read and verify world upgrade output before proceeding.

We have similar issues with e.g. new major glibc versions (you shouldn't do this unless you're in a position to restart at least several services).

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Georgy Yakovlev]

On 25.11.2021 04:21, Thomas Deutschmann wrote:
> Bug: https://bugs.gentoo.org/825234
> Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
> ---
>  ...adb-database-restore-maybe-required.en.txt | 46 +++++++++++++++++++
>  1 file changed, 46 insertions(+)
>  create mode 100644 2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> 
> diff --git a/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> new file mode 100644
> index 0000000..c4a698e
> --- /dev/null
> +++ b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> @@ -0,0 +1,46 @@
> +Title: Full MariaDB database restore maybe required
> +Author: Thomas Deutschmann <whissi@gentoo.org>
> +Posted: 2021-11-23
> +Revision: 1
> +News-Item-Format: 2.0
> +Display-If-Installed: dev-db/mariadb
> +Display-If-Installed: sys-cluster/galera
> +
> +On 2021-11-21, a member of the QA project accidentially de-keyworded
Agreed with others in the thread, this is not professional.

> +MariaDB 10.6 to address a file collision, users, who also had latest
> +dev-db/mariadb-connector-c installed, experienced (NOTE: The default
> +MySQL connector in Gentoo Linux is provided by
> +dev-db/mysql-connector-c) [Link 1].
This ^ is a very confusing sentence and is way too long. Needs shortening and
re-wording, too much commas.

> +
> +However, downgrades are not supported in MySQL/MariaDB [Link 2].
> +
> +In case you already fully upgraded to MariaDB 10.6 (which includes
> +executing mysql_upgrade command) and forcefully downgraded your
> +MariaDB instance afterwards during the time window when keywords were
> +removed, you maybe experiencing different problems:
> +
> +At best, your forcefully downgraded MariaDB instance prevented startup
> +so all you have to do is upgrade to MariaDB 10.6 again to resume
> +services.
> +
> +In case previous MariaDB version was able to start, you are encouraged
> +to do a full backup as soon as possible using mysqldump command and
> +manually restore each database ("logical downgrade") to prevent any
> +data corruption.
> +
> +Depending on used feature set and from which version you upgraded,
> +it is maybe required to do a full restore from a previous backup before
> +MariaDB 10.6 upgrade to restore services and prevent any data loss or
> +future runtime errors.
> +
> +In case you are using MariaDB in a cluster and/or Galera setup you
> +probably have to rebuild the entire cluster in case the upgrade to
> +MariaDB 10.6 was already replicated (using pt-table-checksum from
> +dev-db/percona-toolkit can help you to validate your cluster).
> +
> +Keep in mind that due to the forced downgraded, point-in-time recovery
> +may not be available to the extent that you are used to.
> +
> +Link 1: https://bugs.gentoo.org/825234#c8
> +
> +Link 2: https://mariadb.com/kb/en/downgrading-between-major-versions-of-mariadb/
> -- 
> 2.34.0
> 
> 

-- 
Best regards,
Georgy

[gentoo-dev] Re: [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 825 bytes --]

For the records:
I hope that this news item won't get delayed by the recent ComRel bug 
someone filled against me regarding this news item.

While we often rush with news items and don't wait the 72h listed in 
GLEP 42, this one should go out as soon as possible.

Why?

Because every minute we wait will increase the chance that someone 
affected by this will be unable to recover. This is (for those not 
familiar with database engines) because bin logs are about to expire 
(getting overwritten) in typical setups after 3-5 days. And in case 
someone will learn about this not before next week and has to do a full 
restore, that user will lose about one week of data...

Just saying.


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

[gentoo-dev] Re: [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 210 bytes --]

On 25.11.2021 15.37, Thomas Deutschmann wrote:
> For the records:

For the records:
This post could've been an attempt to get it merged faster. But no, you
chose to prolong it even further.

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1697 bytes --]


> On 25 Nov 2021, at 13:37, Thomas Deutschmann <whissi@gentoo.org> wrote:
> For the records:
> I hope that this news item won't get delayed by the recent ComRel bug someone filled against me regarding this news item.

No need to reference that here. Please keep that to e.g. -core.

> While we often rush with news items and don't wait the 72h listed in GLEP 42, this one should go out as soon as possible.
> Why?
> Because every minute we wait will increase the chance that someone affected by this will be unable to recover. This is (for those not familiar with database engines) because bin logs are about to expire (getting overwritten) in typical setups after 3-5 days. And in case someone will learn about this not before next week and has to do a full restore, that user will lose about one week of data...

While it seems like the news item may be worth pursuing out of caution, I would expect people operating sophisticated database setups to do at least one of:
- slot the DB version in their world file;
- mask newer versions and unmask when ready;
- upgrade all machines to the same version at the same time (for clustering);
- read the world upgrade output and not permit the downgrade?

I also echo the other requests to add a downgrade warning based on ${REPLACING_VERSIONS} given this is a
common issue with databases.

Also, while this does not affect the rationale for this news item, I would note that I would expect
anybody using Gentoo in production to be using stable or at the very least to have backups
of a server running on ~arch that they're upgrading without reading the upgrades/downgrades
carefully for software they rely on.

thanks,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-dev] [PATCHv2] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

Bug: https://bugs.gentoo.org/825234
Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
---
 ...adb-database-restore-maybe-required.en.txt | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt

diff --git a/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
new file mode 100644
index 0000000..5d8afda
--- /dev/null
+++ b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
@@ -0,0 +1,45 @@
+Title: Full MariaDB database restore maybe required
+Author: Thomas Deutschmann <whissi@gentoo.org>
+Posted: 2021-11-23
+Revision: 1
+News-Item-Format: 2.0
+Display-If-Installed: dev-db/mariadb
+Display-If-Installed: sys-cluster/galera
+
+On 2021-11-21, keywords for dev-db/mariadb:10.6 were removed to
+address a file collision with dev-db/mariadb-connector-c. This
+unintentionally triggered a version downgrade for users who had
+successfully upgraded to dev-db/mariadb:10.6 already. [Link 1].
+
+However, downgrades are not supported in MySQL/MariaDB [Link 2].
+
+In case you already fully upgraded to MariaDB 10.6 (which includes
+executing mysql_upgrade command) and unintentionally downgraded your
+MariaDB instance afterwards during the time window when keywords were
+removed, you maybe experiencing different problems:
+
+At best, your unwanted downgraded MariaDB instance prevented startup
+so all you have to do is upgrade to MariaDB 10.6 again to resume
+services.
+
+In case previous MariaDB version was able to start, you are encouraged
+to do a full backup as soon as possible using mysqldump command and
+manually restore each database ("logical downgrade") to prevent any
+data corruption.
+
+Depending on used feature set and from which version you upgraded,
+it is maybe required to do a full restore from a previous backup before
+MariaDB 10.6 upgrade to restore services and prevent any data loss or
+future runtime errors.
+
+In case you are using MariaDB in a cluster and/or Galera setup you
+probably have to rebuild the entire cluster in case the upgrade to
+MariaDB 10.6 was already replicated (using pt-table-checksum from
+dev-db/percona-toolkit can help you to validate your cluster).
+
+Keep in mind that due to the forced downgraded, point-in-time recovery
+may not be available to the extent that you are used to.
+
+Link 1: https://bugs.gentoo.org/825234
+
+Link 2: https://mariadb.com/kb/en/downgrading-between-major-versions-of-mariadb/
-- 
2.34.0

Re: [gentoo-dev] [PATCHv2] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Mike Gilbert]

On Thu, Nov 25, 2021 at 1:26 PM Thomas Deutschmann <whissi@gentoo.org> wrote:
> +Keep in mind that due to the forced downgraded, point-in-time recovery
> +may not be available to the extent that you are used to.

Sorry, I guess I missed this instance of "forced". Also, "downgraded"
is the incorrect form to use here. It should be something like:

"Keep in mind that due to the downgrade, point-in-time recovery may
not be available to the extent that you are used to."

[gentoo-dev] [PATCHv3] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Thomas Deutschmann]

Bug: https://bugs.gentoo.org/825234
Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
---
 ...adb-database-restore-maybe-required.en.txt | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt

diff --git a/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
new file mode 100644
index 0000000..c977ae7
--- /dev/null
+++ b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
@@ -0,0 +1,45 @@
+Title: Full MariaDB database restore maybe required
+Author: Thomas Deutschmann <whissi@gentoo.org>
+Posted: 2021-11-23
+Revision: 1
+News-Item-Format: 2.0
+Display-If-Installed: dev-db/mariadb
+Display-If-Installed: sys-cluster/galera
+
+On 2021-11-21, keywords for dev-db/mariadb:10.6 were removed to
+address a file collision with dev-db/mariadb-connector-c. This
+unintentionally triggered a version downgrade for users who had
+successfully upgraded to dev-db/mariadb:10.6 already. [Link 1].
+
+However, downgrades are not supported in MySQL/MariaDB [Link 2].
+
+In case you already fully upgraded to MariaDB 10.6 (which includes
+executing mysql_upgrade command) and unintentionally downgraded your
+MariaDB instance afterwards during the time window when keywords were
+removed, you maybe experiencing different problems:
+
+At best, your unwanted downgraded MariaDB instance prevented startup
+so all you have to do is upgrade to MariaDB 10.6 again to resume
+services.
+
+In case previous MariaDB version was able to start, you are encouraged
+to do a full backup as soon as possible using mysqldump command and
+manually restore each database ("logical downgrade") to prevent any
+data corruption.
+
+Depending on used feature set and from which version you upgraded,
+it is maybe required to do a full restore from a previous backup before
+MariaDB 10.6 upgrade to restore services and prevent any data loss or
+future runtime errors.
+
+In case you are using MariaDB in a cluster and/or Galera setup you
+probably have to rebuild the entire cluster in case the upgrade to
+MariaDB 10.6 was already replicated (using pt-table-checksum from
+dev-db/percona-toolkit can help you to validate your cluster).
+
+Keep in mind that due to the downgrade, point-in-time recovery may
+not be available to the extent that you are used to.
+
+Link 1: https://bugs.gentoo.org/825234
+
+Link 2: https://mariadb.com/kb/en/downgrading-between-major-versions-of-mariadb/
-- 
2.34.0

Re: [gentoo-dev] [PATCHv3] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 1687 bytes --]

On 25.11.2021 22.54, Thomas Deutschmann wrote:
> Bug: https://bugs.gentoo.org/825234
> Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
> ---
>  ...adb-database-restore-maybe-required.en.txt | 45 +++++++++++++++++++
>  1 file changed, 45 insertions(+)
>  create mode 100644 2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> 
> diff --git a/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> new file mode 100644
> index 0000000..c977ae7
> --- /dev/null
> +++ b/2021-11-23-mariadb-database-restore-maybe-required/2021-11-23-mariadb-database-restore-maybe-required.en.txt
> @@ -0,0 +1,45 @@
> +Title: Full MariaDB database restore maybe required
> +Author: Thomas Deutschmann <whissi@gentoo.org>
> +Posted: 2021-11-23
> +Revision: 1
> +News-Item-Format: 2.0
> +Display-If-Installed: dev-db/mariadb
> +Display-If-Installed: sys-cluster/galera
> +
> +On 2021-11-21, keywords for dev-db/mariadb:10.6 were removed to
> +address a file collision with dev-db/mariadb-connector-c. This
> +unintentionally triggered a version downgrade for users who had
> +successfully upgraded to dev-db/mariadb:10.6 already. [Link 1].

I'd add:
"successfully upgraded to dev-db/mariadb:10.6 already during a short
time period. [Link 1]." to make it clearer this doesn't concern
absolutely everyone.

v3 LGTM regardless of that addition. More comments, more acks to get
this merged even possibly tonight (UTC)? Anyone from PR?

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCHv3] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

>>>>> On Fri, 26 Nov 2021, Joonas Niilola wrote:

> v3 LGTM regardless of that addition. More comments, more acks to get
> this merged even possibly tonight (UTC)? Anyone from PR?

Ack. Looks good now.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] 2021-11-23-mariadb-database-restore-maybe-required: add item

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]



> On 25 Nov 2021, at 03:21, Thomas Deutschmann <whissi@gentoo.org> wrote:
> 
> Bug: https://bugs.gentoo.org/825234
> Signed-off-by: Thomas Deutschmann <whissi@gentoo.org>
> ---
> [snip]

> +On 2021-11-21, a member of the QA project accidentially de-keyworded
> +MariaDB 10.6 to address a file collision, users, who also had latest
> +dev-db/mariadb-connector-c installed, experienced (NOTE: The default
> +MySQL connector in Gentoo Linux is provided by
> +dev-db/mysql-connector-c) [Link 1].

There's no reason to refer to MySQL here (nor am I sure that we're
really "defaulting" to either MariaDB or MySQL).

> +
> +However, downgrades are not supported in MySQL/MariaDB [Link 2].

There's no reason to refer to MySQL here.

> +Link 1: https://bugs.gentoo.org/825234#c8

No need to link to the comment. Also, please use the standard [0] and [1] reference
notation.


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]