[gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Dale]

Howdy,

Last night we had some bad weather where I live and we ended up with
some power problems.  Ironically they went out a few hours after the
storm was gone.  Anyway.  I had all sorts of encrypted drives open.  My
usual drives inside my puter plus the large 14TB external backup drive
that is still copying files over.  Glad my UPS held up while I closed
all those drives and did a proper shutdown.  Doing all that tho, it made
me think about if I wasn't here to do all that.  Being Linux, I'd
suspect that upsmon would tell the puter to do a proper shutdown which
includes unmounting the file system, closing the encrypted drives, like
I do with cryptsetup close <name> etc and then shutting down.  However,
one has to ask, is it set up to do so by default?  I manage the
encrypted drives manually.  I don't use the crypt services for that like
people do when all of the system drive(s) is encrypted or when just
/home is encrypted.  My encrypted stuff is mounted within /home or for
the external backups, in /mnt.  Thing is, some aren't open unless I'm
using them or are external.  Since I do it manually, is there a tool
that sees they need unmounting and closing and does it or do I need to
do something to make sure it is done before a shutdown? 

I suspect this would happen on its own but I'd like to make sure.  I'd
hate to mess up the file system badly on any of my drives or in a worst
case scenario, brick a hard drive with some 1 in a million chance problem.

I thought about having a drive connected, open and mounted that I don't
really need and just do a shutdown, see what happens.  Then again, why
not ask and see if anyone else has had this happen and if things turned
out OK or if there was problems.  I'm lucky, most of the time I'm either
home or very close by.  Still, it can happen when I'm not here.  I
already wonder if upsmon will kick in correctly and do a proper
shutdown.  After all, it has never had to before.  I'm running on faith
that it will.  I hope I'm right. 

Thoughts?  Default will take care of things?  I need to take steps to be
sure in case I'm not here?  Personal experience?  A good theory?  ;-)

Thanks.

Dale

:-)  :-)

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[tastytea]

On 2022-09-11 20:56-0500 Dale <rdalek1967@gmail.com> wrote:

> Howdy,
> 
> Last night we had some bad weather where I live and we ended up with
> some power problems.  Ironically they went out a few hours after the
> storm was gone.  Anyway.  I had all sorts of encrypted drives open.
> My usual drives inside my puter plus the large 14TB external backup
> drive that is still copying files over.  Glad my UPS held up while I
> closed all those drives and did a proper shutdown.  Doing all that
> tho, it made me think about if I wasn't here to do all that.  Being
> Linux, I'd suspect that upsmon would tell the puter to do a proper
> shutdown which includes unmounting the file system, closing the
> encrypted drives, like I do with cryptsetup close <name> etc and then
> shutting down.  However, one has to ask, is it set up to do so by
> default?  I manage the encrypted drives manually.  I don't use the
> crypt services for that like people do when all of the system
> drive(s) is encrypted or when just /home is encrypted.  My encrypted
> stuff is mounted within /home or for the external backups, in /mnt.
> Thing is, some aren't open unless I'm using them or are external.
> Since I do it manually, is there a tool that sees they need
> unmounting and closing and does it or do I need to do something to
> make sure it is done before a shutdown? 
> 
> I suspect this would happen on its own but I'd like to make sure.  I'd
> hate to mess up the file system badly on any of my drives or in a
> worst case scenario, brick a hard drive with some 1 in a million
> chance problem.
> 
> I thought about having a drive connected, open and mounted that I
> don't really need and just do a shutdown, see what happens.  Then
> again, why not ask and see if anyone else has had this happen and if
> things turned out OK or if there was problems.  I'm lucky, most of
> the time I'm either home or very close by.  Still, it can happen when
> I'm not here.  I already wonder if upsmon will kick in correctly and
> do a proper shutdown.  After all, it has never had to before.  I'm
> running on faith that it will.  I hope I'm right. 
> 
> Thoughts?  Default will take care of things?  I need to take steps to
> be sure in case I'm not here?  Personal experience?  A good theory?
> ;-)

Yes, /etc/init.d/mount-ro will take care of that. It first calls `sync`
and then calls `umount -r` on everything. It's set up to ruin on
shutdown by default. I'm sure systemd does something similar.

I don't think `cryptsetup luksClose` is necessary on shutdown, since it
only sets up the mapping(?).

Kind regards, tastytea

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Dale]

tastytea wrote:
> On 2022-09-11 20:56-0500 Dale <rdalek1967@gmail.com> wrote:
>
>> Howdy,
>>
>> Last night we had some bad weather where I live and we ended up with
>> some power problems.  Ironically they went out a few hours after the
>> storm was gone.  Anyway.  I had all sorts of encrypted drives open.
>> My usual drives inside my puter plus the large 14TB external backup
>> drive that is still copying files over.  Glad my UPS held up while I
>> closed all those drives and did a proper shutdown.  Doing all that
>> tho, it made me think about if I wasn't here to do all that.  Being
>> Linux, I'd suspect that upsmon would tell the puter to do a proper
>> shutdown which includes unmounting the file system, closing the
>> encrypted drives, like I do with cryptsetup close <name> etc and then
>> shutting down.  However, one has to ask, is it set up to do so by
>> default?  I manage the encrypted drives manually.  I don't use the
>> crypt services for that like people do when all of the system
>> drive(s) is encrypted or when just /home is encrypted.  My encrypted
>> stuff is mounted within /home or for the external backups, in /mnt.
>> Thing is, some aren't open unless I'm using them or are external.
>> Since I do it manually, is there a tool that sees they need
>> unmounting and closing and does it or do I need to do something to
>> make sure it is done before a shutdown? 
>>
>> I suspect this would happen on its own but I'd like to make sure.  I'd
>> hate to mess up the file system badly on any of my drives or in a
>> worst case scenario, brick a hard drive with some 1 in a million
>> chance problem.
>>
>> I thought about having a drive connected, open and mounted that I
>> don't really need and just do a shutdown, see what happens.  Then
>> again, why not ask and see if anyone else has had this happen and if
>> things turned out OK or if there was problems.  I'm lucky, most of
>> the time I'm either home or very close by.  Still, it can happen when
>> I'm not here.  I already wonder if upsmon will kick in correctly and
>> do a proper shutdown.  After all, it has never had to before.  I'm
>> running on faith that it will.  I hope I'm right. 
>>
>> Thoughts?  Default will take care of things?  I need to take steps to
>> be sure in case I'm not here?  Personal experience?  A good theory?
>> ;-)
> Yes, /etc/init.d/mount-ro will take care of that. It first calls `sync`
> and then calls `umount -r` on everything. It's set up to ruin on
> shutdown by default. I'm sure systemd does something similar.
>
> I don't think `cryptsetup luksClose` is necessary on shutdown, since it
> only sets up the mapping(?).
>
> Kind regards, tastytea
>
>


Thanks much for this info.  I figured there was some tool that would do
that regardless of what it was.  I know regular file systems would be
and couldn't imagine that encrypted would be any different but I didn't
want to find out I was wrong the hard way.  After all, this 14TB backup
has been running for a few days now. Even when it gets through, I have
to run it again because of additions and other changes I made in the
past few days.  While I could just start over with a fresh backup if it
got damaged, it would be time consuming to do so.  Also, it would put
data at risk if I had a failure of the running drives while that backup
was not available.  Not likely but bad things happen. 

Next time power fails, I'll just stop all the processes I can and then
do a shutdown, knowing that everything will close safely.  That will
save me some battery time as well. 

Thanks much. 

Dale

:-)  :-)

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[William Kenworthy]

If your using nut, it has to be setup - and should be regularly tested 
to make sure it works.

BillK

'

On 12/9/22 09:56, Dale wrote:
> Howdy,
>
> Last night we had some bad weather where I live and we ended up with
> some power problems.  Ironically they went out a few hours after the
> storm was gone.  Anyway.  I had all sorts of encrypted drives open.  My
> usual drives inside my puter plus the large 14TB external backup drive
> that is still copying files over.  Glad my UPS held up while I closed
> all those drives and did a proper shutdown.  Doing all that tho, it made
> me think about if I wasn't here to do all that.  Being Linux, I'd
> suspect that upsmon would tell the puter to do a proper shutdown which
> includes unmounting the file system, closing the encrypted drives, like
> I do with cryptsetup close <name> etc and then shutting down.  However,
> one has to ask, is it set up to do so by default?  I manage the
> encrypted drives manually.  I don't use the crypt services for that like
> people do when all of the system drive(s) is encrypted or when just
> /home is encrypted.  My encrypted stuff is mounted within /home or for
> the external backups, in /mnt.  Thing is, some aren't open unless I'm
> using them or are external.  Since I do it manually, is there a tool
> that sees they need unmounting and closing and does it or do I need to
> do something to make sure it is done before a shutdown?
>
> I suspect this would happen on its own but I'd like to make sure.  I'd
> hate to mess up the file system badly on any of my drives or in a worst
> case scenario, brick a hard drive with some 1 in a million chance problem.
>
> I thought about having a drive connected, open and mounted that I don't
> really need and just do a shutdown, see what happens.  Then again, why
> not ask and see if anyone else has had this happen and if things turned
> out OK or if there was problems.  I'm lucky, most of the time I'm either
> home or very close by.  Still, it can happen when I'm not here.  I
> already wonder if upsmon will kick in correctly and do a proper
> shutdown.  After all, it has never had to before.  I'm running on faith
> that it will.  I hope I'm right.
>
> Thoughts?  Default will take care of things?  I need to take steps to be
> sure in case I'm not here?  Personal experience?  A good theory?  ;-)
>
> Thanks.
>
> Dale
>
> :-)  :-)
>

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Dale]

William Kenworthy wrote:
> If your using nut, it has to be setup - and should be regularly tested
> to make sure it works.
>
> BillK


I think upsmon is part of nut.  I keep forgetting that since the service
is ups something.  Thing is, I've never quite figured out how to test it
without unplugging and running down the batteries.  I do have everything
configured and have ever since I built this puter.  I did the same on
previous puter and UPS.  Pretty sure it will work.  When power blinks
etc, it does notice it and logs it in messages file. Also, the upsc
command outputs the info correctly when run.

I wish I could send a command to the UPS to fake a power failure, wait
say one minute and then it tell puter to shutdown all on its own.  If it
does it correctly, it should work in the event of a actual power failure
and not run down my batteries either.  I try to keep the batteries
topped off at all times since we do on occasion have some crazy driver
not watching where he/she is going and hits a power pole.  Doesn't do
the pole any good and the car seems to not enjoy it either.  :/ 
Sometimes the driver is no longer caring about it. 

I actually wish I had a much larger external battery.  Thing is, I'm
concerned about the charging bit.  The charging section in a UPS isn't
really that powerful since it mostly just keeps the batteries topped
off.  I have 7Amp/hr batteries and I think it takes like 7 or 8 hours to
charge from almost dead.  I'd guess it is at most a 1 to 1.5 amp
charging circuit. 

If you know of a command to test without running down batteries, I have
a Cyberpower UPS and I'm certainly interested.  It's only a decade or so
old so has quite a few features.  There may be a way to do this but I've
yet to find it. 

Thanks.

Dale

:-)  :-)

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Michael]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]

On Monday, 12 September 2022 06:57:58 BST Dale wrote:
> William Kenworthy wrote:
> > If your using nut, it has to be setup - and should be regularly tested
> > to make sure it works.
> > 
> > BillK
> 
> I think upsmon is part of nut.  I keep forgetting that since the service
> is ups something.  Thing is, I've never quite figured out how to test it
> without unplugging and running down the batteries.  I do have everything
> configured and have ever since I built this puter.  I did the same on
> previous puter and UPS.  Pretty sure it will work.  When power blinks
> etc, it does notice it and logs it in messages file. Also, the upsc
> command outputs the info correctly when run.
> 
> I wish I could send a command to the UPS to fake a power failure, wait
> say one minute and then it tell puter to shutdown all on its own.  If it
> does it correctly, it should work in the event of a actual power failure
> and not run down my batteries either.  I try to keep the batteries
> topped off at all times since we do on occasion have some crazy driver
> not watching where he/she is going and hits a power pole.  Doesn't do
> the pole any good and the car seems to not enjoy it either.  :/ 
> Sometimes the driver is no longer caring about it. 
> 
> I actually wish I had a much larger external battery.  Thing is, I'm
> concerned about the charging bit.  The charging section in a UPS isn't
> really that powerful since it mostly just keeps the batteries topped
> off.  I have 7Amp/hr batteries and I think it takes like 7 or 8 hours to
> charge from almost dead.  I'd guess it is at most a 1 to 1.5 amp
> charging circuit. 

Many UPSs containing two batteries connected in series always end up with one 
battery less charged than the other.  This is because although the current 
which flows through the circuit is the same, due to Ohm's Law the voltage 
drops across each battery as the resistance decreases along the circuit.  
Therefore the first battery never gets fully charged.  The opposite happens 
with a discharge cycle.

Once every three months or every time I experience a noticeably prolonged 
power cut, I shutdown the loads, take the UPS off line, disconnect the 
batteries and top up each battery's charge with a car battery charger.  Then I 
replace them but make sure I reverse their positions.  I have found this is 
the best way to maximise their service life.

I also use the car battery charger to top up single battery UPSs, but I only 
do this twice a year.  I have found the charge these single batteries need is 
comparably shorter.


> If you know of a command to test without running down batteries, I have
> a Cyberpower UPS and I'm certainly interested.  It's only a decade or so
> old so has quite a few features.  There may be a way to do this but I've
> yet to find it. 
> 
> Thanks.
> 
> Dale
> 
> :-)  :-)

Check the NUT command set and options, in particular upsmon.  You'll probably 
want to try issuing a 'forced shutdown' command 'upsmon -c fsd', which won't 
wait for the batteries to discharge first, like it does with upssched.  Before 
you try this, best sync your disks first and remount them read only just in 
case.

There may be some UPS specific test function too, this very much depends on 
the UPS and driver.  You can check what variables a UPS has and how to tweak 
them to cause a shutdown by running 'upsrw -l', if any of the exposed UPS 
variable are tweakable.


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Laurence Perkins]

>-----Original Message-----
>From: Dale <rdalek1967@gmail.com> 
>Sent: Sunday, September 11, 2022 10:58 PM
>To: gentoo-user@lists.gentoo.org
>Subject: Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.
>
>William Kenworthy wrote:
>> If your using nut, it has to be setup - and should be regularly tested 
>> to make sure it works.
>>
>> BillK
>
>
>I think upsmon is part of nut.  I keep forgetting that since the service is ups something.  Thing is, I've never quite figured out how to test it without unplugging and running down the batteries.  I do have everything configured and have ever since I built this puter.  I did the same on previous puter and UPS.  Pretty sure it will work.  When power blinks etc, it does notice it and logs it in messages file. Also, the upsc command outputs the info correctly when run.
>
>I wish I could send a command to the UPS to fake a power failure, wait say one minute and then it tell puter to shutdown all on its own.  If it does it correctly, it should work in the event of a actual power failure and not run down my batteries either.  I try to keep the batteries topped off at all times since we do on occasion have some crazy driver not watching where he/she is going and hits a power pole.  Doesn't do the pole any good and the car seems to not enjoy it either.  :/ Sometimes the driver is no longer caring about it. 
>
>I actually wish I had a much larger external battery.  Thing is, I'm concerned about the charging bit.  The charging section in a UPS isn't really that powerful since it mostly just keeps the batteries topped off.  I have 7Amp/hr batteries and I think it takes like 7 or 8 hours to charge from almost dead.  I'd guess it is at most a 1 to 1.5 amp charging circuit. 
>
>If you know of a command to test without running down batteries, I have a Cyberpower UPS and I'm certainly interested.  It's only a decade or so old so has quite a few features.  There may be a way to do this but I've yet to find it. 
>
>Thanks.
>
>Dale
>
>:-)  :-)
>

Some of the higher-end UPS models do have diagnostic modes for simulating various events to make sure the connected systems behave as desired.  A very few of the consumer-grade ones do as well.  But how to do it is model specific, so you'll have to dig up the documentation.

Commercial-grade units also often have a DC port on the back so you can plug in larger battery banks and/or hotswap battery banks during extended outages.

If you want an arbitrarily large battery bank, just get a decent power inverter heavy enough to run your load and a battery float charger that can push enough amps to keep up, then put as big a stack of batteries as you like between the two.  The nicer inverters will even warn you when the batteries get low.

You can often get used batteries from the local automotive shop for just the core charge.  Just because it can't provide 600 amps to start a car any more doesn't mean it can't provide 60 to run your computers.  Obviously they'll require more regular maintenance, but it's hard to beat the price.

LMP

[gentoo-user] RE: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Thomas Mueller]

from Laurence Perkins:

> Some of the higher-end UPS models do have diagnostic modes for simulating various events to make sure the connected systems behave as desired.  A very few of the consumer-grade ones do as well.  But how to do it is model specific,
> so you'll have to dig up the documentation.

> Commercial-grade units also often have a DC port on the back so you can plug in larger battery banks and/or hotswap battery banks during extended outages.

> If you want an arbitrarily large battery bank, just get a decent power inverter heavy enough to run your load and a battery float charger that can push enough amps to keep up, then put as big a stack of batteries as you like
> between the two.  The nicer inverters will even warn you when the batteries get low.

> You can often get used batteries from the local automotive shop for just the core charge.  Just because it can't provide 600 amps to start a car any more doesn't mean it can't provide 60 to run your computers.  Obviously they'll
> require more regular maintenance, but it's hard to beat the price.

> LMP     

How would you physically connect the automotive battery to the computer, and would you need the shell of the old UPS?

I have an old Tripp-Lite UPS, batteries are dead and no longer rechargeable.

Would you connect only the computer, or would you connect the monitor as well?  Would you connect networking equipment?

Tom

Re: [gentoo-user] RE: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2507 bytes --]

On Tuesday, 13 September 2022 06:47:21 BST Thomas Mueller wrote:
> from Laurence Perkins:
> > Some of the higher-end UPS models do have diagnostic modes for simulating
> > various events to make sure the connected systems behave as desired.  A
> > very few of the consumer-grade ones do as well.  But how to do it is
> > model specific, so you'll have to dig up the documentation.
> > 
> > Commercial-grade units also often have a DC port on the back so you can
> > plug in larger battery banks and/or hotswap battery banks during extended
> > outages.
> > 
> > If you want an arbitrarily large battery bank, just get a decent power
> > inverter heavy enough to run your load and a battery float charger that
> > can push enough amps to keep up, then put as big a stack of batteries as
> > you like between the two.  The nicer inverters will even warn you when
> > the batteries get low.
> > 
> > You can often get used batteries from the local automotive shop for just
> > the core charge.  Just because it can't provide 600 amps to start a car
> > any more doesn't mean it can't provide 60 to run your computers. 
> > Obviously they'll require more regular maintenance, but it's hard to beat
> > the price.
> > 
> > LMP
> 
> How would you physically connect the automotive battery to the computer, and
> would you need the shell of the old UPS?

Yes, you need the *contents* of the UPS shell. It contains the rectifier to be 
able to recharge the battery/batteries and the inverter to be able to feed the 
PC from the batteries.

I've also used a car battery out of an old Mini to keep equipment running 
during prolonged power outages.  I connected it in parallel to the UPS 
battery.  After the power returned, almost two days later, I disconnected it 
and recharged it with a car battery charger, which could take the higher 
amperage.  Trickle-charging should be OK via the UPS.


> I have an old Tripp-Lite UPS, batteries are dead and no longer rechargeable.

Why don't you replace them?  They are not particularly expensive, although I 
have not looked at UPS battery prices lately.


> Would you connect only the computer, or would you connect the monitor as
> well?  Would you connect networking equipment?
> 
> Tom

I connect modem, router, PC and monitor, so that whatever operation is taking 
place it can be completed, applications running ended and the PC shutdown 
manually.  If the PC is running unattended when the power interruption 
happens, eventually the upssched will shutdown the PC.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [gentoo-user] RE: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Laurence Perkins]

>> If you want an arbitrarily large battery bank, just get a decent power 
>> inverter heavy enough to run your load and a battery float charger that can push enough amps to keep up, then put as big a stack of batteries as you like between the two.  The nicer inverters will even warn you when the batteries get low.
>
>> You can often get used batteries from the local automotive shop for 
>> just the core charge.  Just because it can't provide 600 amps to start a car any more doesn't mean it can't provide 60 to run your computers.  Obviously they'll require more regular maintenance, but it's hard to beat the price.
>
>> LMP     
>
>How would you physically connect the automotive battery to the computer, and would you need the shell of the old UPS?
>
>I have an old Tripp-Lite UPS, batteries are dead and no longer rechargeable.
>
>Would you connect only the computer, or would you connect the monitor as well?  Would you connect networking equipment?
>
>Tom
>
>

If your old UPS had sufficient capacity, you can usually just open it up (make sure it's unplugged and turned off first, obviously) and replace the battery with one of the same size.  Often there's an access panel for doing exactly that.

Otherwise, you'll need to extend the battery leads out through the side of the case to whatever battery bank you're using.  Keeping the case would definitely be recommended since there will be high voltage in the system when it's in operation.  Also the new battery bank will need to be the same chemistry and voltage as the original.  Usually it's lead-acid and multiples of 12 volts, (hence why old automotive batteries can work, but will require test and service after every power outage because they're designed for high output, not deep discharge) but do double check.

What you connect to it depends on what you figure needs battery backup and how many amps the inverter in the UPS can supply without overheating.

Note that messing too much with the internals of a power inverter can cause fire or electrocution if done improperly, so make sure you research how it all works before modifying anything.

LMP

Re: [gentoo-user] Encrypted hard drives on LVM and urgent power shutdowns.

[Rich Freeman]

On Sun, Sep 11, 2022 at 9:56 PM Dale <rdalek1967@gmail.com> wrote:
>
> I suspect this would happen on its own but I'd like to make sure.  I'd
> hate to mess up the file system badly on any of my drives or in a worst
> case scenario, brick a hard drive with some 1 in a million chance problem.
>

I just wanted to comment that LUKS encryption on linux is pretty-much
a block-level passthrough.  So if your filesystem is journaled and
using barriers or syncing to ensure consistency, and you add LUKS to
it, then you shouldn't really see any difference in behavior if it is
interrupted uncleanly by a power loss.  The encryption could add a bit
of latency but that shouldn't change much.

Of course different filesystems handle interruptions differently, and
all those caveats still apply.

As far as unmounting goes, you just need to umount the filesystem.
umount will block until all writes are synced to disk, and that
includes all layers like LVM/LUKS/mdadm/whatever that might be
underneath it.  If umount returns, then all your data is written to
disk and if at that instant you lose power there will be no data loss
for that filesystem.  I guess if you're using mdadm and you have
multiple filesystems not aligned to a stripe boundary, then the raid
write hole might still apply, and that is true at anytime whether the
filesystem is mounted or not - data on a stripe shared with some other
active filesystem could get lost in some situations.

Obviously if you lose the key to a LUKS filesystem or if there is some
kind of bug in LUKS the use of encryption could hinder data recovery.
Beyond that it really shouldn't have any impact on anything.  I guess
it would also give you more exposure to RAM errors (since that is
another code path that stores stuff in RAM).

As already discussed, clean shutdowns triggered by NUT/etc are of
course best, but the use of LUKS shouldn't change much with the use of
a UPS otherwise.

-- 
Rich