[gentoo-user] qbittorrent crashes after to many files open error

[gentoo-user] qbittorrent crashes after to many files open error

[Dale]

Howdy,

As some know, I discovered torrentting a while back.  It has caused
issues ever since.  LOL  I recently upgraded qbittorrent.  Other than
having to limit some speed settings since it would make my desktop
response slow, it has worked OK, better than ktorrent at least.  Then a
couple days ago, perhaps related to a upgrade, it would crash.  In the
notifications I would find a error like below.  I have changed the names
to protect the innocent.  ;-)

An I/O error occurred for torrent 'ABCDEF'.
Reason: ABCDEF file_open (/home/dale/Desktop/Videos/ABCDEF error: Too
many open files


I did a google search and found out more info which lead me to this
eventually: 


https://www.tecmint.com/increase-set-open-file-limits-in-linux/

Then this:

root@fireball / # sysctl fs.file-max
fs.file-max = 3289952
root@fireball / # sysctl -w fs.file-max=32899520
fs.file-max = 32899520
root@fireball / # cat /proc/sys/fs/file-max
32899520


So, I increased the limit on open files by a factor of ten, I added a
zero on the end.  It was easy enough and didn't require a calculator or
other fancy maths.  Question is, is this a better fix or could it just
be qbittorrent itself having issues?  Could there be more to this or
something else causing this error?

I was using qbittorrent-4.4.5 but downgraded to qbittorrent-4.4.4.  I'm
hoping one or the other will fix this crashing issue.  It did only start
after the upgrade but could be a coincidence to I guess. 


By the way, the 10TB drive I bought a couple weeks or so ago, well, this
is it now. 

/dev/mapper/10tb           9.1T  8.7T  345G  97% /mnt/10tb


The 14TB is supposed to be here this weekend.  I have really got to
create a solution to this.  My current plan, make the 14TB my backup
drive.  Put the 10TB in my rig, for now.  It's getting full too.  ROFL 
I could cut off the internet I guess.  ROFLMBO


Thoughts on the files open error?  Qbittorrent crashing?  Anyone else
ran into this before?  Proper solution?  Better solution? 


Dale

:-)  :-) 

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Mateusz Okulus]

> root@fireball / # sysctl fs.file-max
> fs.file-max = 3289952
> root@fireball / # sysctl -w fs.file-max=32899520
> fs.file-max = 32899520
> root@fireball / # cat /proc/sys/fs/file-max
> 32899520

On my pc the limit is 2^63. But this is max files for kernel, which is
different for max files per user and max files per process.

For example if I run:

$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            524288               524288               files

524288 = 2^19

QBittorrent increases it's process limit to hard limit. With 10 TB of
torrents it's hard to tell if it's a bug or just 500k files is not
enough, I'd say it's probably the latter.

It seems you can change the default hard limit in
/etc/security/limits.conf

For example:

# 2^24, increased from 2^19
* hard nofile 16777216

Then after reloging:

$ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            16777216             16777216             files

So this works, at least for me.

Check also limits.conf man page.

Regards,
mmokulus

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Lee]

[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]

Divine punishment perhaps?

Lee 😎

On Wed, Sep 7, 2022, 10:26 PM Dale <rdalek1967@gmail.com> wrote:

> Howdy,
>
> As some know, I discovered torrentting a while back.  It has caused
> issues ever since.  LOL  I recently upgraded qbittorrent.  Other than
> having to limit some speed settings since it would make my desktop
> response slow, it has worked OK, better than ktorrent at least.  Then a
> couple days ago, perhaps related to a upgrade, it would crash.  In the
> notifications I would find a error like below.  I have changed the names
> to protect the innocent.  ;-)
>
> An I/O error occurred for torrent 'ABCDEF'.
> Reason: ABCDEF file_open (/home/dale/Desktop/Videos/ABCDEF error: Too
> many open files
>
>
> I did a google search and found out more info which lead me to this
> eventually:
>
>
> https://www.tecmint.com/increase-set-open-file-limits-in-linux/
>
> Then this:
>
> root@fireball / # sysctl fs.file-max
> fs.file-max = 3289952
> root@fireball / # sysctl -w fs.file-max=32899520
> fs.file-max = 32899520
> root@fireball / # cat /proc/sys/fs/file-max
> 32899520
>
>
> So, I increased the limit on open files by a factor of ten, I added a
> zero on the end.  It was easy enough and didn't require a calculator or
> other fancy maths.  Question is, is this a better fix or could it just
> be qbittorrent itself having issues?  Could there be more to this or
> something else causing this error?
>
> I was using qbittorrent-4.4.5 but downgraded to qbittorrent-4.4.4.  I'm
> hoping one or the other will fix this crashing issue.  It did only start
> after the upgrade but could be a coincidence to I guess.
>
>
> By the way, the 10TB drive I bought a couple weeks or so ago, well, this
> is it now.
>
> /dev/mapper/10tb           9.1T  8.7T  345G  97% /mnt/10tb
>
>
> The 14TB is supposed to be here this weekend.  I have really got to
> create a solution to this.  My current plan, make the 14TB my backup
> drive.  Put the 10TB in my rig, for now.  It's getting full too.  ROFL
> I could cut off the internet I guess.  ROFLMBO
>
>
> Thoughts on the files open error?  Qbittorrent crashing?  Anyone else
> ran into this before?  Proper solution?  Better solution?
>
>
> Dale
>
> :-)  :-)
>
>
>

[-- Attachment #2: Type: text/html, Size: 2931 bytes --]

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Dale]

Mateusz Okulus wrote:
>> root@fireball / # sysctl fs.file-max
>> fs.file-max = 3289952
>> root@fireball / # sysctl -w fs.file-max=32899520
>> fs.file-max = 32899520
>> root@fireball / # cat /proc/sys/fs/file-max
>> 32899520
> On my pc the limit is 2^63. But this is max files for kernel, which is
> different for max files per user and max files per process.
>
> For example if I run:
>
> $ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
> Limit                     Soft Limit           Hard Limit           Units
> Max open files            524288               524288               files
>
> 524288 = 2^19
>
> QBittorrent increases it's process limit to hard limit. With 10 TB of
> torrents it's hard to tell if it's a bug or just 500k files is not
> enough, I'd say it's probably the latter.
>
> It seems you can change the default hard limit in
> /etc/security/limits.conf
>
> For example:
>
> # 2^24, increased from 2^19
> * hard nofile 16777216
>
> Then after reloging:
>
> $ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
> Limit                     Soft Limit           Hard Limit           Units
> Max open files            16777216             16777216             files
>
> So this works, at least for me.
>
> Check also limits.conf man page.
>
> Regards,
> mmokulus
>
> .
>


It appears the change I made wasn't what was needed.  I took a nap,
after doing today what I used to do on Friday, which involves going to
Doctor, getting shots, then shopping around town etc etc.  Anyway, I
woke up to Qbittorrent being gone.  It crashed at some point.  In the
notification thingy, it had a few of the file open errors again.  So, I
ran some of the commands you posted, my hard limit was set to 4096.  I
whipped out my calculator and doubled it.  I set it in the limits.conf
file like so: 

root@fireball / # cat /etc/security/limits.conf | grep nofile
#        - nofile - max number of open file descriptors
*               hard    nofile          8192
root@fireball / #


I logged out and back in.  I ran the command you shared and it was
indeed set to the new amount.  I'm going to upgrade qbittorrent to the
new version again and test it some more. 

Thanks much for the help.  Maybe this will fix it.

Dale

:-)  :-) 

Re: [gentoo-user] qbittorrent crashes after to many files open error

[mad.scientist.at.large]

I'd suggest looking at the "advanced" section of the preferences.  Lots of way to limit/expand memory use etc.  You really should also check all the other Qbit preferences, there are a lot.  For example you can limit number of I/O threads, block multiple connections from the same host (this prevents some known abuses related to your' client.

--"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas




Sep 8, 2022, 16:24 by rdalek1967@gmail.com:

> Mateusz Okulus wrote:
>
>>> root@fireball / # sysctl fs.file-max
>>> fs.file-max = 3289952
>>> root@fireball / # sysctl -w fs.file-max=32899520
>>> fs.file-max = 32899520
>>> root@fireball / # cat /proc/sys/fs/file-max
>>> 32899520
>>>
>> On my pc the limit is 2^63. But this is max files for kernel, which is
>> different for max files per user and max files per process.
>>
>> For example if I run:
>>
>> $ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
>> Limit                     Soft Limit           Hard Limit           Units
>> Max open files            524288               524288               files
>>
>> 524288 = 2^19
>>
>> QBittorrent increases it's process limit to hard limit. With 10 TB of
>> torrents it's hard to tell if it's a bug or just 500k files is not
>> enough, I'd say it's probably the latter.
>>
>> It seems you can change the default hard limit in
>> /etc/security/limits.conf
>>
>> For example:
>>
>> # 2^24, increased from 2^19
>> * hard nofile 16777216
>>
>> Then after reloging:
>>
>> $ grep 'Limit\|open' /proc/$(pgrep qbittorrent | head -1)/limits
>> Limit                     Soft Limit           Hard Limit           Units
>> Max open files            16777216             16777216             files
>>
>> So this works, at least for me.
>>
>> Check also limits.conf man page.
>>
>> Regards,
>> mmokulus
>>
>> .
>>
>
>
> It appears the change I made wasn't what was needed.  I took a nap,
> after doing today what I used to do on Friday, which involves going to
> Doctor, getting shots, then shopping around town etc etc.  Anyway, I
> woke up to Qbittorrent being gone.  It crashed at some point.  In the
> notification thingy, it had a few of the file open errors again.  So, I
> ran some of the commands you posted, my hard limit was set to 4096.  I
> whipped out my calculator and doubled it.  I set it in the limits.conf
> file like so: 
>
> root@fireball / # cat /etc/security/limits.conf | grep nofile
> #        - nofile - max number of open file descriptors
> *               hard    nofile          8192
> root@fireball / #
>
>
> I logged out and back in.  I ran the command you shared and it was
> indeed set to the new amount.  I'm going to upgrade qbittorrent to the
> new version again and test it some more. 
>
> Thanks much for the help.  Maybe this will fix it.
>
> Dale
>
> :-)  :-) 
>

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Mateusz Okulus]

On 22/09/08 05:24PM, Dale wrote:
> root@fireball / # cat /etc/security/limits.conf | grep nofile
> #        - nofile - max number of open file descriptors
> *               hard    nofile          8192

> I logged out and back in.  I ran the command you shared and it was
> indeed set to the new amount.  I'm going to upgrade qbittorrent to the
> new version again and test it some more. 

I don't think 8k opened files will be enough for 10TB of torrents. Keep
in mind this is more for multi user purpose, in case someone writes and
uses a buggy program, so the whole system won't crash for other users.
If you just have single user computer it's safe to just set it to the
max limit.

I'm certain it's not a bug in qbittorrent, but rather this limit is just
very low by default for you, as I've mentioned for me it was 2^19, and
I'm on 512GB laptop.

Regards,
mmokulus

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Dale]

mad.scientist.at.large@tutanota.com wrote:
> I'd suggest looking at the "advanced" section of the preferences.  Lots of way to limit/expand memory use etc.  You really should also check all the other Qbit preferences, there are a lot.  For example you can limit number of I/O threads, block multiple connections from the same host (this prevents some known abuses related to your' client.
>
> --"Fascism begins the moment a ruling class, fearing the people may use their political democracy to gain economic democracy, begins to destroy political democracy in order to retain its power of exploitation and special privilege." Tommy Douglas
>
>

I've looked in there and even looked at the help pages it links to.  I
think everything there is set as it should, for what I understand
anyway.  I did change a couple really early on but those were changes
others made to make it work better.  I'm not sure they would even be
related to open file things. 

I wish I could recall what I changed now.  :/

Dale

:-)  :-) 

Re: [gentoo-user] qbittorrent crashes after to many files open error

[Dale]

Mateusz Okulus wrote:
> On 22/09/08 05:24PM, Dale wrote:
>> root@fireball / # cat /etc/security/limits.conf | grep nofile
>> #        - nofile - max number of open file descriptors
>> *               hard    nofile          8192
>> I logged out and back in.  I ran the command you shared and it was
>> indeed set to the new amount.  I'm going to upgrade qbittorrent to the
>> new version again and test it some more. 
> I don't think 8k opened files will be enough for 10TB of torrents. Keep
> in mind this is more for multi user purpose, in case someone writes and
> uses a buggy program, so the whole system won't crash for other users.
> If you just have single user computer it's safe to just set it to the
> max limit.
>
> I'm certain it's not a bug in qbittorrent, but rather this limit is just
> very low by default for you, as I've mentioned for me it was 2^19, and
> I'm on 512GB laptop.
>
> Regards,
> mmokulus
>
>


You make a excellent point.  While I have not had this error since the
change I made, I do have large storage media here.  My crypt mount point
is two 8TB hard drives on LVM.  The drive that qbit uses to store files
on is a 6TB drive.  Not including the drive for the OS itself, I have
20TBs of drive space.  Those will be changing soon as I grow things
here.  Since that setting likely affects things not even related to
qbittorrent, these larger storage drives can result in reducing what
file qbit can have open. 

I'm going to up that setting a bit shortly.  I don't think it would hurt
anything given I have lots of memory and a pretty fast CPU so it
shouldn't cause anything to fail due to lack of memory or CPU power.  I
don't see a downside to this change. 

Thanks much for the help. 

Dale

:-)  :-)