[gentoo-user] youtube-dl has been very slow

[gentoo-user] youtube-dl has been very slow

[John Covici]

Hi.  If I try to play a youtube video using a console app like mpv, or
if I try to download a youtube video directly by using youtube-dl, the
speed is about dialup speed.  Now, I have a Winblows box in the same
network and I don't get the slowdown.  I have a program there which
downloads the same exact youtube video quite rapidly.  Is youtube-dl
doing something wrong maybe by having a wrong user agent, or is there
something else to be done?  My version is 2021.06.06.  This seems
quite old to me for this program.

Thanks in advance for any suggestions.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] youtube-dl has been very slow

[tastytea]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

On 2021-11-06 16:32-0400 John Covici <covici@ccs.covici.com> wrote:

> Hi.  If I try to play a youtube video using a console app like mpv, or
> if I try to download a youtube video directly by using youtube-dl, the
> speed is about dialup speed.  Now, I have a Winblows box in the same
> network and I don't get the slowdown.  I have a program there which
> downloads the same exact youtube video quite rapidly.  Is youtube-dl
> doing something wrong maybe by having a wrong user agent, or is there
> something else to be done?  My version is 2021.06.06.  This seems
> quite old to me for this program.
> 
> Thanks in advance for any suggestions.
> 

Google changed something to make youtube-dl slow and its development
seems to have stopped.

The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It
downloads with normal speeds. The newest version of the ebuild installs
a compatibility-wrapper so programs calling youtube-dl will
automatically use yt-dlp instead.

Kind regards, tastytea

-- 
Get my PGP key with `gpg --locate-keys tastytea@tastytea.de` or at
<https://tastytea.de/tastytea.asc>.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

Am Sat, Nov 06, 2021 at 09:51:47PM +0100 schrieb tastytea:
> On 2021-11-06 16:32-0400 John Covici <covici@ccs.covici.com> wrote:
> 
> > Hi.  If I try to play a youtube video using a console app like mpv, or
> > if I try to download a youtube video directly by using youtube-dl, the
> > speed is about dialup speed.

I’ve also been observing this for a while now. Since I’m patient, I couldn’t
bother yet to dig deeper. I checked the user agent and it seems
inconspicuous. Maybe they detect it through the way youtube-dl first loads
the video’s manifest, or because it’s missing a cookie or something.

> > Now, I have a Winblows box in the same
> > network and I don't get the slowdown.

Watching in the browser is also no problem here. Back in the very old days,
I opened the video in the browser, let it load and then copied the actual
video file from the adobe flash cache directory. :D


> > Is youtube-dl doing something wrong maybe by having a wrong user agent,
> > or is there something else to be done?  My version is 2021.06.06.  This
> > seems quite old to me for this program.

Same here on Arch.

> The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It downloads
> with normal speeds.

Thanks for the tip, I installed it right away and will try to change my
muscle memory. Incidentally, I set up a bash alias ,y so I could type less
to call youtube-dl, but then I removed it because too often I was at a
system where the alias wasn’t present. Now I would have saved me the pain of
change.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Where there’s a villa, there’s a way.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[John Covici]

On Sat, 06 Nov 2021 16:51:47 -0400,
tastytea wrote:
> 
> [1  <text/plain; US-ASCII (quoted-printable)>]
> On 2021-11-06 16:32-0400 John Covici <covici@ccs.covici.com> wrote:
> 
> > Hi.  If I try to play a youtube video using a console app like mpv, or
> > if I try to download a youtube video directly by using youtube-dl, the
> > speed is about dialup speed.  Now, I have a Winblows box in the same
> > network and I don't get the slowdown.  I have a program there which
> > downloads the same exact youtube video quite rapidly.  Is youtube-dl
> > doing something wrong maybe by having a wrong user agent, or is there
> > something else to be done?  My version is 2021.06.06.  This seems
> > quite old to me for this program.
> > 
> > Thanks in advance for any suggestions.
> > 
> 
> Google changed something to make youtube-dl slow and its development
> seems to have stopped.
> 
> The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It
> downloads with normal speeds. The newest version of the ebuild installs
> a compatibility-wrapper so programs calling youtube-dl will
> automatically use yt-dlp instead.


Thanks much, I will try this out.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici wb2una
         covici@ccs.covici.com

Re: [gentoo-user] youtube-dl has been very slow

[Matt Connell (Gmail)]

On Sat, 2021-11-06 at 21:51 +0100, tastytea wrote:
> The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It
> downloads with normal speeds.

I'm a bit surprised such a venerable piece of software has been
seemingly-abandoned so abruptly.  I'm also upset that I had missed the
memo on this until now... guess I need to expand my news net a bit.

What was the reasoning/motivation behind the fork?  I see the yt-dl
subreddit recommending yt-dlp as you have here, but not explanation
other than the throttling being experienced and described here.

[gentoo-user] Re: youtube-dl has been very slow

[Nikos Chantziaras]

On 08/11/2021 17:57, Matt Connell (Gmail) wrote:
> What was the reasoning/motivation behind the fork?  I see the yt-dl
> subreddit recommending yt-dlp as you have here, but not explanation
> other than the throttling being experienced and described here.

It was a fork that added things youtube-dl didn't want to add. They are 
described on their github page:

   https://github.com/yt-dlp/yt-dlp

One thing they added in particular that I find great is the ability to 
specify which codec to prefer. I want AV1, which is the highest quality 
codec on youtube. Before, I had to do this:

--format=bestvideo[vcodec^=av01][width>?2500][width<=?2560]+bestaudio/bestvideo[ext=webm][width>?2500][width<=?2560]+bestaudio/bestvideo[ext=mp4][width>?2500][width<=?2560]+bestaudio/bestvideo[vcodec^=av01][width>?1800][width<=?1920]+bestaudio/bestvideo[ext=webm][width>?1800][width<=?1920]+bestaudio/bestvideo[ext=mp4][width>?1800][width<=?1920]+bestaudio/bestvideo[vcodec^=av01][width>?1100][width<=?1280]+bestaudio/bestvideo[ext=webm][width>?1100][width<=?1280]+bestaudio/bestvideo[ext=mp4][width>?1100][width<=?1280]+bestaudio/bestvideo[vcodec^=av01][width>?800][width<=?1024]+bestaudio/bestvideo[ext=webm][width>?800][width<=?1024]+bestaudio/bestvideo[ext=mp4][width>?800][width<=?1024]+bestaudio/bestvideo[vcodec^=av01][width>?600][width<=?800]+bestaudio/bestvideo[ext=webm][width>?600][width<=?800]+bestaudio/bestvideo[ext=mp4][width>?600][width<=?800]+bestaudio/bestvideo[vcodec^=av01][width>?400][width<=?580]+bestaudio/bestvideo[ext=webm][width>?400][width<=?580]+bestaudio/bestvideo[ext=mp4][width>?400][width<=?580]+bestaudio/bestvideo[vcodec^=av01][width>?200][width<=?400]+bestaudio/bestvideo[ext=webm][width>?200][width<=?400]+bestaudio/bestvideo[ext=mp4][width>?200][width<=?400]+bestaudio/best

But with yt-dlp I only have to do this:

   --format-sort=width:2560,height:1440,vcodec:av1

If that's not a win, I don't what is :)

Re: [gentoo-user] youtube-dl has been very slow

[Skippy]

On 11/6/21 14:51, tastytea wrote:
snip

> 
> The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It
> downloads with normal speeds. The newest version of the ebuild installs
> a compatibility-wrapper so programs calling youtube-dl will
> automatically use yt-dlp instead.
> 
> Kind regards, tastytea
> 

I didn't know abouot yt-dlp either. Thanks for the info.  I installed it 
but it will not work with gPodder.

When in gPodder i enable use of Youtube-dl I get the message

Extension cannot be activated.
Python module not found: youtube_dl

Anyone know how to make gPodder use yt-dlp?

Do I simply need a link that sends youtube_dl to yt-dlp?

Thank you.

Re: [gentoo-user] youtube-dl has been very slow

[tastytea]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On 2021-11-15 08:33-0700 Skippy <linuxgn2@204EastSouth.com> wrote:

> On 11/6/21 14:51, tastytea wrote:
> snip
> 
> > 
> > The fork everyone is using now is yt-dlp (net-misc/yt-dlp). It
> > downloads with normal speeds. The newest version of the ebuild
> > installs a compatibility-wrapper so programs calling youtube-dl will
> > automatically use yt-dlp instead.
> > 
> > Kind regards, tastytea
> >   
> 
> I didn't know abouot yt-dlp either. Thanks for the info.  I installed
> it but it will not work with gPodder.
> 
> When in gPodder i enable use of Youtube-dl I get the message
> 
> Extension cannot be activated.
> Python module not found: youtube_dl

That looks like it does not use the program, but the python module
directly. You could try if symlinking
/usr/lib/python3.9/site-packages/yt_dlp to
/usr/lib/python3.9/site-packages/youtube_dl helps. Replace python3.9
with your version of course.
If that works, please file a bug on bugs.gentoo.org.

> Anyone know how to make gPodder use yt-dlp?
> 
> Do I simply need a link that sends youtube_dl to yt-dlp?

yt-dlp already installs a wrapper script in /usr/bin/youtube-dl.

Kind regards, tastytea

-- 
Get my PGP key with `gpg --locate-keys tastytea@tastytea.de` or at
<https://tastytea.de/tastytea.asc>.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[Skippy]

On 11/15/21 09:14, tastytea wrote:

snip

>>
>> When in gPodder i enable use of Youtube-dl I get the message
>>
>> Extension cannot be activated.
>> Python module not found: youtube_dl
> 
> That looks like it does not use the program, but the python module
> directly. You could try if symlinking
> /usr/lib/python3.9/site-packages/yt_dlp to
> /usr/lib/python3.9/site-packages/youtube_dl helps. Replace python3.9
> with your version of course.
> If that works, please file a bug on bugs.gentoo.org.
> 
>> Anyone know how to make gPodder use yt-dlp?
>>
>> Do I simply need a link that sends youtube_dl to yt-dlp?
> 
> yt-dlp already installs a wrapper script in /usr/bin/youtube-dl.
> 
> Kind regards, tastytea
> 

I created the symlink as you instructed and it appears to be working. 
When I enable use of youtube-dl in gPodder I get no error messages.  I 
just added a youtube feed and am downloading some videos.

Thanks for the assistance.

In regards to filing a bug - I'm unclear exactly what the bug is.  Is 
the bug the fact that gPodder will not use yt-dlp in place of youtube-dl 
without your work-around?

Re: [gentoo-user] youtube-dl has been very slow

[tastytea]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

On 2021-11-15 11:42-0700 Skippy <linuxgn2@204EastSouth.com> wrote:

> On 11/15/21 09:14, tastytea wrote:
> 
> snip
> 
> >>
> >> When in gPodder i enable use of Youtube-dl I get the message
> >>
> >> Extension cannot be activated.
> >> Python module not found: youtube_dl  
> > 
> > That looks like it does not use the program, but the python module
> > directly. You could try if symlinking
> > /usr/lib/python3.9/site-packages/yt_dlp to
> > /usr/lib/python3.9/site-packages/youtube_dl helps. Replace python3.9
> > with your version of course.
> > If that works, please file a bug on bugs.gentoo.org.
> >   
> >> Anyone know how to make gPodder use yt-dlp?
> >>
> >> Do I simply need a link that sends youtube_dl to yt-dlp?  
> > 
> > yt-dlp already installs a wrapper script in /usr/bin/youtube-dl.
> > 
> > Kind regards, tastytea
> >   
> 
> I created the symlink as you instructed and it appears to be working. 
> When I enable use of youtube-dl in gPodder I get no error messages.
> I just added a youtube feed and am downloading some videos.
> 
> Thanks for the assistance.
> 
> In regards to filing a bug - I'm unclear exactly what the bug is.  Is 
> the bug the fact that gPodder will not use yt-dlp in place of
> youtube-dl without your work-around?

It's not really a bug, more a workaround for projects that haven't
switched yet. I thought it makes sense to let the ebuild add the
symlink since yt-dlp and youtube-dl can't be installed at the same time.

But ionen mentioned in the other reply¹, that

> While the "youtube-dl" command from the yt-dlp ebuild is mostly
> compatible, the python module can need small workarounds so opted
> not to symlink the module in the ebuild.

So it seems to be not such a good idea. 😊

¹ <https://archives.gentoo.org/gentoo-user/message/c76c383b9ff73e7c9cd48f76bad4f752>

-- 
Get my PGP key with `gpg --locate-keys tastytea@tastytea.de` or at
<https://tastytea.de/tastytea.asc>.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[Rich Freeman]

On Mon, Nov 15, 2021 at 1:42 PM Skippy <linuxgn2@204eastsouth.com> wrote:
>
> I created the symlink as you instructed and it appears to be working.
> When I enable use of youtube-dl in gPodder I get no error messages.  I
> just added a youtube feed and am downloading some videos.
>

Keep in mind that some APIs might break and others might not, so
whether there is a problem is going to depend on a lot of details.
One program might work with this workaround, and another might fail in
subtle ways.  So, it isn't a good idea to just symlink this for all
users.

However, as you seem to have found, it may work in some cases, and if
it does that is great.  I imagine you aren't using much software that
makes use of the python APIs, so the risks are fairly low if you are
happy with what you found in testing.

> In regards to filing a bug - I'm unclear exactly what the bug is.  Is
> the bug the fact that gPodder will not use yt-dlp in place of youtube-dl
> without your work-around?

Correct.  The fact that you didn't run into issues will probably be
useful feedback as well.

-- 
Rich

Re: [gentoo-user] youtube-dl has been very slow

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

On Mon, Nov 15, 2021 at 11:42:28AM -0700, Skippy wrote:
> On 11/15/21 09:14, tastytea wrote:
> 
> snip
> 
> >>
> >> When in gPodder i enable use of Youtube-dl I get the message
> >>
> >> Extension cannot be activated.
> >> Python module not found: youtube_dl
> > 
> > That looks like it does not use the program, but the python module
> > directly. You could try if symlinking
> > /usr/lib/python3.9/site-packages/yt_dlp to
> > /usr/lib/python3.9/site-packages/youtube_dl helps. Replace python3.9
> > with your version of course.
> > If that works, please file a bug on bugs.gentoo.org.
> > 
> >> Anyone know how to make gPodder use yt-dlp?
> >>
> >> Do I simply need a link that sends youtube_dl to yt-dlp?
> > 
> > yt-dlp already installs a wrapper script in /usr/bin/youtube-dl.
> > 
> > Kind regards, tastytea
> > 
> 
> I created the symlink as you instructed and it appears to be working. 
> When I enable use of youtube-dl in gPodder I get no error messages.  I 
> just added a youtube feed and am downloading some videos.
> 
> Thanks for the assistance.
> 

Hm, slightly surprised it works. I've had a poorer experience with
symlinking the python module (not fully compatible). I considered
having the ebuild do it but ultimately skipped that.

However, if you say it works I think I'll have a 2nd look a bit later.
It's fairly harmless to add either way.

(I picked up maintaining yt-dlp in ::gentoo recently)

> In regards to filing a bug - I'm unclear exactly what the bug is.  Is 
> the bug the fact that gPodder will not use yt-dlp in place of youtube-dl 
> without your work-around?
> 

gpodder's upstream already has a bug report to use yt-dlp

Mentioned bugs.gentoo.org, so imagine the idea was that gentoo's ebuild
could add the symlink. I got the message anyhow so can skip the bug.

-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[Skippy]

I've been short on time, but just wanted to jump in and say that it 
still seems to be working fine. If anything blows up I'll post on this 
thread.

Thanks again to everyone for the assistance.

Re: [gentoo-user] youtube-dl has been very slow

[Skippy]

On 11/15/21 12:02, Ionen Wolkens wrote:
> On Mon, Nov 15, 2021 at 11:42:28AM -0700, Skippy wrote:
>> On 11/15/21 09:14, tastytea wrote:
>>
>> snip
>>
>>>>
>>>> When in gPodder i enable use of Youtube-dl I get the message
>>>>
>>>> Extension cannot be activated.
>>>> Python module not found: youtube_dl
>>>
>>> That looks like it does not use the program, but the python module
>>> directly. You could try if symlinking
>>> /usr/lib/python3.9/site-packages/yt_dlp to
>>> /usr/lib/python3.9/site-packages/youtube_dl helps. Replace python3.9
>>> with your version of course.
>>> If that works, please file a bug on bugs.gentoo.org.
>>>
>>>> Anyone know how to make gPodder use yt-dlp?
>>>>
>>>> Do I simply need a link that sends youtube_dl to yt-dlp?
>>>
>>> yt-dlp already installs a wrapper script in /usr/bin/youtube-dl.
>>>
>>> Kind regards, tastytea
>>>
>>
>> I created the symlink as you instructed and it appears to be working.
>> When I enable use of youtube-dl in gPodder I get no error messages.  I
>> just added a youtube feed and am downloading some videos.
>>
>> Thanks for the assistance.
>>
> 
> Hm, slightly surprised it works. I've had a poorer experience with
> symlinking the python module (not fully compatible). I considered
> having the ebuild do it but ultimately skipped that.
> 
> However, if you say it works I think I'll have a 2nd look a bit later.
> It's fairly harmless to add either way.
> 
> (I picked up maintaining yt-dlp in ::gentoo recently)
> 
>> In regards to filing a bug - I'm unclear exactly what the bug is.  Is
>> the bug the fact that gPodder will not use yt-dlp in place of youtube-dl
>> without your work-around?
>>
> 
> gpodder's upstream already has a bug report to use yt-dlp
> 
> Mentioned bugs.gentoo.org, so imagine the idea was that gentoo's ebuild
> could add the symlink. I got the message anyhow so can skip the bug.
> 

Update on what's happening with this - now it's not behaving nicely. 
I'm not sure what has changed.


With use Youtube-dl enabled updating the feeds is very slow.

I get this error message under each non-YouTube feed in the main window:

ERROR: 'str' object has no attriubte 'suitable'

And I get this for each feed in my log file:

2021-11-22 07:58:58,085 [gpodder.gtkui.main] ERROR: Error: 'str' object 
has no attribute 'suitable'
Traceback (most recent call last):
   File "/usr/lib/python3.9/site-packages/gpodder/gtkui/main.py", line 
2671, in update_feed_cache_proc
     channel.update(max_episodes=self.config.max_episodes_per_feed)
   File "/usr/lib/python3.9/site-packages/gpodder/model.py", line 1166, 
in update
     self._consume_updated_feed(result.feed, max_episodes)
   File "/usr/lib/python3.9/site-packages/gpodder/model.py", line 1076, 
in _consume_updated_feed
     new_episodes, seen_guids = feed.get_new_episodes(self, existing_guids)
   File "/usr/lib/python3.9/site-packages/gpodder/model.py", line 155, 
in get_new_episodes
     episode = channel.EpisodeClass.from_podcastparser_entry(entry, channel)
   File "/usr/lib/python3.9/site-packages/gpodder/model.py", line 298, 
in from_podcastparser_entry
     link_has_media = registry.custom_downloader.resolve(None, None, 
episode) is not None
   File "/usr/lib/python3.9/site-packages/gpodder/registry.py", line 31, 
in resolve
     result = resolver(item, *args)
   File "/usr/share/gpodder/extensions/youtube-dl.py", line 443, in 
custom_downloader
     if self.is_supported_url(episode.url):
   File "/usr/share/gpodder/extensions/youtube-dl.py", line 432, in 
is_supported_url
     if ie.suitable(url) and ie.ie_key() not in self.ie_blacklist:
AttributeError: 'str' object has no attribute 'suitable'
2021-11-22 07:59:40,788 [urllib3.connectionpool] WARNING: Retrying 
(Retry(total=2, connect=None, read=None, redirect=None, status=None)) 
after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in 
violation of protocol (_ssl.c:1145)'))': /rss2.xml


All non YouTube files fail to download.  For each file I get the error:
ERROR: 'str' object has no attriubte 'suitable'

When I disable using Youtube-dl the non-YouTube files download without 
problem.

The work around is simply to turn off 'use Youtube-dl' to download podcasts.

I just wanted to get this information out to you.

Re: [gentoo-user] youtube-dl has been very slow

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Mon, Nov 22, 2021 at 11:04:45AM -0700, Skippy wrote:
> Update on what's happening with this - now it's not behaving nicely. 
> I'm not sure what has changed.
> 
> 
> With use Youtube-dl enabled updating the feeds is very slow.
> 
> I get this error message under each non-YouTube feed in the main window:
> 
> ERROR: 'str' object has no attriubte 'suitable'
> 
> [snip]

Thanks, looks like this was mentioned upstream:
https://github.com/gpodder/gpodder/issues/1176#issuecomment-949674623

Will have to wait for them to work on this.

Guess site-packages symlink will have limited uses but given the yt-dlp
ebuild blocks youtube-dl either way (opted for no USE for the wrapper
script given ultimately want to get rid of youtube-dl), it should
hopefully not do more harm than good for the few cases where it can
work.

-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-user] youtube-dl has been very slow

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Mon, Nov 15, 2021 at 08:33:04AM -0700, Skippy wrote:
> I didn't know abouot yt-dlp either. Thanks for the info.  I installed it 
> but it will not work with gPodder.
> 
> When in gPodder i enable use of Youtube-dl I get the message
> 
> Extension cannot be activated.
> Python module not found: youtube_dl
> 
> Anyone know how to make gPodder use yt-dlp?
> 
> Do I simply need a link that sends youtube_dl to yt-dlp?
> 
> Thank you.

Haven't tried but see upstream issue, there's a WIP PR too
https://github.com/gpodder/gpodder/issues/1176
https://github.com/gpodder/gpodder/pull/1182

While the "youtube-dl" command from the yt-dlp ebuild is mostly
compatible, the python module can need small workarounds so opted
not to symlink the module in the ebuild.

That aside, been debating whether I should allow youtube-dl and yt-dlp
to be installed concurrently meanwhile (IUSE=youtube-dl on yt-dlp),
although at this point youtube-dl feels nearly unusable to me so
not sure if it's worth it :?
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]