[gentoo-user] I/net server throttling

[gentoo-user] I/net server throttling

[Philip Webb]

In recent weeks, I've seen many instances of downloads of files
coming in small segments, which seem to be the same for each server,
but vary between different servers.  Some Gentoo mirrors suffer this way
-- not all -- & some general news sites, eg when delivering videos,
which run for a few secs, then freeze, then resume etc.
Once I observed a server doing this for several hours (a large file),
then suddenly loosening up & delivering the rest of the file in one go.

Does anyone else have similar experiences or any comments ?

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 986 bytes --]

On Friday 22 Jun 2012 05:41:14 Philip Webb wrote:
> In recent weeks, I've seen many instances of downloads of files
> coming in small segments, which seem to be the same for each server,
> but vary between different servers.  Some Gentoo mirrors suffer this way
> -- not all -- & some general news sites, eg when delivering videos,
> which run for a few secs, then freeze, then resume etc.
> Once I observed a server doing this for several hours (a large file),
> then suddenly loosening up & delivering the rest of the file in one go.
> 
> Does anyone else have similar experiences or any comments ?

Yes, when I use the wireless network at work.  It could be the ISP or it could 
be other users taking up bandwidth.  Really early in the morning performance 
is generally higher than during work hours.

So what I'm saying is that this could be due to contention on the local 
network, or the ISP's pipe.  Do you get such problems off peak hours?
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] I/net server throttling

[Philip Webb]

120622 Mick wrote:
> On Friday 22 Jun 2012 05:41:14 Philip Webb wrote:
>> In recent weeks, I've seen many instances of downloads of files
>> coming in small segments, which seem to be the same for each server,
>> but vary between different servers.  Some Gentoo mirrors suffer this way
>> -- not all -- & some general news sites, eg when delivering videos,
>> which run for a few secs, then freeze, then resume etc.
>> Once I observed a server doing this for several hours (a large file),
>> then suddenly loosening up & delivering the rest of the file in one go.
> Yes, when I use the wireless network at work.
> It could be the ISP or it could be other users taking up bandwidth.
> Really early in the morning performance is generally higher
> than during work hours.  this could be due to contention
> on the local network or the ISP's pipe.
> Do you get such problems off peak hours?

I think you're describing a different phenomenon, traffic jams (smile).
That sometimes happens here & may persist for  1 - 2 weeks ,
but it feels just like driving on the highway when  1  lane is closed.
That probably is some piece of the Internet or ISP under repair or test.

What I'm refering to is getting downloaded files in slices,
eg using Wget, a piece of the file downloads for  22 sec , then stops;
Wget tries again & another  22 sec  piece comes down the pipe, then stops.
This can go on for hours with a big file
& doesn't seem related to the local time of day.
My guess is that the server has been programmed to stop after  22 sec
in an effort to share access among many clients,
but it is irritating & also suggests the server needs faster hardware.

Have others noticed this -- it seems to be a recent innovation --
& is it a known ploy of server managers ?

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[William Kenworthy]

On Fri, 2012-06-22 at 05:10 -0400, Philip Webb wrote:
> 120622 Mick wrote:
> > On Friday 22 Jun 2012 05:41:14 Philip Webb wrote:
> >> In recent weeks, I've seen many instances of downloads of files
> >> coming in small segments, which seem to be the same for each server,
> >> but vary between different servers.  Some Gentoo mirrors suffer this way
> >> -- not all -- & some general news sites, eg when delivering videos,
> >> which run for a few secs, then freeze, then resume etc.
> >> Once I observed a server doing this for several hours (a large file),
> >> then suddenly loosening up & delivering the rest of the file in one go.
> > Yes, when I use the wireless network at work.
> > It could be the ISP or it could be other users taking up bandwidth.
> > Really early in the morning performance is generally higher
> > than during work hours.  this could be due to contention
> > on the local network or the ISP's pipe.
> > Do you get such problems off peak hours?
> 
> I think you're describing a different phenomenon, traffic jams (smile).
> That sometimes happens here & may persist for  1 - 2 weeks ,
> but it feels just like driving on the highway when  1  lane is closed.
> That probably is some piece of the Internet or ISP under repair or test.
> 
> What I'm refering to is getting downloaded files in slices,
> eg using Wget, a piece of the file downloads for  22 sec , then stops;
> Wget tries again & another  22 sec  piece comes down the pipe, then stops.
> This can go on for hours with a big file
> & doesn't seem related to the local time of day.
> My guess is that the server has been programmed to stop after  22 sec
> in an effort to share access among many clients,
> but it is irritating & also suggests the server needs faster hardware.
> 
> Have others noticed this -- it seems to be a recent innovation --
> & is it a known ploy of server managers ?
> 

Are you using traffic shaping like a "police filter", or is there
shaping somewhere in the path? - most of the linux shaping methods work
on a burst principle that produces a given throughput by gating the
traffic for an average throughput.  I have seen the effect you mention
with block transfer protocols (ftp) and wget when trying to use its
built in bandwidth regulation which works similarly to when traffic
shaping is in effect.  There seems to be an interaction between the two
instances of regulating the traffic to create an effect like you have
seen.  This also affects other traffic trying to use the link at the
same time as the available bandwidth gets very "choppy".

I have less experience with other than linux based shaping methods such
as Cisco QoS methods but have not seen the same effect there.

BillK

Re: [gentoo-user] I/net server throttling

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 3613 bytes --]

On Friday 22 Jun 2012 11:29:00 William Kenworthy wrote:
> On Fri, 2012-06-22 at 05:10 -0400, Philip Webb wrote:
> > 120622 Mick wrote:
> > > On Friday 22 Jun 2012 05:41:14 Philip Webb wrote:
> > >> In recent weeks, I've seen many instances of downloads of files
> > >> coming in small segments, which seem to be the same for each server,
> > >> but vary between different servers.  Some Gentoo mirrors suffer this
> > >> way -- not all -- & some general news sites, eg when delivering
> > >> videos, which run for a few secs, then freeze, then resume etc.
> > >> Once I observed a server doing this for several hours (a large file),
> > >> then suddenly loosening up & delivering the rest of the file in one
> > >> go.
> > > 
> > > Yes, when I use the wireless network at work.
> > > It could be the ISP or it could be other users taking up bandwidth.
> > > Really early in the morning performance is generally higher
> > > than during work hours.  this could be due to contention
> > > on the local network or the ISP's pipe.
> > > Do you get such problems off peak hours?
> > 
> > I think you're describing a different phenomenon, traffic jams (smile).
> > That sometimes happens here & may persist for  1 - 2 weeks ,
> > but it feels just like driving on the highway when  1  lane is closed.
> > That probably is some piece of the Internet or ISP under repair or test.
> > 
> > What I'm refering to is getting downloaded files in slices,
> > eg using Wget, a piece of the file downloads for  22 sec , then stops;
> > Wget tries again & another  22 sec  piece comes down the pipe, then
> > stops. This can go on for hours with a big file
> > & doesn't seem related to the local time of day.
> > My guess is that the server has been programmed to stop after  22 sec
> > in an effort to share access among many clients,
> > but it is irritating & also suggests the server needs faster hardware.
> > 
> > Have others noticed this -- it seems to be a recent innovation --
> > & is it a known ploy of server managers ?
> 
> Are you using traffic shaping like a "police filter", or is there
> shaping somewhere in the path? - most of the linux shaping methods work
> on a burst principle that produces a given throughput by gating the
> traffic for an average throughput.  I have seen the effect you mention
> with block transfer protocols (ftp) and wget when trying to use its
> built in bandwidth regulation which works similarly to when traffic
> shaping is in effect.  There seems to be an interaction between the two
> instances of regulating the traffic to create an effect like you have
> seen.  This also affects other traffic trying to use the link at the
> same time as the available bandwidth gets very "choppy".
> 
> I have less experience with other than linux based shaping methods such
> as Cisco QoS methods but have not seen the same effect there.

Now that I understood what Philip was describing I have not seen anything like 
that (yet).  When there is throttling at the server it just shaves off 
anything above a certain transmission rate.  So a bar chart that would 
otherwise show a variable bar height as the client-server try to negotiate 
maximum throughout up to the ADSL max capacity, is now cropped at a lower than 
max throughput.  There is no starting and stopping.

If there is contention on the line then there will be some throughput 
variability, but not really stopping completely and then restarting.  Unless 
perhaps the contention is so abysmal or the server/ISP are under some DDoS 
attack?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] I/net server throttling

[Philip Webb]

Recently, I asked on the list whether anyone had seen this phenomenon :

> What I'm refering to is getting downloaded files in slices,
> eg using Wget, a piece of the file downloads for  22 sec , then stops;
> Wget tries again & another  22 sec  piece comes down the pipe, then
> stops. This can go on for hours with a big file
> & doesn't seem related to the local time of day.
> My guess is that the server has been programmed to stop after  22 sec
> in an effort to share access among many clients,
> but it is irritating & also suggests the server needs faster hardware.
> Have others noticed this -- it seems to be a recent innovation --
> & is it a known ploy of server managers ?

> NB This is not the same as traffic jams.
> That sometimes happens here & may persist for  1 - 2 weeks ,
> but it feels just like driving on the highway when  1  lane is closed.
> That probably is some piece of the Internet or ISP under repair or test.

I now have an explicit example of a server doing this :

  503: hold> wget --no-check-certificate -c http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
  --2012-07-01 15:07:18--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
  Resolving www.math.wisc.edu... 144.92.166.196
  Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
  HTTP request sent, awaiting response... 302 Found
  Location: https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf [following]
  --2012-07-01 15:07:18--  https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
  Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
  WARNING: The certificate of `www.math.wisc.edu' is not trusted.
  WARNING: The certificate of `www.math.wisc.edu' hasn't got a known issuer.
  HTTP request sent, awaiting response... 200 OK
  Length: 24184097 (23M) [application/pdf]
  Saving to: `keislercalc-2-12.pdf'

  6% [==========>                  ] 1,638,400   64.5K/s  eta 38m 5s

NB the word 'throttle' in the 2nd URL.  It has just completed
a 2nd slice downloading  3 %  of the file each time.
Yesterday when I tried, it refused to honor the '-c' flag,
which made downloading impossible, but today it seems to be honoring it.

BTW the Internet is ordinarily responsive at the moment here.

Has anyone else come across this ?  Any thoughts or comments ?

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[Jarry]

On 01-Jul-12 21:20, Philip Webb wrote:

> I now have an explicit example of a server doing this :
>
>    503: hold> wget --no-check-certificate -c http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
>    --2012-07-01 15:07:18--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
>    Resolving www.math.wisc.edu... 144.92.166.196
>    Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
>    HTTP request sent, awaiting response... 302 Found
>    Location: https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf [following]
>    --2012-07-01 15:07:18--  https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
>    Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
>    WARNING: The certificate of `www.math.wisc.edu' is not trusted.
>    WARNING: The certificate of `www.math.wisc.edu' hasn't got a known issuer.
>    HTTP request sent, awaiting response... 200 OK
>    Length: 24184097 (23M) [application/pdf]
>    Saving to: `keislercalc-2-12.pdf'
>
>    6% [==========>                  ] 1,638,400   64.5K/s  eta 38m 5s
>

Everything OK here:

obelix ~ # wget --no-check-certificate -c 
http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
--2012-07-01 19:35:14-- 
http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
Resolving www.math.wisc.edu... 144.92.166.196
Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: 
https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf 
[following]
--2012-07-01 19:35:14-- 
https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24184097 (23M) [application/pdf]
Saving to: `keislercalc-2-12.pdf'

100%[======================================>] 24,184,097  6.88M/s   in 4.4s

2012-07-01 19:35:19 (5.25 MB/s) - `keislercalc-2-12.pdf' saved 
[24184097/24184097]

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] I/net server throttling

[Alan McKinnon]

On Sun, 01 Jul 2012 21:34:23 +0200
Jarry <mr.jarry@gmail.com> wrote:

> On 01-Jul-12 21:20, Philip Webb wrote:
> 
> > I now have an explicit example of a server doing this :
> >
> >    503: hold> wget --no-check-certificate -c
> > http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf --2012-07-01
> > 15:07:18--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> > Resolving www.math.wisc.edu... 144.92.166.196 Connecting to
> > www.math.wisc.edu|144.92.166.196|:80... connected. HTTP request
> > sent, awaiting response... 302 Found Location:
> > https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
> > [following] --2012-07-01 15:07:18--
> > https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
> > Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
> > WARNING: The certificate of `www.math.wisc.edu' is not trusted.
> > WARNING: The certificate of `www.math.wisc.edu' hasn't got a known
> > issuer. HTTP request sent, awaiting response... 200 OK Length:
> > 24184097 (23M) [application/pdf] Saving to: `keislercalc-2-12.pdf'
> >
> >    6% [==========>                  ] 1,638,400   64.5K/s  eta 38m
> > 5s
> >
> 
> Everything OK here:
> 
> obelix ~ # wget --no-check-certificate -c 
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> --2012-07-01 19:35:14-- 
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> Resolving www.math.wisc.edu... 144.92.166.196
> Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location: 
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf 
> [following]
> --2012-07-01 19:35:14-- 
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
> Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 24184097 (23M) [application/pdf]
> Saving to: `keislercalc-2-12.pdf'
> 
> 100%[======================================>] 24,184,097  6.88M/s
> in 4.4s
> 
> 2012-07-01 19:35:19 (5.25 MB/s) - `keislercalc-2-12.pdf' saved 
> [24184097/24184097]
> 
> Jarry
> 

Same for me, download is OK

Maybe Philip has downloaded too much stuff from that site, and hit a
trigger amount that puts him in the category of special users. I have
no idea what wisc.edu's policy is in this regard so can't comment, it's
just an idea



-- 
Alan McKinnnon
alan.mckinnon@gmail.com

Re: [gentoo-user] I/net server throttling

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 4747 bytes --]

On Sunday 01 Jul 2012 20:34:23 Jarry wrote:
> On 01-Jul-12 21:20, Philip Webb wrote:
> > I now have an explicit example of a server doing this :
> >    503: hold> wget --no-check-certificate -c
> >    http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf --2012-07-01
> >    15:07:18--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> >    Resolving www.math.wisc.edu... 144.92.166.196
> >    Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
> >    HTTP request sent, awaiting response... 302 Found
> >    Location:
> >    https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keisler
> >    calc-2-12.pdf [following] --2012-07-01 15:07:18-- 
> >    https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keisler
> >    calc-2-12.pdf Connecting to www.math.wisc.edu|144.92.166.196|:443...
> >    connected. WARNING: The certificate of `www.math.wisc.edu' is not
> >    trusted. WARNING: The certificate of `www.math.wisc.edu' hasn't got a
> >    known issuer. HTTP request sent, awaiting response... 200 OK
> >    Length: 24184097 (23M) [application/pdf]
> >    Saving to: `keislercalc-2-12.pdf'
> >    
> >    6% [==========>                  ] 1,638,400   64.5K/s  eta 38m 5s
> 
> Everything OK here:
> 
> obelix ~ # wget --no-check-certificate -c
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> --2012-07-01 19:35:14--
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> Resolving www.math.wisc.edu... 144.92.166.196
> Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location:
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2
> -12.pdf [following]
> --2012-07-01 19:35:14--
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2
> -12.pdf Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 24184097 (23M) [application/pdf]
> Saving to: `keislercalc-2-12.pdf'
> 
> 100%[======================================>] 24,184,097  6.88M/s   in 4.4s
> 
> 2012-07-01 19:35:19 (5.25 MB/s) - `keislercalc-2-12.pdf' saved
> [24184097/24184097]
> 
> Jarry

Everything OK here too maxing out my ADSL connection (I'm envious of Jarry's 
fibre! :p )

$ wget --no-check-certificate -c 
http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
--2012-07-01 20:54:06--  
http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
Resolving www.math.wisc.edu... 144.92.166.196
Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: 
https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf 
[following]
--2012-07-01 20:54:07--  
https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24184097 (23M) [application/pdf]
Saving to: `keislercalc-2-12.pdf'

100%[======================================>] 24,184,097   821K/s   in 38s     

2012-07-01 20:54:46 (619 KB/s) - `keislercalc-2-12.pdf' saved 
[24184097/24184097]


This pretty good given that I am connecting all the way from ol' Blighty and 
there is some latency and potentially dropped packets:

# mtr -c 3 -s 955 -u -l -r wisc.edu
HOST: dell_xps                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- router                     0.0%     3    4.0   3.9   3.9   4.0   0.0
  2.|-- b.gormless.thn.aa.net.uk   0.0%     3   43.5  44.9  43.5  46.1   1.3
  3.|-- a.aimless.thn.aa.net.uk    0.0%     3   45.6  45.0  43.1  46.3   1.7
  4.|-- 10gigabitethernet1-1.core  0.0%     3   47.7  58.2  45.1  81.9  20.5
  5.|-- 10gigabitethernet7-4.core  0.0%     3  112.5 112.9 112.5 113.7   0.7
  6.|-- 10gigabitethernet8-3.core  0.0%     3  131.9 136.3 131.9 139.8   4.0
  7.|-- ???                       100.0     3    0.0   0.0   0.0   0.0   0.0
  8.|-- r-uwmadison-isp-xe-8-0-1-  0.0%     3  165.7 166.8 161.1 173.6   6.3
    |  `|-- 140.189.8.1
  9.|-- r-peer-xe-1-1-0-945.net.w  0.0%     3  160.0 161.0 160.0 161.9   1.0
 10.|-- r-cssc-b280c-9-core-vlan-  0.0%     3  161.9 160.4 159.5 161.9   1.3
    |  `|-- 146.151.167.85
 11.|-- r-csscplat-b380-1-node-vl  0.0%     3  159.9 160.4 159.6 161.7   1.2
 12.|-- ???                       100.0     3    0.0   0.0   0.0   0.0   0.0


If as I recall you said this happens at all times and it is not related to 
Internet congestion, then all I can suggest is to try using a different ISP's 
network and see if you are getting similar problems.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] I/net server throttling

[Michael Mol]

On Sun, Jul 1, 2012 at 4:12 PM, Mick <michaelkintzios@gmail.com> wrote:
> On Sunday 01 Jul 2012 20:34:23 Jarry wrote:
>> On 01-Jul-12 21:20, Philip Webb wrote:
>> > I now have an explicit example of a server doing this :
>> >    503: hold> wget --no-check-certificate -c
>> >    http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf --2012-07-01
>> >    15:07:18--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
>> >    Resolving www.math.wisc.edu... 144.92.166.196
>> >    Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
>> >    HTTP request sent, awaiting response... 302 Found
>> >    Location:
>> >    https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keisler
>> >    calc-2-12.pdf [following] --2012-07-01 15:07:18--
>> >    https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keisler
>> >    calc-2-12.pdf Connecting to www.math.wisc.edu|144.92.166.196|:443...
>> >    connected. WARNING: The certificate of `www.math.wisc.edu' is not
>> >    trusted. WARNING: The certificate of `www.math.wisc.edu' hasn't got a
>> >    known issuer. HTTP request sent, awaiting response... 200 OK
>> >    Length: 24184097 (23M) [application/pdf]
>> >    Saving to: `keislercalc-2-12.pdf'
>> >
>> >    6% [==========>                  ] 1,638,400   64.5K/s  eta 38m 5s
>>
>> Everything OK here:
>>
>> obelix ~ # wget --no-check-certificate -c
>> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
>> --2012-07-01 19:35:14--
>> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
>> Resolving www.math.wisc.edu... 144.92.166.196
>> Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
>> HTTP request sent, awaiting response... 302 Found
>> Location:
>> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2
>> -12.pdf [following]
>> --2012-07-01 19:35:14--
>> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2
>> -12.pdf Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 24184097 (23M) [application/pdf]
>> Saving to: `keislercalc-2-12.pdf'
>>
>> 100%[======================================>] 24,184,097  6.88M/s   in 4.4s
>>
>> 2012-07-01 19:35:19 (5.25 MB/s) - `keislercalc-2-12.pdf' saved
>> [24184097/24184097]
>>
>> Jarry
>
> Everything OK here too maxing out my ADSL connection (I'm envious of Jarry's
> fibre! :p )
>
> $ wget --no-check-certificate -c
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> --2012-07-01 20:54:06--
> http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
> Resolving www.math.wisc.edu... 144.92.166.196
> Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location:
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
> [following]
> --2012-07-01 20:54:07--
> https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
> Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 24184097 (23M) [application/pdf]
> Saving to: `keislercalc-2-12.pdf'
>
> 100%[======================================>] 24,184,097   821K/s   in 38s
>
> 2012-07-01 20:54:46 (619 KB/s) - `keislercalc-2-12.pdf' saved
> [24184097/24184097]
>
>
> This pretty good given that I am connecting all the way from ol' Blighty and
> there is some latency and potentially dropped packets:
>
> # mtr -c 3 -s 955 -u -l -r wisc.edu
> HOST: dell_xps                    Loss%   Snt   Last   Avg  Best  Wrst StDev
>   1.|-- router                     0.0%     3    4.0 3.9 3.9 4.0 0.0
>   2.|-- b.gormless.thn.aa.net.uk   0.0%     3   43.5  44.9  43.5  46.1   1.3
>   3.|-- a.aimless.thn.aa.net.uk    0.0%     3   45.6  45.0  43.1  46.3   1.7
>   4.|-- 10gigabitethernet1-1.core  0.0%     3   47.7  58.2  45.1  81.9  20.5
>   5.|-- 10gigabitethernet7-4.core  0.0%     3  112.5 112.9 112.5 113.7   0.7
>   6.|-- 10gigabitethernet8-3.core  0.0%     3  131.9 136.3 131.9 139.8   4.0
>   7.|-- ???                       100.0     3    0.0   0.0   0.0   0.0   0.0
>   8.|-- r-uwmadison-isp-xe-8-0-1-  0.0%     3  165.7 166.8 161.1 173.6   6.3
>     |  `|-- 140.189.8.1
>   9.|-- r-peer-xe-1-1-0-945.net.w  0.0%     3  160.0 161.0 160.0 161.9   1.0
>  10.|-- r-cssc-b280c-9-core-vlan-  0.0%     3  161.9 160.4 159.5 161.9   1.3
>     |  `|-- 146.151.167.85
>  11.|-- r-csscplat-b380-1-node-vl  0.0%     3  159.9 160.4 159.6 161.7   1.2
>  12.|-- ???                       100.0     3    0.0   0.0   0.0   0.0   0.0
>
>
> If as I recall you said this happens at all times and it is not related to
> Internet congestion, then all I can suggest is to try using a different ISP's
> network and see if you are getting similar problems.

It could be a connection duration limit. Either the web server or an
intermediate proxy server may place a limit on how long a connection
may be open. If you have a low-bandwidth pipe, you'd be more
vulnerable to hitting that limit than someone with a high-bandwidth
pipe.

Possible sources off the top of my head: PHP scripts' script time
limits. squid connection time limits.

Try launching two simultaneous wget processes, and see if they both
fail out around the same time.



-- 
:wq

Re: [gentoo-user] I/net server throttling

[Philip Webb]

120701 Michael Mol wrote:
> It could be a connection duration limit.
> Either the web server or an intermediate proxy server
> may place a limit on how long a connection may be open.
> If you have a low-bandwidth pipe, you'd be more vulnerable
> to hitting that limit than someone with a high-bandwidth pipe.
> Possible sources off the top of my head:
> PHP scripts' script time limits, squid connection time limits.

Yes, the limit is one of time, not of number of bytes transmitted.
My earlier connection stayed open for  32 s  & got  819 K ;
one of the other repliers -- thanks to all -- got  38 s  & the whole file.

So  2  conclusions for me : (1) yes, servers do impose time-slices ;
(2) my basic problem remains the very low bandwidth I'm getting,
which I have to take up with my ISP once I've clarified other aspects.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[Michael Mol]

On Sun, Jul 1, 2012 at 8:26 PM, Philip Webb <purslow@ca.inter.net> wrote:
> 120701 Michael Mol wrote:
>> It could be a connection duration limit.
>> Either the web server or an intermediate proxy server
>> may place a limit on how long a connection may be open.
>> If you have a low-bandwidth pipe, you'd be more vulnerable
>> to hitting that limit than someone with a high-bandwidth pipe.
>> Possible sources off the top of my head:
>> PHP scripts' script time limits, squid connection time limits.
>
> Yes, the limit is one of time, not of number of bytes transmitted.
> My earlier connection stayed open for  32 s  & got  819 K ;
> one of the other repliers -- thanks to all -- got  38 s  & the whole file.
>
> So  2  conclusions for me : (1) yes, servers do impose time-slices ;
> (2) my basic problem remains the very low bandwidth I'm getting,
> which I have to take up with my ISP once I've clarified other aspects.

There's probably nothing your ISP can do about it, unless you're
talking about upgrading your service.

One thing you might be able to do is pay $5/mo or so for a Linux VM at
some VPS provider, install and configure Squid, and bounce your own
traffic off of it. Squid will pull down the file faster than you, and
won't impose a connection time limit on you. (Unless you configure it
to do so...)

If you do something like that, be sure to properly secure it.

-- 
:wq

Re: [gentoo-user] I/net server throttling

[Adam Carter]

> One thing you might be able to do is pay $5/mo or so for a Linux VM at
> some VPS provider, install and configure Squid, and bounce your own
> traffic off of it. Squid will pull down the file faster than you, and
> won't impose a connection time limit on you. (Unless you configure it
> to do so...)

And you could connect to that squid using an ssh tunnel with
compression enabled at its maximum level. Otherwise the gzip ecap
module https://code.google.com/p/squid-ecap-gzip/ may help a little
but IIRC it only compresses text eg plain, html etc.

Re: [gentoo-user] I/net server throttling

[Philip Webb]

120701 Michael Mol wrote:
> On Sun, Jul 1, 2012 at 8:26 PM, Philip Webb <purslow@ca.inter.net> wrote:
>> So  2  conclusions for me : (1) yes, servers do impose time-slices ;
>> (2) my basic problem remains the very low bandwidth I'm getting,
>> which I have to take up with my ISP once I've clarified other aspects.
> There's probably nothing your ISP can do about it,
> unless you're talking about upgrading your service.

The ISP doesn't own the physical wiring, recently upgraded to fibre.
A friend on the opposite side of town using the same ISP
gets  10 times  the speed I get (  5 Mb/s  a/a  0,5 Mb/s ),
while the man next-door with the same wiring coming into the building
but using a different ISP gets  12 Mb/s .
My speed was cut in half  c 12 mth ago  for an unknown reason.
This is downtown Toronto, where I should get a good speed,
so I suspect either some accidental misconfiguration or dirty tricks.

> One thing you might be able to do is pay $5/mo or so
> for a Linux VM at some VPS provider, install and configure Squid
> and bounce your own traffic off of it.
> Squid will pull down the file faster than you
> and won't impose a connection time limit on you.
> If you do something like that, be sure to properly secure it.

Well, the simpler alternative in my case wb to use the UoT service.
I can 'ssh' into a CLI on the CHASS machine, which runs Irix,
then use 'wget' from there with a very fast connection.
Downloading a file from there to here would remain fairly slow,
but it wouldn't be subject to any throttling or slicing.
I tend to forget that resource, however (smile).

Thanks again for the advice & anyone's further thoughts.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[Philip Webb]

120701 Philip Webb wrote:
> Well, the simpler alternative in my case wb to use the UoT service.
> I can 'ssh' into a CLI on the CHASS machine, which runs Irix,
> then use 'wget' from there with a very fast connection.
> Downloading a file from there to here would remain fairly slow,
> but it wouldn't be subject to any throttling or slicing.

I just did just that.  CHASS has a  5 Gb/s  Internet connection,
so the whole file ( 25 MB ) arrived before the boom fell after  c 39 s .
They don't have Wget (!), but Lynx does the job equally well.
Then I used Fuse to mount the remote dir locally
& downloaded the file to my machine at  c 75 KB/s  in  c 5 min ;
I could have used Krusader instead & got a pretty progress box.

BTW the file is a book mentioned by Carla Schroeder yesterday.
I don't really need it, but it looked interesting.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-user] I/net server throttling

[Mick]

On 2 July 2012 05:41, Philip Webb <purslow@ca.inter.net> wrote:
> 120701 Philip Webb wrote:
>> Well, the simpler alternative in my case wb to use the UoT service.
>> I can 'ssh' into a CLI on the CHASS machine, which runs Irix,
>> then use 'wget' from there with a very fast connection.
>> Downloading a file from there to here would remain fairly slow,
>> but it wouldn't be subject to any throttling or slicing.
>
> I just did just that.  CHASS has a  5 Gb/s  Internet connection,
> so the whole file ( 25 MB ) arrived before the boom fell after  c 39 s .
> They don't have Wget (!), but Lynx does the job equally well.
> Then I used Fuse to mount the remote dir locally
> & downloaded the file to my machine at  c 75 KB/s  in  c 5 min ;
> I could have used Krusader instead & got a pretty progress box.

Look at this:

$ wget --no-check-certificate -c
http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
--2012-07-02 11:17:01--  http://www.math.wisc.edu/~keisler/keislercalc-2-12.pdf
Resolving www.math.wisc.edu... 144.92.166.196
Connecting to www.math.wisc.edu|144.92.166.196|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
[following]
--2012-07-02 11:17:01--
https://www.math.wisc.edu/formMail/throttle.php?URL=/~keisler/keislercalc-2-12.pdf
Connecting to www.math.wisc.edu|144.92.166.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24184097 (23M) [application/pdf]
Saving to: `keislercalc-2-12.pdf'

100%[=====================================>] 24,184,097   130K/s   in 2m 50s

2012-07-02 11:19:52 (139 KB/s) - `keislercalc-2-12.pdf' saved
[24184097/24184097]


I managed to download it using a slow connection over 2m 50s, while
the connection was completely uninterrupted.  So the 38s threshold
does not seem true, unless ...

I downloaded this behind a corporate gateway, so I don't know if the
slowness of the connection is only up to my gateway, rather than
between the gateway and the Uni server.


If you prefer to use a proxy anyway, you could set up a SOCKS 5 proxy
connection via ssh to the proxy server in question like so:

ssh -ND 12500 user@proxy_server  (you don't have to use port 12500,
this is just an example)

then either set up the SOCKS 5 proxy in a browser/ftp client and use
that, or use proxychains.  Running:

 proxychains kdeinit4

should allow you to run Krusader to download the file without even
having to restart the Krusader application or set up its proxy
configuration.
-- 
Regards,
Mick