mjhjmtu

I'm using pv for sending files via ssh.

I can change "active pv" the limit at under 100M without any problem.
When i set active pv process to 100M or 1G or higher I cant change rate anymore...

BUT! if i change 5-10 times 1M to 2M, 2M to 1M pv can set sometimes to new rate.

I couldn't find any solution for the problem. Any idea?

Examples:

pv -R "15778" -f -F "%p***%t***%e***%r***%b" -L 1M 
pv -R "15778" -f -F "%p***%t***%e***%r***%b" -L 1G 
pv -R "15778" -f -F "%p***%t***%e***%r***%b" -L 1M (not working anymore) 

This is caused by accounting in pv, which effectively means its rate-limiting is read-limited rather than write-limited. Looking at the source code shows that rate-limiting is driven by a “target”, which is the amount remaining to send. If rate-limiting is on, once per rate limit evaluation cycle, the target is increased by however much we’re supposed to send according to the rate limit; the target is then decreased by however much is actually written. This means that if you set the rate limit to a value larger than the actual write capacity, the target will keep going up; reducing the rate limit won’t then have any effect until pv has caught up with its target (including what it’s allowed to write according to the new rate limit).

To see this in action, start a basic pv:

pv /dev/zero /dev/null

Then control that:

pv -R 32605 -L 1M; sleep 10; pv -R 32605 -L 1G; sleep 1; pv -R 32605 -L 1M

You’ll see the impact of the target calculations by varying the duration of the second sleep...

Because of the write limitation, this only causes an issue when you set the rate limit to a value greater than the write capacity.

In a little more detail, here’s how the accounting works with a flow initially limited to 1M, then to 1G for 5s, then back to 1M, on a connection capable of transmitting 400M:

Time Rate Target Sent Remaining
1 1M 1M 1M 0
2 1G 1G 400M 600M
3 1G 1.6G 400M 1.2G
4 1G 2.2G 400M 1.8G
5 1G 2.8G 400M 2.4G
6 1G 3.4G 400M 3G
7 1M 3001M 400M 2601M
8 1M 2602M 400M 2202M
9 1M 2203M 400M 1803M
10 1M 1804M 400M 1404M
11 1M 1405M 400M 1005M
12 1M 1006M 400M 606M
13 1M 607M 400M 207M
14 1M 208M 208M 0
15 1M 1M 1M 0

It takes 7s for the rate limit to be applied again. The longer the time spent with a high rate limit, the longer it takes for the reduced rate limit to be enforced...

The fix for this is quite straightforward, if you can recompile pv: in loop.c, change line 154 to target = (from target +=), resulting in

 || (cur_time.tv_sec == next_ratecheck.tv_sec
 && cur_time.tv_usec >=
 next_ratecheck.tv_usec)) {
 target =
 ((long double) (state->rate_limit)) /
 (long double) (1000000 /
 RATE_GRANULARITY);

Once that’s done, rate limit reductions are applied immediately (well, within one rate-limit cycle).

It seems to be a buffering issue. Here is my test-bed:

pv --pidfile /tmp/pv.pid --rate-limit 1K </dev/zero |
 ssh remote 'cat>/dev/null'

and here is my control:

pv --rate-limit 100M --remote $(cat /tmp/pv.pid)
sleep 1
pv --rate-limit 1K --remote $(cat /tmp/pv.pid)

With a one second interval it takes around 13 seconds for the running pv to reduce from its attempt at 100MB/s (1Gb/s) down to its final target of 1KB/s. Increasing the sleep interval by 1 second increases the time to achieve the final target by almost 10 seconds:

Sleep Delay
 1 13
 2 22
 3 28
 4 37

Four samples isn't really enough for a trend line so I'm going to avoid suggesting it's a linear correlation.

I am correcting myself; pv can change speed..
I don't know why but it need just some time according to your speed limit...
If you set it to 1G, you have to wait 45 sec to reduce speed.

5G - 5 min.

10G - 7 min.

For example:

Commands:

pv --pidfile /tmp/pv.pid --rate-limit 10G </dev/zero | ssh 10.1.1.5 'cat>/dev/null'
pv --rate-limit 1M --remote $(cat /tmp/pv.pid)

-ON 10Gb/s network card:

3.99GiB 0:02:26 [ 157MiB/s] (Right here i just changed to 1M)
26.1GiB 0:02:30 [ 160MiB/s]
77.6GiB 0:09:38 [1.01MiB/s]

After 7 minutes the speed changed finally...

-ON 1Gb/s network card:

I started again with 10G limit.

770MiB 0:00:07 [ 112MiB/s]
44.5GiB 0:06:49 [ 111MiB/s]
46.4GiB 0:07:31 [1.00MiB/s]

Results are the same.
If you change speed 10G to 1M you need to wait 7min at least.
But if you change speed 1M to 10G you don't need to wait any second.
I don't think its just about with buffer because 7min (45Gb) must too big for buffer. but this is just my opinion.