mjhjmtu

I am gathering some data locally at home on a Raspberry Pi and I want to send the data as fast as possible to a REST API (which I own) on a server on the web.

The locally gathered data could flow as fast as 100 records per second.

If I execute a curl command in a loop it will send the data... Wait for the 200 response then process the next record... Much slower than my internal data flow.

I have found some hints here on Stackoverflow and tried to adapt them but it wouldn't send the curl commands in parallel.

I know my code is not the prettiest (particularly about the usage of the mycurl function ) and I am ready for suggestions

#!/bin/bash

host="localhost"
port="********"

mycurl() 
 data="field1=$1&field2=$2&field3=$3&field4=$4&field5=$5&field6=$6&field7=$7&field8=$8&field9=$9&field10=$10"
 curl --output /dev/null -d $data --silent -X POST https://myapi/myendpoint;

export -f mycurl

#----------------------LOOP starts------------------------
while true; 
do 

 nc -d $host $port | while IFS="," read -r f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 f15 f16 f17 f18 f19 f20 f21 f22
 do 
 if [ "$f15" != "" ]; then 
 seq 1000 | parallel --no-notice --joblog log -j0 mycurl ::: $f5 ::: $f7 ::: $f8 ::: $f15 ::: $f16 ::: $17 ::: $18 ::: $19 ::: $20 ::: $21;
 fi
 done

done

GNU Parallel has an overhead of 2-5 ms per job which is getting close to your 100 entries per second. To mitigate that you can wrap it with parallel --pipe. This should run around 25000 jobs in parallel per second on a 100 core machine:

#!/bin/bash

host="localhost"
port="********"

mycurl() 
 if [ "$15" != "" ]; then
 # The numbering here should probably be adjusted
 data="field1=$1&field2=$2&field3=$3&field4=$4&field5=$5&field6=$6&field7=$7&field8=$8&field9=$9&field10=$10"
 curl --output /dev/null -d $data --silent -X POST https://myapi/myendpoint;
 fi

export -f mycurl

#----------------------LOOP starts------------------------
while true; 
do 
 nc -d $host $port
done |
 parallel -j100 --pipe --block 100k parallel --colsep , -j0 mycurl