Curl request exits at five minutes unexpectedly
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I'm using a shell script to fire off a curl request that deletes all documents of a certain type in my Elasticsearch cluster. If successful, Logstash starts and attempts to reindex the deleted type's documents.
printf "n%sn" "Sending delete request..."
RESPONSE=$(curl --max-time 600 -XPOST
"https://mycluster.com:9243/locations/ca/_delete_by_query?conflicts=proceed" -H
'Content-Type: application/json' -d' "query": "match_all": '
--user user:password)
if [[ $RESPONSE == *"deleted"* ]]; then
echo $RESPONSE
printf "n%sn" "Delete successful, starting Logstash..."
cd /home/denroot/EP-logstash && bin/logstash -f /home/denroot/EP-
logstash-configs/locations/create/california.config
else
echo $RESPONSE
printf "n%sn" "Delete failed, shutting down..."
fi
The issue I've encountered is this particular type has over 5 million documents. The Elasticsearch delete task cannot finish before 5 minutes have elapsed, and my curl request exits with an exit code of 0. I can't seem to prevent this from happening with --max-time. Normally, when the delete task finishes in Elasticsearch, it sends back a JSON response body which I need for my control-flow logic to work properly. Any help preventing this 5 minute exit issue would be greatly appreciated as I can't seem to find a precedent.
curl timeout
asked Nov 9 '17 at 16:45
taylorhessel
1
add a comment |Â
up vote
0
down vote
favorite
I'm using a shell script to fire off a curl request that deletes all documents of a certain type in my Elasticsearch cluster. If successful, Logstash starts and attempts to reindex the deleted type's documents.
printf "n%sn" "Sending delete request..."
RESPONSE=$(curl --max-time 600 -XPOST
"https://mycluster.com:9243/locations/ca/_delete_by_query?conflicts=proceed" -H
'Content-Type: application/json' -d' "query": "match_all": '
--user user:password)
if [[ $RESPONSE == *"deleted"* ]]; then
echo $RESPONSE
printf "n%sn" "Delete successful, starting Logstash..."
cd /home/denroot/EP-logstash && bin/logstash -f /home/denroot/EP-
logstash-configs/locations/create/california.config
else
echo $RESPONSE
printf "n%sn" "Delete failed, shutting down..."
fi
The issue I've encountered is this particular type has over 5 million documents. The Elasticsearch delete task cannot finish before 5 minutes have elapsed, and my curl request exits with an exit code of 0. I can't seem to prevent this from happening with --max-time. Normally, when the delete task finishes in Elasticsearch, it sends back a JSON response body which I need for my control-flow logic to work properly. Any help preventing this 5 minute exit issue would be greatly appreciated as I can't seem to find a precedent.
curl timeout
asked Nov 9 '17 at 16:45
taylorhessel
1
1
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Exit code 0. Socurlthinks the request is complete. What HTTP status do you get back?
– roaima
Nov 9 '17 at 22:19
When curl exits, the value of$RESPONSEis empty.
– taylorhessel
Nov 10 '17 at 16:43
curlwould return exit code 0 even on a 4xx HTTP error. Try runing yourcurlcommand with-Ito view the status code returned by the server and other HTTP headers.
– cherdt
Nov 11 '17 at 0:35
add a comment |Â
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm using a shell script to fire off a curl request that deletes all documents of a certain type in my Elasticsearch cluster. If successful, Logstash starts and attempts to reindex the deleted type's documents.
printf "n%sn" "Sending delete request..."
RESPONSE=$(curl --max-time 600 -XPOST
"https://mycluster.com:9243/locations/ca/_delete_by_query?conflicts=proceed" -H
'Content-Type: application/json' -d' "query": "match_all": '
--user user:password)
if [[ $RESPONSE == *"deleted"* ]]; then
echo $RESPONSE
printf "n%sn" "Delete successful, starting Logstash..."
cd /home/denroot/EP-logstash && bin/logstash -f /home/denroot/EP-
logstash-configs/locations/create/california.config
else
echo $RESPONSE
printf "n%sn" "Delete failed, shutting down..."
fi
The issue I've encountered is this particular type has over 5 million documents. The Elasticsearch delete task cannot finish before 5 minutes have elapsed, and my curl request exits with an exit code of 0. I can't seem to prevent this from happening with --max-time. Normally, when the delete task finishes in Elasticsearch, it sends back a JSON response body which I need for my control-flow logic to work properly. Any help preventing this 5 minute exit issue would be greatly appreciated as I can't seem to find a precedent.
curl timeout
asked Nov 9 '17 at 16:45
taylorhessel
1
I'm using a shell script to fire off a curl request that deletes all documents of a certain type in my Elasticsearch cluster. If successful, Logstash starts and attempts to reindex the deleted type's documents.
printf "n%sn" "Sending delete request..."
RESPONSE=$(curl --max-time 600 -XPOST
"https://mycluster.com:9243/locations/ca/_delete_by_query?conflicts=proceed" -H
'Content-Type: application/json' -d' "query": "match_all": '
--user user:password)
if [[ $RESPONSE == *"deleted"* ]]; then
echo $RESPONSE
printf "n%sn" "Delete successful, starting Logstash..."
cd /home/denroot/EP-logstash && bin/logstash -f /home/denroot/EP-
logstash-configs/locations/create/california.config
else
echo $RESPONSE
printf "n%sn" "Delete failed, shutting down..."
fi
The issue I've encountered is this particular type has over 5 million documents. The Elasticsearch delete task cannot finish before 5 minutes have elapsed, and my curl request exits with an exit code of 0. I can't seem to prevent this from happening with --max-time. Normally, when the delete task finishes in Elasticsearch, it sends back a JSON response body which I need for my control-flow logic to work properly. Any help preventing this 5 minute exit issue would be greatly appreciated as I can't seem to find a precedent.
curl timeout
asked Nov 9 '17 at 16:45
taylorhessel
1
asked Nov 9 '17 at 16:45
taylorhessel
1
asked Nov 9 '17 at 16:45
taylorhessel
1
asked Nov 9 '17 at 16:45
taylorhessel
1
1
1
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Exit code 0. Socurlthinks the request is complete. What HTTP status do you get back?
– roaima
Nov 9 '17 at 22:19
When curl exits, the value of$RESPONSEis empty.
– taylorhessel
Nov 10 '17 at 16:43
curlwould return exit code 0 even on a 4xx HTTP error. Try runing yourcurlcommand with-Ito view the status code returned by the server and other HTTP headers.
– cherdt
Nov 11 '17 at 0:35
add a comment |Â
1
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Exit code 0. Socurlthinks the request is complete. What HTTP status do you get back?
– roaima
Nov 9 '17 at 22:19
When curl exits, the value of$RESPONSEis empty.
– taylorhessel
Nov 10 '17 at 16:43
curlwould return exit code 0 even on a 4xx HTTP error. Try runing yourcurlcommand with-Ito view the status code returned by the server and other HTTP headers.
– cherdt
Nov 11 '17 at 0:35
1
1
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Exit code 0. So
curl thinks the request is complete. What HTTP status do you get back?– roaima
Nov 9 '17 at 22:19
Exit code 0. So
curl thinks the request is complete. What HTTP status do you get back?– roaima
Nov 9 '17 at 22:19
When curl exits, the value of
$RESPONSE is empty.– taylorhessel
Nov 10 '17 at 16:43
When curl exits, the value of
$RESPONSE is empty.– taylorhessel
Nov 10 '17 at 16:43
curl would return exit code 0 even on a 4xx HTTP error. Try runing your curl command with -I to view the status code returned by the server and other HTTP headers.– cherdt
Nov 11 '17 at 0:35
curl would return exit code 0 even on a 4xx HTTP error. Try runing your curl command with -I to view the status code returned by the server and other HTTP headers.– cherdt
Nov 11 '17 at 0:35
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
0
down vote
Your script should exit after 10 minutes as it is.
Try increasing (or removing) max-time for lifting that limit
If the 5 minutes timeout is imposed on server side, you could try running the curl command multiple times until you receive the expected output/exit code.
answered Nov 9 '17 at 20:12
user48452
465
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Your script should exit after 10 minutes as it is.
Try increasing (or removing) max-time for lifting that limit
If the 5 minutes timeout is imposed on server side, you could try running the curl command multiple times until you receive the expected output/exit code.
answered Nov 9 '17 at 20:12
user48452
465
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
add a comment |Â
up vote
0
down vote
Your script should exit after 10 minutes as it is.
Try increasing (or removing) max-time for lifting that limit
If the 5 minutes timeout is imposed on server side, you could try running the curl command multiple times until you receive the expected output/exit code.
answered Nov 9 '17 at 20:12
user48452
465
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
add a comment |Â
up vote
0
down vote
up vote
0
down vote
Your script should exit after 10 minutes as it is.
Try increasing (or removing) max-time for lifting that limit
If the 5 minutes timeout is imposed on server side, you could try running the curl command multiple times until you receive the expected output/exit code.
answered Nov 9 '17 at 20:12
user48452
465
Your script should exit after 10 minutes as it is.
Try increasing (or removing) max-time for lifting that limit
If the 5 minutes timeout is imposed on server side, you could try running the curl command multiple times until you receive the expected output/exit code.
answered Nov 9 '17 at 20:12
user48452
465
answered Nov 9 '17 at 20:12
user48452
465
answered Nov 9 '17 at 20:12
user48452
465
answered Nov 9 '17 at 20:12
user48452
465
465
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
add a comment |Â
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
You may be onto something that the timeout is imposed server-side. Unfortunately, there doesn't seem to be any documentation in the Elasticsearch delete-by-query API about response timeouts. Thank you anyway!
– taylorhessel
Nov 9 '17 at 20:28
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f403561%2fcurl-request-exits-at-five-minutes-unexpectedly%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Can you rework the logic to delete only a few (tens of) thousand documents at a time, and work at it in smaller bites?
– DopeGhoti
Nov 9 '17 at 16:48
Exit code 0. So
curlthinks the request is complete. What HTTP status do you get back?– roaima
Nov 9 '17 at 22:19
When curl exits, the value of
$RESPONSEis empty.– taylorhessel
Nov 10 '17 at 16:43
curlwould return exit code 0 even on a 4xx HTTP error. Try runing yourcurlcommand with-Ito view the status code returned by the server and other HTTP headers.– cherdt
Nov 11 '17 at 0:35