Infinite loop not releasing memory in Service script
Clash Royale CLAN TAG#URR8PPP
up vote
0
down vote
favorite
I have a script which is started as part of a daemon, and runs an infinite loop to load data into a database. Depending on the hour, the data is loaded into one of 24 tables, and all data from the table that holds data for the next hour ahead is removed in order that there is never more than a days worth of data stored in the database. The data is loaded in using an application called dataflow which I believe is written in Java. My script is loosing on average 15m of memory each iteration of the loop - which continues running down until empty (normally takes a couple of days).
In order to more easily test where the leak was coming from, I put the commands that were running in the loop into a separate script, and called that script from the loop. The following is the content of that script:
#!/bin/bash
MYTABLECLEAR=$1
LOG_FILE=$2
DR_HOME=$3
CP_PLUGIN=$4
HOME_DIR=$5
PROPERTIES_HC=$6
#Run SQL to clear down table table 1 hour ahead of now (23 hours behind)
(sql -uactian echo <<-EOSQL
MODIFY reverb$MYTABLECLEAR TO TRUNCATED;g
EOSQL
) 1> /dev/null 2>> $LOG_FILE
#Run Dataflow
$DR_HOME/bin/dr -cp $CP_PLUGIN -Xmx64g --engine parallelism=1 --runjson $HOME_DIR/workflows/dl_Reverb.dr --overridefile $PROPERTIES_HC 1> /dev/null 2>> $LOG_FILE
If I comment out the Run SQL and the Run Dataflow sections of this code whilst my service is running, my memory loss flattens out i.e. I don't get anything back, but it doesn't get worse. If I then un-comment either of these, my leak resumes.
If I stop the service, I regain all memory immediately.
I have tried monitoring the PIDs of the processes I am running, by adding a echo "PID: " $! >> $LOG_FILE after each process, but the log file just shows blank.
Any ideas why my script is holding onto memory and how can I make it not?
shell-script memory daemon memory-leaks
asked Feb 7 at 10:49
paul frith
33
 |Â
show 3 more comments
up vote
0
down vote
favorite
I have a script which is started as part of a daemon, and runs an infinite loop to load data into a database. Depending on the hour, the data is loaded into one of 24 tables, and all data from the table that holds data for the next hour ahead is removed in order that there is never more than a days worth of data stored in the database. The data is loaded in using an application called dataflow which I believe is written in Java. My script is loosing on average 15m of memory each iteration of the loop - which continues running down until empty (normally takes a couple of days).
In order to more easily test where the leak was coming from, I put the commands that were running in the loop into a separate script, and called that script from the loop. The following is the content of that script:
#!/bin/bash
MYTABLECLEAR=$1
LOG_FILE=$2
DR_HOME=$3
CP_PLUGIN=$4
HOME_DIR=$5
PROPERTIES_HC=$6
#Run SQL to clear down table table 1 hour ahead of now (23 hours behind)
(sql -uactian echo <<-EOSQL
MODIFY reverb$MYTABLECLEAR TO TRUNCATED;g
EOSQL
) 1> /dev/null 2>> $LOG_FILE
#Run Dataflow
$DR_HOME/bin/dr -cp $CP_PLUGIN -Xmx64g --engine parallelism=1 --runjson $HOME_DIR/workflows/dl_Reverb.dr --overridefile $PROPERTIES_HC 1> /dev/null 2>> $LOG_FILE
If I comment out the Run SQL and the Run Dataflow sections of this code whilst my service is running, my memory loss flattens out i.e. I don't get anything back, but it doesn't get worse. If I then un-comment either of these, my leak resumes.
If I stop the service, I regain all memory immediately.
I have tried monitoring the PIDs of the processes I am running, by adding a echo "PID: " $! >> $LOG_FILE after each process, but the log file just shows blank.
Any ideas why my script is holding onto memory and how can I make it not?
shell-script memory daemon memory-leaks
asked Feb 7 at 10:49
paul frith
33
1
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actuallysleep 10sas part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer
– paul frith
Feb 7 at 11:18
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41
 |Â
show 3 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have a script which is started as part of a daemon, and runs an infinite loop to load data into a database. Depending on the hour, the data is loaded into one of 24 tables, and all data from the table that holds data for the next hour ahead is removed in order that there is never more than a days worth of data stored in the database. The data is loaded in using an application called dataflow which I believe is written in Java. My script is loosing on average 15m of memory each iteration of the loop - which continues running down until empty (normally takes a couple of days).
In order to more easily test where the leak was coming from, I put the commands that were running in the loop into a separate script, and called that script from the loop. The following is the content of that script:
#!/bin/bash
MYTABLECLEAR=$1
LOG_FILE=$2
DR_HOME=$3
CP_PLUGIN=$4
HOME_DIR=$5
PROPERTIES_HC=$6
#Run SQL to clear down table table 1 hour ahead of now (23 hours behind)
(sql -uactian echo <<-EOSQL
MODIFY reverb$MYTABLECLEAR TO TRUNCATED;g
EOSQL
) 1> /dev/null 2>> $LOG_FILE
#Run Dataflow
$DR_HOME/bin/dr -cp $CP_PLUGIN -Xmx64g --engine parallelism=1 --runjson $HOME_DIR/workflows/dl_Reverb.dr --overridefile $PROPERTIES_HC 1> /dev/null 2>> $LOG_FILE
If I comment out the Run SQL and the Run Dataflow sections of this code whilst my service is running, my memory loss flattens out i.e. I don't get anything back, but it doesn't get worse. If I then un-comment either of these, my leak resumes.
If I stop the service, I regain all memory immediately.
I have tried monitoring the PIDs of the processes I am running, by adding a echo "PID: " $! >> $LOG_FILE after each process, but the log file just shows blank.
Any ideas why my script is holding onto memory and how can I make it not?
shell-script memory daemon memory-leaks
asked Feb 7 at 10:49
paul frith
33
I have a script which is started as part of a daemon, and runs an infinite loop to load data into a database. Depending on the hour, the data is loaded into one of 24 tables, and all data from the table that holds data for the next hour ahead is removed in order that there is never more than a days worth of data stored in the database. The data is loaded in using an application called dataflow which I believe is written in Java. My script is loosing on average 15m of memory each iteration of the loop - which continues running down until empty (normally takes a couple of days).
In order to more easily test where the leak was coming from, I put the commands that were running in the loop into a separate script, and called that script from the loop. The following is the content of that script:
#!/bin/bash
MYTABLECLEAR=$1
LOG_FILE=$2
DR_HOME=$3
CP_PLUGIN=$4
HOME_DIR=$5
PROPERTIES_HC=$6
#Run SQL to clear down table table 1 hour ahead of now (23 hours behind)
(sql -uactian echo <<-EOSQL
MODIFY reverb$MYTABLECLEAR TO TRUNCATED;g
EOSQL
) 1> /dev/null 2>> $LOG_FILE
#Run Dataflow
$DR_HOME/bin/dr -cp $CP_PLUGIN -Xmx64g --engine parallelism=1 --runjson $HOME_DIR/workflows/dl_Reverb.dr --overridefile $PROPERTIES_HC 1> /dev/null 2>> $LOG_FILE
If I comment out the Run SQL and the Run Dataflow sections of this code whilst my service is running, my memory loss flattens out i.e. I don't get anything back, but it doesn't get worse. If I then un-comment either of these, my leak resumes.
If I stop the service, I regain all memory immediately.
I have tried monitoring the PIDs of the processes I am running, by adding a echo "PID: " $! >> $LOG_FILE after each process, but the log file just shows blank.
Any ideas why my script is holding onto memory and how can I make it not?
shell-script memory daemon memory-leaks
asked Feb 7 at 10:49
paul frith
33
edited Feb 7 at 11:10
asked Feb 7 at 10:49
paul frith
33
asked Feb 7 at 10:49
paul frith
33
asked Feb 7 at 10:49
paul frith
33
33
1
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actuallysleep 10sas part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer
– paul frith
Feb 7 at 11:18
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41
 |Â
show 3 more comments
1
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actuallysleep 10sas part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer
– paul frith
Feb 7 at 11:18
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41
1
1
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actually
sleep 10s as part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer– paul frith
Feb 7 at 11:18
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actually
sleep 10s as part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer– paul frith
Feb 7 at 11:18
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41
 |Â
show 3 more comments
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f422512%2finfinite-loop-not-releasing-memory-in-service-script%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
1
Does this have to run in an infinite loop or can it be set up to run periodically?
– Raman Sailopal
Feb 7 at 11:11
It needs to run as often as possible, and at a more granular level than a cronjob can. I do actually
sleep 10sas part of the loop - I have tried increasing this value, but I am still losing as much memory, just at a slightly slower rate, due to the iterations taking longer– paul frith
Feb 7 at 11:18
The systemd timers allow for greater granularity (milliseconds) Not sure whether this can be set up a service unit and if this is an option to you.
– Raman Sailopal
Feb 7 at 11:27
Is there an issue with using an infinite loop then?
– paul frith
Feb 7 at 11:28
By settings it up as a service, with a timer, it will stop and then restart clearing any memory. It can of course be running in an infinite look but the software would have to be engineered in such a way that memory is released.
– Raman Sailopal
Feb 7 at 11:41