Running Python script over multiple directories and multiple files

Clash Royale CLAN TAG#URR8PPP
Here is my command line for python:
python script.py -a .faa -n .fna
I have a directory containing multiple directories. Out of these multiple directories, there could be 1 or more folders.
In these 1 or more folders, I need to run my python script on the .faa and .fna files.
How do I do this?
Example tree:
Staph1 ----> [CO1, CO2] ----> [.fna, .faa] from CO1, [.fna, .faa] from CO2
Staph2 ----> [CO6] ----> [.fna, .faa] from CO6My Python script will use the inputs .fna and .faa and output within the folder. Also the folder and directory are synonymous.
bash python
|
show 1 more comment
Here is my command line for python:
python script.py -a .faa -n .fna
I have a directory containing multiple directories. Out of these multiple directories, there could be 1 or more folders.
In these 1 or more folders, I need to run my python script on the .faa and .fna files.
How do I do this?
Example tree:
Staph1 ----> [CO1, CO2] ----> [.fna, .faa] from CO1, [.fna, .faa] from CO2
Staph2 ----> [CO6] ----> [.fna, .faa] from CO6My Python script will use the inputs .fna and .faa and output within the folder. Also the folder and directory are synonymous.
bash python
3
Did you codescript.py? Why not useos.walkto navigate the directory from withinscript.py?
– iruvar
Aug 14 '14 at 20:23
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
1
You use bothfolderanddirectoryin your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.
– dg99
Aug 14 '14 at 20:46
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
@Jeff doesscript.pytakes file names or directories? Do.fnaand.faacome in pairs?
– Simply_Me
Aug 14 '14 at 21:48
|
show 1 more comment
Here is my command line for python:
python script.py -a .faa -n .fna
I have a directory containing multiple directories. Out of these multiple directories, there could be 1 or more folders.
In these 1 or more folders, I need to run my python script on the .faa and .fna files.
How do I do this?
Example tree:
Staph1 ----> [CO1, CO2] ----> [.fna, .faa] from CO1, [.fna, .faa] from CO2
Staph2 ----> [CO6] ----> [.fna, .faa] from CO6My Python script will use the inputs .fna and .faa and output within the folder. Also the folder and directory are synonymous.
bash python
Here is my command line for python:
python script.py -a .faa -n .fna
I have a directory containing multiple directories. Out of these multiple directories, there could be 1 or more folders.
In these 1 or more folders, I need to run my python script on the .faa and .fna files.
How do I do this?
Example tree:
Staph1 ----> [CO1, CO2] ----> [.fna, .faa] from CO1, [.fna, .faa] from CO2
Staph2 ----> [CO6] ----> [.fna, .faa] from CO6My Python script will use the inputs .fna and .faa and output within the folder. Also the folder and directory are synonymous.
bash python
bash python
edited Dec 16 '18 at 11:44
Rui F Ribeiro
40.1k1479136
40.1k1479136
asked Aug 14 '14 at 20:20
JeffJeff
12
12
3
Did you codescript.py? Why not useos.walkto navigate the directory from withinscript.py?
– iruvar
Aug 14 '14 at 20:23
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
1
You use bothfolderanddirectoryin your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.
– dg99
Aug 14 '14 at 20:46
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
@Jeff doesscript.pytakes file names or directories? Do.fnaand.faacome in pairs?
– Simply_Me
Aug 14 '14 at 21:48
|
show 1 more comment
3
Did you codescript.py? Why not useos.walkto navigate the directory from withinscript.py?
– iruvar
Aug 14 '14 at 20:23
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
1
You use bothfolderanddirectoryin your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.
– dg99
Aug 14 '14 at 20:46
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
@Jeff doesscript.pytakes file names or directories? Do.fnaand.faacome in pairs?
– Simply_Me
Aug 14 '14 at 21:48
3
3
Did you code
script.py? Why not use os.walk to navigate the directory from within script.py?– iruvar
Aug 14 '14 at 20:23
Did you code
script.py? Why not use os.walk to navigate the directory from within script.py?– iruvar
Aug 14 '14 at 20:23
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
1
1
You use both
folder and directory in your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.– dg99
Aug 14 '14 at 20:46
You use both
folder and directory in your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.– dg99
Aug 14 '14 at 20:46
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
@Jeff does
script.py takes file names or directories? Do .fna and .faa come in pairs?– Simply_Me
Aug 14 '14 at 21:48
@Jeff does
script.py takes file names or directories? Do .fna and .faa come in pairs?– Simply_Me
Aug 14 '14 at 21:48
|
show 1 more comment
1 Answer
1
active
oldest
votes
I think this should help:
The script will use os.walk, as @1_CR suggested, and will call hello_world.py script with the path to both files when fna and faa files exist together (i.e. in same direcotry).
import glob, os
def scanfolder():
for path, dirs, files in os.walk('/home/shadowe/test1/test1/'):
flag_faa = 0
flag_fna = 0
for f in files:
if f.endswith('.faa'):
flag_faa = 1
faa_file_path = os.path.join(path,f)
if f.endswith('.fna'):
flag_fna = 1
fna_file_path = os.path.join(path,f)
if flag_faa == 1 and flag_fna == 1:
print "Calling script"
os.system("python hello_world.py" + " -a "+ faa_file_path + " -b " + fna_file_path)
flag_faa = 0
flag_fna = 0
scanfolder()
Second script:
import sys
print "Hello World"
print "This argument passing qualifies " + sys.argv[1] + " " + sys.argv[2] + " " + sys.argv[3] + " " + sys.argv[4]
Output:
$ python test.py
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test2/test3/two.faa -b /home/shadowe/test1/test1/test2/test3/two.fna
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test6/test7/four.faa -b /home/shadowe/test1/test1/test6/test7/five.fna
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from yourscript.py?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
|
show 1 more comment
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f150300%2frunning-python-script-over-multiple-directories-and-multiple-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I think this should help:
The script will use os.walk, as @1_CR suggested, and will call hello_world.py script with the path to both files when fna and faa files exist together (i.e. in same direcotry).
import glob, os
def scanfolder():
for path, dirs, files in os.walk('/home/shadowe/test1/test1/'):
flag_faa = 0
flag_fna = 0
for f in files:
if f.endswith('.faa'):
flag_faa = 1
faa_file_path = os.path.join(path,f)
if f.endswith('.fna'):
flag_fna = 1
fna_file_path = os.path.join(path,f)
if flag_faa == 1 and flag_fna == 1:
print "Calling script"
os.system("python hello_world.py" + " -a "+ faa_file_path + " -b " + fna_file_path)
flag_faa = 0
flag_fna = 0
scanfolder()
Second script:
import sys
print "Hello World"
print "This argument passing qualifies " + sys.argv[1] + " " + sys.argv[2] + " " + sys.argv[3] + " " + sys.argv[4]
Output:
$ python test.py
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test2/test3/two.faa -b /home/shadowe/test1/test1/test2/test3/two.fna
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test6/test7/four.faa -b /home/shadowe/test1/test1/test6/test7/five.fna
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from yourscript.py?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
|
show 1 more comment
I think this should help:
The script will use os.walk, as @1_CR suggested, and will call hello_world.py script with the path to both files when fna and faa files exist together (i.e. in same direcotry).
import glob, os
def scanfolder():
for path, dirs, files in os.walk('/home/shadowe/test1/test1/'):
flag_faa = 0
flag_fna = 0
for f in files:
if f.endswith('.faa'):
flag_faa = 1
faa_file_path = os.path.join(path,f)
if f.endswith('.fna'):
flag_fna = 1
fna_file_path = os.path.join(path,f)
if flag_faa == 1 and flag_fna == 1:
print "Calling script"
os.system("python hello_world.py" + " -a "+ faa_file_path + " -b " + fna_file_path)
flag_faa = 0
flag_fna = 0
scanfolder()
Second script:
import sys
print "Hello World"
print "This argument passing qualifies " + sys.argv[1] + " " + sys.argv[2] + " " + sys.argv[3] + " " + sys.argv[4]
Output:
$ python test.py
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test2/test3/two.faa -b /home/shadowe/test1/test1/test2/test3/two.fna
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test6/test7/four.faa -b /home/shadowe/test1/test1/test6/test7/five.fna
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from yourscript.py?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
|
show 1 more comment
I think this should help:
The script will use os.walk, as @1_CR suggested, and will call hello_world.py script with the path to both files when fna and faa files exist together (i.e. in same direcotry).
import glob, os
def scanfolder():
for path, dirs, files in os.walk('/home/shadowe/test1/test1/'):
flag_faa = 0
flag_fna = 0
for f in files:
if f.endswith('.faa'):
flag_faa = 1
faa_file_path = os.path.join(path,f)
if f.endswith('.fna'):
flag_fna = 1
fna_file_path = os.path.join(path,f)
if flag_faa == 1 and flag_fna == 1:
print "Calling script"
os.system("python hello_world.py" + " -a "+ faa_file_path + " -b " + fna_file_path)
flag_faa = 0
flag_fna = 0
scanfolder()
Second script:
import sys
print "Hello World"
print "This argument passing qualifies " + sys.argv[1] + " " + sys.argv[2] + " " + sys.argv[3] + " " + sys.argv[4]
Output:
$ python test.py
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test2/test3/two.faa -b /home/shadowe/test1/test1/test2/test3/two.fna
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test6/test7/four.faa -b /home/shadowe/test1/test1/test6/test7/five.fna
I think this should help:
The script will use os.walk, as @1_CR suggested, and will call hello_world.py script with the path to both files when fna and faa files exist together (i.e. in same direcotry).
import glob, os
def scanfolder():
for path, dirs, files in os.walk('/home/shadowe/test1/test1/'):
flag_faa = 0
flag_fna = 0
for f in files:
if f.endswith('.faa'):
flag_faa = 1
faa_file_path = os.path.join(path,f)
if f.endswith('.fna'):
flag_fna = 1
fna_file_path = os.path.join(path,f)
if flag_faa == 1 and flag_fna == 1:
print "Calling script"
os.system("python hello_world.py" + " -a "+ faa_file_path + " -b " + fna_file_path)
flag_faa = 0
flag_fna = 0
scanfolder()
Second script:
import sys
print "Hello World"
print "This argument passing qualifies " + sys.argv[1] + " " + sys.argv[2] + " " + sys.argv[3] + " " + sys.argv[4]
Output:
$ python test.py
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test2/test3/two.faa -b /home/shadowe/test1/test1/test2/test3/two.fna
Calling script
Hello World
This argument passing qualifies -a /home/shadowe/test1/test1/test6/test7/four.faa -b /home/shadowe/test1/test1/test6/test7/five.fna
edited Aug 14 '14 at 22:08
answered Aug 14 '14 at 20:54
Simply_MeSimply_Me
1,4781810
1,4781810
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from yourscript.py?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
|
show 1 more comment
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from yourscript.py?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
No such file or directory: 'CP003033.faa', is there a way to specify the path?
– Jeff
Aug 14 '14 at 21:46
@Jeff is that from your
script.py?– Simply_Me
Aug 14 '14 at 21:47
@Jeff is that from your
script.py?– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
@Jeff do you need to pass file names as well?
– Simply_Me
Aug 14 '14 at 21:47
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
Yes I need to pass in 2 file names.
– Jeff
Aug 14 '14 at 21:48
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
For example: python myscript.py -a .faa -b .fna
– Jeff
Aug 14 '14 at 21:49
|
show 1 more comment
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f150300%2frunning-python-script-over-multiple-directories-and-multiple-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
Did you code
script.py? Why not useos.walkto navigate the directory from withinscript.py?– iruvar
Aug 14 '14 at 20:23
There are over 4000 genomes, each containing 1 or more contigs. would have to use my command line 4000+ times just to get the files i need.
– Jeff
Aug 14 '14 at 20:38
1
You use both
folderanddirectoryin your question. Is there a difference between those two things? If so, what is the difference? Also, please edit your question to include a portion of an example directory structure for us to look at.– dg99
Aug 14 '14 at 20:46
A tree representing the directory hierarchy would be nice.
– Cristian Ciupitu
Aug 14 '14 at 20:56
@Jeff does
script.pytakes file names or directories? Do.fnaand.faacome in pairs?– Simply_Me
Aug 14 '14 at 21:48