Detecting misbehaving USB flash memory devices, that block the rest's IO
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I'm using f3 and a custom Bash script to test USB flash memory in big amounts.
A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling the testing process.
For example - when leaving 50 USB drives for testing I often find out after an hour that 48 are doing nothing, and 2 are blinking their LEDs. Removing these two drives suddenly resumes all others drives testing.
There sometimes are more complex situations where 24 of the drives are stalled, and the rest seems to work fine. Except a few drives don't make any progress after 20 minutes. You plug them out, the rest is coming back to life, and the testing continues.
However - I also found out it's enough to stop testing the faulty drives to make the rest come to life.
I'm looking for a way to find out which drives are causing this file operation blockage on others so I can automatically stop them in my script.
I've been watching atop, iostat, htop and dmesg to try and find a discriminating factor, but I can't see anything. I've found there's so called usbmon kernel debugging interface, though it's so low-level that I don't really know how to use it. The raw USB packets don't tell me anything.
Are there any other tools I could use to tell which drives are misbehaving?
I use f3write and f3read programs to test the drives. The f3write program creates 1GB files that the f3read program then reads identifying any data damage that occurred in the process.
Also - it's strange, but when a misbehaving drive is present, the rest of the "healthy" drives will finish their work on the current file. Say - writing or reading a 1GB sized file - but will not create a new file until the misbehaving drives are removed. It's like opening a new file becomes impossible in the presence of an "IO hog" drive.
What can I do to tell them apart?
linux usb io
edited Nov 29 at 14:57
sourcejedi
22.3k43398
asked Nov 29 at 12:24
unfa
546313
add a comment |
up vote
1
down vote
favorite
I'm using f3 and a custom Bash script to test USB flash memory in big amounts.
A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling the testing process.
For example - when leaving 50 USB drives for testing I often find out after an hour that 48 are doing nothing, and 2 are blinking their LEDs. Removing these two drives suddenly resumes all others drives testing.
There sometimes are more complex situations where 24 of the drives are stalled, and the rest seems to work fine. Except a few drives don't make any progress after 20 minutes. You plug them out, the rest is coming back to life, and the testing continues.
However - I also found out it's enough to stop testing the faulty drives to make the rest come to life.
I'm looking for a way to find out which drives are causing this file operation blockage on others so I can automatically stop them in my script.
I've been watching atop, iostat, htop and dmesg to try and find a discriminating factor, but I can't see anything. I've found there's so called usbmon kernel debugging interface, though it's so low-level that I don't really know how to use it. The raw USB packets don't tell me anything.
Are there any other tools I could use to tell which drives are misbehaving?
I use f3write and f3read programs to test the drives. The f3write program creates 1GB files that the f3read program then reads identifying any data damage that occurred in the process.
Also - it's strange, but when a misbehaving drive is present, the rest of the "healthy" drives will finish their work on the current file. Say - writing or reading a 1GB sized file - but will not create a new file until the misbehaving drives are removed. It's like opening a new file becomes impossible in the presence of an "IO hog" drive.
What can I do to tell them apart?
linux usb io
edited Nov 29 at 14:57
sourcejedi
22.3k43398
asked Nov 29 at 12:24
unfa
546313
Hmm. How wide did you make youratopwindow? Can you see thedirtyfigure? I actually have to decrease font size to get it to show on my laptop.
– sourcejedi
Nov 29 at 15:07
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm using f3 and a custom Bash script to test USB flash memory in big amounts.
A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling the testing process.
For example - when leaving 50 USB drives for testing I often find out after an hour that 48 are doing nothing, and 2 are blinking their LEDs. Removing these two drives suddenly resumes all others drives testing.
There sometimes are more complex situations where 24 of the drives are stalled, and the rest seems to work fine. Except a few drives don't make any progress after 20 minutes. You plug them out, the rest is coming back to life, and the testing continues.
However - I also found out it's enough to stop testing the faulty drives to make the rest come to life.
I'm looking for a way to find out which drives are causing this file operation blockage on others so I can automatically stop them in my script.
I've been watching atop, iostat, htop and dmesg to try and find a discriminating factor, but I can't see anything. I've found there's so called usbmon kernel debugging interface, though it's so low-level that I don't really know how to use it. The raw USB packets don't tell me anything.
Are there any other tools I could use to tell which drives are misbehaving?
I use f3write and f3read programs to test the drives. The f3write program creates 1GB files that the f3read program then reads identifying any data damage that occurred in the process.
Also - it's strange, but when a misbehaving drive is present, the rest of the "healthy" drives will finish their work on the current file. Say - writing or reading a 1GB sized file - but will not create a new file until the misbehaving drives are removed. It's like opening a new file becomes impossible in the presence of an "IO hog" drive.
What can I do to tell them apart?
linux usb io
edited Nov 29 at 14:57
sourcejedi
22.3k43398
asked Nov 29 at 12:24
unfa
546313
I'm using f3 and a custom Bash script to test USB flash memory in big amounts.
A common problem I encounter is that some faulty drives would starve all the healthy ones for IO, effectively stalling the testing process.
For example - when leaving 50 USB drives for testing I often find out after an hour that 48 are doing nothing, and 2 are blinking their LEDs. Removing these two drives suddenly resumes all others drives testing.
There sometimes are more complex situations where 24 of the drives are stalled, and the rest seems to work fine. Except a few drives don't make any progress after 20 minutes. You plug them out, the rest is coming back to life, and the testing continues.
However - I also found out it's enough to stop testing the faulty drives to make the rest come to life.
I'm looking for a way to find out which drives are causing this file operation blockage on others so I can automatically stop them in my script.
I've been watching atop, iostat, htop and dmesg to try and find a discriminating factor, but I can't see anything. I've found there's so called usbmon kernel debugging interface, though it's so low-level that I don't really know how to use it. The raw USB packets don't tell me anything.
Are there any other tools I could use to tell which drives are misbehaving?
I use f3write and f3read programs to test the drives. The f3write program creates 1GB files that the f3read program then reads identifying any data damage that occurred in the process.
Also - it's strange, but when a misbehaving drive is present, the rest of the "healthy" drives will finish their work on the current file. Say - writing or reading a 1GB sized file - but will not create a new file until the misbehaving drives are removed. It's like opening a new file becomes impossible in the presence of an "IO hog" drive.
What can I do to tell them apart?
linux usb io
linux usb io
edited Nov 29 at 14:57
sourcejedi
22.3k43398
asked Nov 29 at 12:24
unfa
546313
edited Nov 29 at 14:57
sourcejedi
22.3k43398
asked Nov 29 at 12:24
unfa
546313
edited Nov 29 at 14:57
sourcejedi
22.3k43398
edited Nov 29 at 14:57
sourcejedi
22.3k43398
edited Nov 29 at 14:57
sourcejedi
22.3k43398
22.3k43398
asked Nov 29 at 12:24
unfa
546313
asked Nov 29 at 12:24
unfa
546313
asked Nov 29 at 12:24
unfa
546313
546313
Hmm. How wide did you make youratopwindow? Can you see thedirtyfigure? I actually have to decrease font size to get it to show on my laptop.
– sourcejedi
Nov 29 at 15:07
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50
add a comment |
Hmm. How wide did you make youratopwindow? Can you see thedirtyfigure? I actually have to decrease font size to get it to show on my laptop.
– sourcejedi
Nov 29 at 15:07
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50
Hmm. How wide did you make your
atop window? Can you see the dirty figure? I actually have to decrease font size to get it to show on my laptop.– sourcejedi
Nov 29 at 15:07
Hmm. How wide did you make your
atop window? Can you see the dirty figure? I actually have to decrease font size to get it to show on my laptop.– sourcejedi
Nov 29 at 15:07
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f484900%2fdetecting-misbehaving-usb-flash-memory-devices-that-block-the-rests-io%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Hmm. How wide did you make your
atopwindow? Can you see thedirtyfigure? I actually have to decrease font size to get it to show on my laptop.– sourcejedi
Nov 29 at 15:07
I haven't check that one! Will try it! Thanks!
– unfa
Nov 29 at 18:50