Extremely slow dnf

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP












2















I got a strange issue with extremely slow dnf, to the point where computer freezes for a few seconds during installation of packages. Running Fedora 29, but this is happening for years. Otherwise the computer works fine, never crashes, obviously only dnf hits some weak spot. Not only can "dnf upgrade" take an hour for 100 packages, every year when I upgrade Fedora, I wait for Christmas holidays so I can live without computer over the weekend as 3000+ package update starts on Friday evening and finishes on Sunday.



Root directory is mounted to an old SSD disk, /home on a new classic disk (WD Caviar Black), there is another disk just for backups (just to tell there are multiple disks installed). I have suspected the old SSD to misbehave, moved the root partition to one of the classic disks and it did not improve the situation (was even slower).



Journalctl or dmesg or /var/log/messages do not show anything special. Sysbench says that disk performance of every installed disk is fine, just as fine as on another (older) computer that also runs Fedora 29 and where dnf is fine (upgrade in a few minutes, annual release upgrade during lunch).



Does anyone have an idea where to look for what is happening?



P.S.
top, iotop, iostat, dmesg, journalctl, such elementary things were checked a lot. iotop shows 99.99% activity on disk by python... (dnf) when it does the upgrade but I guess that's normal. Even sysbench (as stated above) shows the disks are a bit faster than at another computer that does dnf in a breeze.



For the make: i7-920 processor, 24 G RAM, no swap










share|improve this question
























  • Just to be sure: this is during the installation part, and not network?

    – mattdm
    Jan 26 at 9:58











  • Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

    – Adam Williamson
    Jan 26 at 11:06











  • Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

    – fche
    Jan 26 at 14:48












  • @mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

    – Robert Špendl
    Jan 26 at 15:27











  • I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

    – Robert Špendl
    Jan 26 at 15:41















2















I got a strange issue with extremely slow dnf, to the point where computer freezes for a few seconds during installation of packages. Running Fedora 29, but this is happening for years. Otherwise the computer works fine, never crashes, obviously only dnf hits some weak spot. Not only can "dnf upgrade" take an hour for 100 packages, every year when I upgrade Fedora, I wait for Christmas holidays so I can live without computer over the weekend as 3000+ package update starts on Friday evening and finishes on Sunday.



Root directory is mounted to an old SSD disk, /home on a new classic disk (WD Caviar Black), there is another disk just for backups (just to tell there are multiple disks installed). I have suspected the old SSD to misbehave, moved the root partition to one of the classic disks and it did not improve the situation (was even slower).



Journalctl or dmesg or /var/log/messages do not show anything special. Sysbench says that disk performance of every installed disk is fine, just as fine as on another (older) computer that also runs Fedora 29 and where dnf is fine (upgrade in a few minutes, annual release upgrade during lunch).



Does anyone have an idea where to look for what is happening?



P.S.
top, iotop, iostat, dmesg, journalctl, such elementary things were checked a lot. iotop shows 99.99% activity on disk by python... (dnf) when it does the upgrade but I guess that's normal. Even sysbench (as stated above) shows the disks are a bit faster than at another computer that does dnf in a breeze.



For the make: i7-920 processor, 24 G RAM, no swap










share|improve this question
























  • Just to be sure: this is during the installation part, and not network?

    – mattdm
    Jan 26 at 9:58











  • Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

    – Adam Williamson
    Jan 26 at 11:06











  • Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

    – fche
    Jan 26 at 14:48












  • @mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

    – Robert Špendl
    Jan 26 at 15:27











  • I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

    – Robert Špendl
    Jan 26 at 15:41













2












2








2








I got a strange issue with extremely slow dnf, to the point where computer freezes for a few seconds during installation of packages. Running Fedora 29, but this is happening for years. Otherwise the computer works fine, never crashes, obviously only dnf hits some weak spot. Not only can "dnf upgrade" take an hour for 100 packages, every year when I upgrade Fedora, I wait for Christmas holidays so I can live without computer over the weekend as 3000+ package update starts on Friday evening and finishes on Sunday.



Root directory is mounted to an old SSD disk, /home on a new classic disk (WD Caviar Black), there is another disk just for backups (just to tell there are multiple disks installed). I have suspected the old SSD to misbehave, moved the root partition to one of the classic disks and it did not improve the situation (was even slower).



Journalctl or dmesg or /var/log/messages do not show anything special. Sysbench says that disk performance of every installed disk is fine, just as fine as on another (older) computer that also runs Fedora 29 and where dnf is fine (upgrade in a few minutes, annual release upgrade during lunch).



Does anyone have an idea where to look for what is happening?



P.S.
top, iotop, iostat, dmesg, journalctl, such elementary things were checked a lot. iotop shows 99.99% activity on disk by python... (dnf) when it does the upgrade but I guess that's normal. Even sysbench (as stated above) shows the disks are a bit faster than at another computer that does dnf in a breeze.



For the make: i7-920 processor, 24 G RAM, no swap










share|improve this question
















I got a strange issue with extremely slow dnf, to the point where computer freezes for a few seconds during installation of packages. Running Fedora 29, but this is happening for years. Otherwise the computer works fine, never crashes, obviously only dnf hits some weak spot. Not only can "dnf upgrade" take an hour for 100 packages, every year when I upgrade Fedora, I wait for Christmas holidays so I can live without computer over the weekend as 3000+ package update starts on Friday evening and finishes on Sunday.



Root directory is mounted to an old SSD disk, /home on a new classic disk (WD Caviar Black), there is another disk just for backups (just to tell there are multiple disks installed). I have suspected the old SSD to misbehave, moved the root partition to one of the classic disks and it did not improve the situation (was even slower).



Journalctl or dmesg or /var/log/messages do not show anything special. Sysbench says that disk performance of every installed disk is fine, just as fine as on another (older) computer that also runs Fedora 29 and where dnf is fine (upgrade in a few minutes, annual release upgrade during lunch).



Does anyone have an idea where to look for what is happening?



P.S.
top, iotop, iostat, dmesg, journalctl, such elementary things were checked a lot. iotop shows 99.99% activity on disk by python... (dnf) when it does the upgrade but I guess that's normal. Even sysbench (as stated above) shows the disks are a bit faster than at another computer that does dnf in a breeze.



For the make: i7-920 processor, 24 G RAM, no swap







fedora disk dnf






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 25 at 23:22









Rui F Ribeiro

40.1k1479136




40.1k1479136










asked Jan 25 at 23:09









Robert ŠpendlRobert Špendl

134




134












  • Just to be sure: this is during the installation part, and not network?

    – mattdm
    Jan 26 at 9:58











  • Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

    – Adam Williamson
    Jan 26 at 11:06











  • Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

    – fche
    Jan 26 at 14:48












  • @mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

    – Robert Špendl
    Jan 26 at 15:27











  • I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

    – Robert Špendl
    Jan 26 at 15:41

















  • Just to be sure: this is during the installation part, and not network?

    – mattdm
    Jan 26 at 9:58











  • Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

    – Adam Williamson
    Jan 26 at 11:06











  • Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

    – fche
    Jan 26 at 14:48












  • @mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

    – Robert Špendl
    Jan 26 at 15:27











  • I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

    – Robert Špendl
    Jan 26 at 15:41
















Just to be sure: this is during the installation part, and not network?

– mattdm
Jan 26 at 9:58





Just to be sure: this is during the installation part, and not network?

– mattdm
Jan 26 at 9:58













Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

– Adam Williamson
Jan 26 at 11:06





Can you try / have you tried running dnf through strace to see if that gives any clues? Have you checked the dnf-specific log files, /var/log/dnf*.log? (These contain some stuff that isn't sent to the journal).

– Adam Williamson
Jan 26 at 11:06













Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

– fche
Jan 26 at 14:48






Is it cpu-bound or i/o bound? What's the utilization of the CPU and the disks (iostat -x)? (Long shot, but could try rpm --rebuilddb.)

– fche
Jan 26 at 14:48














@mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

– Robert Špendl
Jan 26 at 15:27





@mattdm It's just installation, not network. Even on release upgrade, when all packages are downloaded and the computers is installing them in single user mode, it takes a long time. So network or any other software interference seems to be irrelevant.

– Robert Špendl
Jan 26 at 15:27













I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

– Robert Špendl
Jan 26 at 15:41





I have tried perf as @Joe Mario suggested and fdatasync sems to be very slow. Any ideas on "step 2"?

– Robert Špendl
Jan 26 at 15:41










1 Answer
1






active

oldest

votes


















1














Robert:
It would be interesting to know what system call is taking the longest.



Can you install a simple (random) package using perf trace and then sort to find the longest calls?



For example:
# perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64



And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.






share|improve this answer























  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

    – Robert Špendl
    Jan 26 at 15:38











  • (eventually I'll get better at editing comments, have patience with a noob)

    – Robert Špendl
    Jan 26 at 15:39











  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

    – Robert Špendl
    Jan 26 at 15:40











  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

    – Robert Špendl
    Jan 26 at 15:57











  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

    – fche
    Jan 26 at 16:13










Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f496775%2fextremely-slow-dnf%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














Robert:
It would be interesting to know what system call is taking the longest.



Can you install a simple (random) package using perf trace and then sort to find the longest calls?



For example:
# perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64



And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.






share|improve this answer























  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

    – Robert Špendl
    Jan 26 at 15:38











  • (eventually I'll get better at editing comments, have patience with a noob)

    – Robert Špendl
    Jan 26 at 15:39











  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

    – Robert Špendl
    Jan 26 at 15:40











  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

    – Robert Špendl
    Jan 26 at 15:57











  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

    – fche
    Jan 26 at 16:13















1














Robert:
It would be interesting to know what system call is taking the longest.



Can you install a simple (random) package using perf trace and then sort to find the longest calls?



For example:
# perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64



And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.






share|improve this answer























  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

    – Robert Špendl
    Jan 26 at 15:38











  • (eventually I'll get better at editing comments, have patience with a noob)

    – Robert Špendl
    Jan 26 at 15:39











  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

    – Robert Špendl
    Jan 26 at 15:40











  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

    – Robert Špendl
    Jan 26 at 15:57











  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

    – fche
    Jan 26 at 16:13













1












1








1







Robert:
It would be interesting to know what system call is taking the longest.



Can you install a simple (random) package using perf trace and then sort to find the longest calls?



For example:
# perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64



And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.






share|improve this answer













Robert:
It would be interesting to know what system call is taking the longest.



Can you install a simple (random) package using perf trace and then sort to find the longest calls?



For example:
# perf trace -s -o /tmp/trace.out dnf install -y xorg-x11-apps-7.7-20.fc28.x86_64



And then post the contents of your /tmp/trace.out file. It's a summary file, so it shouldn't be too large, but it will help point you at what syscall(s) are taking the longest, and if they are way out of normal range.







share|improve this answer












share|improve this answer



share|improve this answer










answered Jan 26 at 13:33









Joe MarioJoe Mario

261




261












  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

    – Robert Špendl
    Jan 26 at 15:38











  • (eventually I'll get better at editing comments, have patience with a noob)

    – Robert Špendl
    Jan 26 at 15:39











  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

    – Robert Špendl
    Jan 26 at 15:40











  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

    – Robert Špendl
    Jan 26 at 15:57











  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

    – fche
    Jan 26 at 16:13

















  • Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

    – Robert Špendl
    Jan 26 at 15:38











  • (eventually I'll get better at editing comments, have patience with a noob)

    – Robert Špendl
    Jan 26 at 15:39











  • It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

    – Robert Špendl
    Jan 26 at 15:40











  • I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

    – Robert Špendl
    Jan 26 at 15:57











  • An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

    – fche
    Jan 26 at 16:13
















Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

– Robert Špendl
Jan 26 at 15:38





Seems it's getting warmer. I have tried your suggestion on both "slow" and "normal" computers and it looks like fdatasync is the culprit. SLOW ` syscall calls total min avg max stddev (msec) (msec) (msec) (msec) (%) --------------- -------- --------- --------- --------- --------- ------ fdatasync 69 20651.658 0.072 299.299 6356.138 37.69% ` NORMAL fdatasync 68 317.992 0.065 4.676 82.016 29.37% `

– Robert Špendl
Jan 26 at 15:38













(eventually I'll get better at editing comments, have patience with a noob)

– Robert Špendl
Jan 26 at 15:39





(eventually I'll get better at editing comments, have patience with a noob)

– Robert Špendl
Jan 26 at 15:39













It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

– Robert Špendl
Jan 26 at 15:40





It takes fdatasync on average 300 ms on the slow system and less than 5 ms on the normal.

– Robert Špendl
Jan 26 at 15:40













I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

– Robert Špendl
Jan 26 at 15:57





I have tried a simple fsync benchmark github.com/ncw/fsyncbench Every disk on both slow & normal computers was OK, SSDs performing better (6-10 ms/fsync) and classid discs 20-40 ms/fsync. Any idea what kind of fsync should I try to find what exactly is causing such delays?

– Robert Špendl
Jan 26 at 15:57













An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

– fche
Jan 26 at 16:13





An individual fdatasync should not take anywhere that long to an SSD. Is it worn out? (smartctl -a /dev/sdXYZ)

– fche
Jan 26 at 16:13

















draft saved

draft discarded
















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f496775%2fextremely-slow-dnf%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown






Popular posts from this blog

How to check contact read email or not when send email to Individual?

Bahrain

Postfix configuration issue with fips on centos 7; mailgun relay