What are the benefits of running a docker container inside a VM vs running docker containers on bare metal?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
2
down vote

favorite












What are the benefits of running a docker container inside a VM vs running docker containers on bare metal (on the host directly)?



I have heard of companies running docker containers inside of a VM, particularly it has been mentioned in docker conferences that some organizations are doing it. Why?



( Comparing Docker container running on host vs Docker container running inside KVM on host )



  • Both Docker and KVM have ways to save their current state, no added benefit here

  • Both Docker and KVM can be provided separate IP's for network use

  • Both Docker and KVM separate running programs and installs from conflicting with host running processes

  • Both Docker and KVM provide easy ways to scale with enterprise growth

  • Both Provide simple methods of moving instances to different hosts

So why would anyone run Docker inside a KVM? Wouldn't they be taking a unnecessary performance hit from the KVM?










share|improve this question























  • Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
    – Ulrich Schwarz
    Aug 15 at 14:45










  • Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
    – A.B
    Aug 15 at 15:50















up vote
2
down vote

favorite












What are the benefits of running a docker container inside a VM vs running docker containers on bare metal (on the host directly)?



I have heard of companies running docker containers inside of a VM, particularly it has been mentioned in docker conferences that some organizations are doing it. Why?



( Comparing Docker container running on host vs Docker container running inside KVM on host )



  • Both Docker and KVM have ways to save their current state, no added benefit here

  • Both Docker and KVM can be provided separate IP's for network use

  • Both Docker and KVM separate running programs and installs from conflicting with host running processes

  • Both Docker and KVM provide easy ways to scale with enterprise growth

  • Both Provide simple methods of moving instances to different hosts

So why would anyone run Docker inside a KVM? Wouldn't they be taking a unnecessary performance hit from the KVM?










share|improve this question























  • Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
    – Ulrich Schwarz
    Aug 15 at 14:45










  • Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
    – A.B
    Aug 15 at 15:50













up vote
2
down vote

favorite









up vote
2
down vote

favorite











What are the benefits of running a docker container inside a VM vs running docker containers on bare metal (on the host directly)?



I have heard of companies running docker containers inside of a VM, particularly it has been mentioned in docker conferences that some organizations are doing it. Why?



( Comparing Docker container running on host vs Docker container running inside KVM on host )



  • Both Docker and KVM have ways to save their current state, no added benefit here

  • Both Docker and KVM can be provided separate IP's for network use

  • Both Docker and KVM separate running programs and installs from conflicting with host running processes

  • Both Docker and KVM provide easy ways to scale with enterprise growth

  • Both Provide simple methods of moving instances to different hosts

So why would anyone run Docker inside a KVM? Wouldn't they be taking a unnecessary performance hit from the KVM?










share|improve this question















What are the benefits of running a docker container inside a VM vs running docker containers on bare metal (on the host directly)?



I have heard of companies running docker containers inside of a VM, particularly it has been mentioned in docker conferences that some organizations are doing it. Why?



( Comparing Docker container running on host vs Docker container running inside KVM on host )



  • Both Docker and KVM have ways to save their current state, no added benefit here

  • Both Docker and KVM can be provided separate IP's for network use

  • Both Docker and KVM separate running programs and installs from conflicting with host running processes

  • Both Docker and KVM provide easy ways to scale with enterprise growth

  • Both Provide simple methods of moving instances to different hosts

So why would anyone run Docker inside a KVM? Wouldn't they be taking a unnecessary performance hit from the KVM?







linux debian virtual-machine docker kvm






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Aug 19 at 14:04









Jeff Schaller

32.6k849110




32.6k849110










asked Aug 15 at 13:48









TrevorKS

1238




1238











  • Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
    – Ulrich Schwarz
    Aug 15 at 14:45










  • Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
    – A.B
    Aug 15 at 15:50

















  • Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
    – Ulrich Schwarz
    Aug 15 at 14:45










  • Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
    – A.B
    Aug 15 at 15:50
















Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
– Ulrich Schwarz
Aug 15 at 14:45




Never underestimate structural inertia – "we already have all the workflow and infrastructure for VMs working and in place, but the customer insists they can only do docker, so we give them a VM they can run their docker image in".
– Ulrich Schwarz
Aug 15 at 14:45












Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
– A.B
Aug 15 at 15:50





Docker security model is a bit more flaky: if you can access the "Docker socket" you can consider yourself as root on the system (eg: by running a privileged container that will mount / and remove the hosts's root password)
– A.B
Aug 15 at 15:50











2 Answers
2






active

oldest

votes

















up vote
4
down vote



accepted










Regarding your main points:



Both Docker and KVM have ways to save their current state, no added benefit here



Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.



Both Docker and KVM can be provided separate IP's for network use



Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.



Both Docker and KVM separate running programs and installs from conflicting with host running processes



VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.



Both Docker and KVM provide easy ways to scale with enterprise growth



This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).



Both Provide simple methods of moving instances to different hosts



No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).



Some extra points:



  • VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.

  • VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).

  • VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.

  • VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.

  • Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.





share|improve this answer






















  • VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
    – Michael Hampton
    Aug 15 at 20:04










  • @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
    – Austin Hemmelgarn
    Aug 16 at 12:39

















up vote
1
down vote













Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts.
Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.



So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.



This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers






share|improve this answer




















    Your Answer







    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "106"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: false,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













     

    draft saved


    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f462757%2fwhat-are-the-benefits-of-running-a-docker-container-inside-a-vm-vs-running-docke%23new-answer', 'question_page');

    );

    Post as a guest






























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    4
    down vote



    accepted










    Regarding your main points:



    Both Docker and KVM have ways to save their current state, no added benefit here



    Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.



    Both Docker and KVM can be provided separate IP's for network use



    Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.



    Both Docker and KVM separate running programs and installs from conflicting with host running processes



    VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.



    Both Docker and KVM provide easy ways to scale with enterprise growth



    This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).



    Both Provide simple methods of moving instances to different hosts



    No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).



    Some extra points:



    • VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.

    • VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).

    • VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.

    • VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.

    • Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.





    share|improve this answer






















    • VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
      – Michael Hampton
      Aug 15 at 20:04










    • @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
      – Austin Hemmelgarn
      Aug 16 at 12:39














    up vote
    4
    down vote



    accepted










    Regarding your main points:



    Both Docker and KVM have ways to save their current state, no added benefit here



    Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.



    Both Docker and KVM can be provided separate IP's for network use



    Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.



    Both Docker and KVM separate running programs and installs from conflicting with host running processes



    VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.



    Both Docker and KVM provide easy ways to scale with enterprise growth



    This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).



    Both Provide simple methods of moving instances to different hosts



    No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).



    Some extra points:



    • VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.

    • VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).

    • VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.

    • VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.

    • Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.





    share|improve this answer






















    • VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
      – Michael Hampton
      Aug 15 at 20:04










    • @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
      – Austin Hemmelgarn
      Aug 16 at 12:39












    up vote
    4
    down vote



    accepted







    up vote
    4
    down vote



    accepted






    Regarding your main points:



    Both Docker and KVM have ways to save their current state, no added benefit here



    Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.



    Both Docker and KVM can be provided separate IP's for network use



    Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.



    Both Docker and KVM separate running programs and installs from conflicting with host running processes



    VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.



    Both Docker and KVM provide easy ways to scale with enterprise growth



    This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).



    Both Provide simple methods of moving instances to different hosts



    No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).



    Some extra points:



    • VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.

    • VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).

    • VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.

    • VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.

    • Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.





    share|improve this answer














    Regarding your main points:



    Both Docker and KVM have ways to save their current state, no added benefit here



    Except that how they store their state is different, and one method or the other may be more efficient. Also, you can't reliably save 100% of the state of a container.



    Both Docker and KVM can be provided separate IP's for network use



    Depending on what VM and container system you use, this may be easier to set up for VM's than for containers. This is especially true if you want a dedicated layer 2 interface for the VM/container, which is almost always easier to do with a VM.



    Both Docker and KVM separate running programs and installs from conflicting with host running processes



    VM's do it better than containers. Containers are still making native system calls to the host OS. That means they can potentially directly exploit any bugs in those system calls. VM's have their own OS, so they're much better isolated.



    Both Docker and KVM provide easy ways to scale with enterprise growth



    This is about even, though I've personally found that VM's done right scale a bit better than containers done right (most likely because VM's done right offload the permissions issues to the hardware, while containers need software to handle it).



    Both Provide simple methods of moving instances to different hosts



    No, not exactly. Both can do offline migration, but a lot of container systems can't do live migration (that is, moving a running container from one host to another). Live migration is very important for manageability reasons if you're running at any reasonable scale (Need to run updates on the host? Migrate everything to another system, reboot the host, migrate everything off of the second host to the first, reboot that, rebalance.).



    Some extra points:



    • VM's generally have easier to work with high-availability options. This isn't to say that containers don't have such options, just that they're typically easier to work with and adapt application code to with VM's.

    • VM's are a bit easier to migrate directly to and from cloud hosting (you don't have to care to quite the same degree what the underlying hosting environment is like).

    • VM's let you run a different platform from the host OS. Even different Linux distributions have sufficient differences in their kernel configuration that stuff written for one is not completely guaranteed to work on another.

    • VM's give you better control of the potential attack surface. With containers, you just can't get rid of the fact that the code for your host OS is still in memory, and therefore a potential attack vector. With VM's, you're running an isolated OS, so you can strip it down to the absolute minimum of what you actually need.

    • Running a group of related containers together in a VM gives you an easy foolproof way to start and stop that group of containers together.






    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Aug 19 at 14:03









    Jeff Schaller

    32.6k849110




    32.6k849110










    answered Aug 15 at 19:55









    Austin Hemmelgarn

    5,2141915




    5,2141915











    • VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
      – Michael Hampton
      Aug 15 at 20:04










    • @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
      – Austin Hemmelgarn
      Aug 16 at 12:39
















    • VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
      – Michael Hampton
      Aug 15 at 20:04










    • @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
      – Austin Hemmelgarn
      Aug 16 at 12:39















    VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
    – Michael Hampton
    Aug 15 at 20:04




    VMs get even more advantageous when you start doing multiple-machine orchestration with something like Kubernetes.
    – Michael Hampton
    Aug 15 at 20:04












    @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
    – Austin Hemmelgarn
    Aug 16 at 12:39




    @MichaelHampton It really depends on the orchestration tool and the underlying container or VM technology. Ansible for example is actually more efficient working with local Docker containers for example than VM's, because it has lower connection overhead.
    – Austin Hemmelgarn
    Aug 16 at 12:39












    up vote
    1
    down vote













    Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts.
    Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.



    So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.



    This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers






    share|improve this answer
























      up vote
      1
      down vote













      Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts.
      Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.



      So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.



      This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers






      share|improve this answer






















        up vote
        1
        down vote










        up vote
        1
        down vote









        Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts.
        Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.



        So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.



        This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers






        share|improve this answer












        Virtualization came to be when hardware became to large for single loads, and a need to make a single powerful machine do more than one task, with the tasks being isolated from each other became apparent. VMs do a great job doing exactly that - splitting a large host into multiple smaller hosts.
        Containers are a way not to split a host into usable chunks, but a way to isolate specific apps and workloads.



        So, having a powerful machine, you might want to split it into several smaller parts with maybe different properties (resources, OS etc), and then run your apps inside those VMs as it makes sense for the use case.



        This is not the absolute and only way to do things of course, but it is the most obvious case for mixing VMs and containers







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Aug 16 at 7:40









        dyasny

        93247




        93247



























             

            draft saved


            draft discarded















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f462757%2fwhat-are-the-benefits-of-running-a-docker-container-inside-a-vm-vs-running-docke%23new-answer', 'question_page');

            );

            Post as a guest













































































            Popular posts from this blog

            How to check contact read email or not when send email to Individual?

            Displaying single band from multi-band raster using QGIS

            How many registers does an x86_64 CPU actually have?