Is it possible to save binaries that pip compiles while installing a package?
Clash Royale CLAN TAG#URR8PPP
up vote
1
down vote
favorite
I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.
Running pip install cython
on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top
shows cc running, trying to compile stuff, using 100% of one core.
I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.
When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?
If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable
arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.
compiling pip virtualenv
add a comment |Â
up vote
1
down vote
favorite
I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.
Running pip install cython
on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top
shows cc running, trying to compile stuff, using 100% of one core.
I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.
When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?
If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable
arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.
compiling pip virtualenv
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, usepip show -f somepackage
to see wherepip
placed all the files resulting from compilingsompackage
â GracefulRestart
Jun 26 at 20:55
add a comment |Â
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.
Running pip install cython
on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top
shows cc running, trying to compile stuff, using 100% of one core.
I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.
When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?
If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable
arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.
compiling pip virtualenv
I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.
Running pip install cython
on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top
shows cc running, trying to compile stuff, using 100% of one core.
I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.
When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?
If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable
arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.
compiling pip virtualenv
asked Jun 26 at 18:59
Tal
592721
592721
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, usepip show -f somepackage
to see wherepip
placed all the files resulting from compilingsompackage
â GracefulRestart
Jun 26 at 20:55
add a comment |Â
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, usepip show -f somepackage
to see wherepip
placed all the files resulting from compilingsompackage
â GracefulRestart
Jun 26 at 20:55
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use
pip show -f somepackage
to see where pip
placed all the files resulting from compiling sompackage
â GracefulRestart
Jun 26 at 20:55
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use
pip show -f somepackage
to see where pip
placed all the files resulting from compiling sompackage
â GracefulRestart
Jun 26 at 20:55
add a comment |Â
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
Just copy the wheel file from the cache dir. When pip install pkgname
is issued and pip
doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue
$ find $HOME/.cache/pip -type f -name Cython*.whl
Backup the file; to install from the local file, just issue
$ pip install path/to/file.whl
Where to go from here: local repository
When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:
âÂÂâÂÂâ repodir
âÂÂâÂÂâ Cython
â âÂÂâÂÂâ Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
...
Start the server with e.g.
$ python3 -m http.server -p 9000
Now you can pass the repo to pip
:
$ pip install Cython --extra-index-url=http://127.0.0.1:9000
or even persist the repo URL in the pip.conf
to not to enter it each time:
# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000
Should you look for more, there are lots of fancy PyPI repo servers out there like devpi
which offer versatile package management, web UI etc.
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
add a comment |Â
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
Just copy the wheel file from the cache dir. When pip install pkgname
is issued and pip
doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue
$ find $HOME/.cache/pip -type f -name Cython*.whl
Backup the file; to install from the local file, just issue
$ pip install path/to/file.whl
Where to go from here: local repository
When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:
âÂÂâÂÂâ repodir
âÂÂâÂÂâ Cython
â âÂÂâÂÂâ Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
...
Start the server with e.g.
$ python3 -m http.server -p 9000
Now you can pass the repo to pip
:
$ pip install Cython --extra-index-url=http://127.0.0.1:9000
or even persist the repo URL in the pip.conf
to not to enter it each time:
# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000
Should you look for more, there are lots of fancy PyPI repo servers out there like devpi
which offer versatile package management, web UI etc.
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
add a comment |Â
up vote
1
down vote
accepted
Just copy the wheel file from the cache dir. When pip install pkgname
is issued and pip
doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue
$ find $HOME/.cache/pip -type f -name Cython*.whl
Backup the file; to install from the local file, just issue
$ pip install path/to/file.whl
Where to go from here: local repository
When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:
âÂÂâÂÂâ repodir
âÂÂâÂÂâ Cython
â âÂÂâÂÂâ Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
...
Start the server with e.g.
$ python3 -m http.server -p 9000
Now you can pass the repo to pip
:
$ pip install Cython --extra-index-url=http://127.0.0.1:9000
or even persist the repo URL in the pip.conf
to not to enter it each time:
# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000
Should you look for more, there are lots of fancy PyPI repo servers out there like devpi
which offer versatile package management, web UI etc.
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
add a comment |Â
up vote
1
down vote
accepted
up vote
1
down vote
accepted
Just copy the wheel file from the cache dir. When pip install pkgname
is issued and pip
doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue
$ find $HOME/.cache/pip -type f -name Cython*.whl
Backup the file; to install from the local file, just issue
$ pip install path/to/file.whl
Where to go from here: local repository
When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:
âÂÂâÂÂâ repodir
âÂÂâÂÂâ Cython
â âÂÂâÂÂâ Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
...
Start the server with e.g.
$ python3 -m http.server -p 9000
Now you can pass the repo to pip
:
$ pip install Cython --extra-index-url=http://127.0.0.1:9000
or even persist the repo URL in the pip.conf
to not to enter it each time:
# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000
Should you look for more, there are lots of fancy PyPI repo servers out there like devpi
which offer versatile package management, web UI etc.
Just copy the wheel file from the cache dir. When pip install pkgname
is issued and pip
doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue
$ find $HOME/.cache/pip -type f -name Cython*.whl
Backup the file; to install from the local file, just issue
$ pip install path/to/file.whl
Where to go from here: local repository
When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:
âÂÂâÂÂâ repodir
âÂÂâÂÂâ Cython
â âÂÂâÂÂâ Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
...
Start the server with e.g.
$ python3 -m http.server -p 9000
Now you can pass the repo to pip
:
$ pip install Cython --extra-index-url=http://127.0.0.1:9000
or even persist the repo URL in the pip.conf
to not to enter it each time:
# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000
Should you look for more, there are lots of fancy PyPI repo servers out there like devpi
which offer versatile package management, web UI etc.
edited Jul 1 at 23:26
answered Jun 28 at 8:26
hoefling
398310
398310
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
add a comment |Â
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
â Tal
Jun 28 at 14:40
add a comment |Â
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f452082%2fis-it-possible-to-save-binaries-that-pip-compiles-while-installing-a-package%23new-answer', 'question_page');
);
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use
pip show -f somepackage
to see wherepip
placed all the files resulting from compilingsompackage
â GracefulRestart
Jun 26 at 20:55