Is it possible to save binaries that pip compiles while installing a package?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
1
down vote

favorite












I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.



Running pip install cython on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top shows cc running, trying to compile stuff, using 100% of one core.



I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.



When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?



If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.




compiling pip virtualenv




share|improve this question



















  • If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
    – GracefulRestart
    Jun 26 at 20:55














up vote
1
down vote

favorite












I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.



Running pip install cython on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top shows cc running, trying to compile stuff, using 100% of one core.



I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.



When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?



If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.




compiling pip virtualenv




share|improve this question



















  • If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
    – GracefulRestart
    Jun 26 at 20:55












up vote
1
down vote

favorite









up vote
1
down vote

favorite











I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.



Running pip install cython on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top shows cc running, trying to compile stuff, using 100% of one core.



I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.



When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?



If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.




compiling pip virtualenv




share|improve this question











I'm working on a project for a Raspberry Pi 3.
One of the things my project requires is cython.



Running pip install cython on a Raspberry Pi inside of a virtualenv takes about 40 minutes to complete, as cython apparently needs to be compiled first. During these 40 minutes, top shows cc running, trying to compile stuff, using 100% of one core.



I need to do this several times every day, as I keep needing to reset the SD card to a fresh image of Raspbian. This is obviously very time-consuming, and is slowing down progress.



When pip is done compiling cython, where are the binaries stored? Inside the virtualenv? Somewhere else on the filesystem?



If they are stored in the virtualenv, can I archive the entire virtualenv folder, and restore it to a fresh Linux, and expect it to work? I know virtualenvs aren't supposed to be portable (without using the --relocatable arg, which seems to have its own issues), but in my case, the path of the virtualenv will remain the same when I restore it.





compiling pip virtualenv





share|improve this question










share|improve this question




share|improve this question









asked Jun 26 at 18:59









Tal

592721




592721











  • If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
    – GracefulRestart
    Jun 26 at 20:55
















  • If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
    – GracefulRestart
    Jun 26 at 20:55















If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
– GracefulRestart
Jun 26 at 20:55




If you are putting the same binaries in the same distribution with the same hardware, you should only compile it once, save and copy as necessary. Pip will store files in the virtualenv and tracks where it places each file, use pip show -f somepackage to see where pip placed all the files resulting from compiling sompackage
– GracefulRestart
Jun 26 at 20:55










1 Answer
1






active

oldest

votes

















up vote
1
down vote



accepted










Just copy the wheel file from the cache dir. When pip install pkgname is issued and pip doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue



$ find $HOME/.cache/pip -type f -name Cython*.whl


Backup the file; to install from the local file, just issue



$ pip install path/to/file.whl


Where to go from here: local repository



When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:



└── repodir
 ├── Cython
 │ └── Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
 ...


Start the server with e.g.



$ python3 -m http.server -p 9000


Now you can pass the repo to pip:



$ pip install Cython --extra-index-url=http://127.0.0.1:9000


or even persist the repo URL in the pip.conf to not to enter it each time:



# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000


Should you look for more, there are lots of fancy PyPI repo servers out there like devpi which offer versatile package management, web UI etc.






share|improve this answer























  • It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
    – Tal
    Jun 28 at 14:40










Your Answer







StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: false,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);








 

draft saved


draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f452082%2fis-it-possible-to-save-binaries-that-pip-compiles-while-installing-a-package%23new-answer', 'question_page');

);

Post as a guest

















1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
1
down vote



accepted










Just copy the wheel file from the cache dir. When pip install pkgname is issued and pip doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue



$ find $HOME/.cache/pip -type f -name Cython*.whl


Backup the file; to install from the local file, just issue



$ pip install path/to/file.whl


Where to go from here: local repository



When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:



└── repodir
 ├── Cython
 │ └── Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
 ...


Start the server with e.g.



$ python3 -m http.server -p 9000


Now you can pass the repo to pip:



$ pip install Cython --extra-index-url=http://127.0.0.1:9000


or even persist the repo URL in the pip.conf to not to enter it each time:



# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000


Should you look for more, there are lots of fancy PyPI repo servers out there like devpi which offer versatile package management, web UI etc.






share|improve this answer























  • It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
    – Tal
    Jun 28 at 14:40














up vote
1
down vote



accepted










Just copy the wheel file from the cache dir. When pip install pkgname is issued and pip doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue



$ find $HOME/.cache/pip -type f -name Cython*.whl


Backup the file; to install from the local file, just issue



$ pip install path/to/file.whl


Where to go from here: local repository



When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:



└── repodir
 ├── Cython
 │ └── Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
 ...


Start the server with e.g.



$ python3 -m http.server -p 9000


Now you can pass the repo to pip:



$ pip install Cython --extra-index-url=http://127.0.0.1:9000


or even persist the repo URL in the pip.conf to not to enter it each time:



# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000


Should you look for more, there are lots of fancy PyPI repo servers out there like devpi which offer versatile package management, web UI etc.






share|improve this answer























  • It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
    – Tal
    Jun 28 at 14:40












up vote
1
down vote



accepted







up vote
1
down vote



accepted






Just copy the wheel file from the cache dir. When pip install pkgname is issued and pip doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue



$ find $HOME/.cache/pip -type f -name Cython*.whl


Backup the file; to install from the local file, just issue



$ pip install path/to/file.whl


Where to go from here: local repository



When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:



└── repodir
 ├── Cython
 │ └── Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
 ...


Start the server with e.g.



$ python3 -m http.server -p 9000


Now you can pass the repo to pip:



$ pip install Cython --extra-index-url=http://127.0.0.1:9000


or even persist the repo URL in the pip.conf to not to enter it each time:



# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000


Should you look for more, there are lots of fancy PyPI repo servers out there like devpi which offer versatile package management, web UI etc.






share|improve this answer















Just copy the wheel file from the cache dir. When pip install pkgname is issued and pip doesn't find a compatible wheel to install from, it builds one from the source dist and stores it in cache to reuse for future reinstalls. To find the wheel file, issue



$ find $HOME/.cache/pip -type f -name Cython*.whl


Backup the file; to install from the local file, just issue



$ pip install path/to/file.whl


Where to go from here: local repository



When you have a lot of prebuilt wheels collected, the next step would be setting up a local PyPI repository to install from. The simplest way is to organize the wheels into package-named dirs and run a simple HTTP server:



└── repodir
 ├── Cython
 │ └── Cython-0.28.0-cp36-cp36m-linux_aarch64.whl
 ...


Start the server with e.g.



$ python3 -m http.server -p 9000


Now you can pass the repo to pip:



$ pip install Cython --extra-index-url=http://127.0.0.1:9000


or even persist the repo URL in the pip.conf to not to enter it each time:



# pip.conf
[global]
extra-index-url=http://127.0.0.1:9000


Should you look for more, there are lots of fancy PyPI repo servers out there like devpi which offer versatile package management, web UI etc.







share|improve this answer















share|improve this answer



share|improve this answer








edited Jul 1 at 23:26


























answered Jun 28 at 8:26









hoefling

398310




398310











  • It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
    – Tal
    Jun 28 at 14:40
















  • It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
    – Tal
    Jun 28 at 14:40















It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
– Tal
Jun 28 at 14:40




It looks like when using virtualenvs, the wheel files are stored inside the virtual env. Archiving the virtualenv with tar, and later restoring it to a clean Linux worked with no issues, and sped up the process of setting up a fresh Linux considerably. Thanks for the idea of a local repo - I might do that.
– Tal
Jun 28 at 14:40












 

draft saved


draft discarded


























 


draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f452082%2fis-it-possible-to-save-binaries-that-pip-compiles-while-installing-a-package%23new-answer', 'question_page');

);

Post as a guest
































































-

Popular posts from this blog

How to check contact read email or not when send email to Individual?

Image
Clash Royale CLAN TAG #URR8PPP up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Sep 26 at 0:12 ToanLuong 49 9 add a comment  |  up vote 1 down vote favorite I'm using WordPress 4.9.8, CiviCRM to 5.5.1, I usually send email to contact by Search> Find contacts View contact details Action> Send email Send email ok, Contact received mail ok like picture But status only Email sent though contact read email or not. So, can CiviCRM can change status to Email read when contact read email? wordpress email share | improve this question asked Se
Read more

Bahrain

Image
For other uses, see Bahrain (disambiguation). Sovereign island state in the Persian Gulf Kingdom of Bahrain مملكة البحرين   (Arabic) Mamlakat al-Baḥrayn Flag Coat of arms Anthem:  نشيد البحرين الوطني Bahrainona Our Bahrain Location of   Bahrain    (circled in red) Capital and largest city Manama 26°13′N 50°35′E  /  26.217°N 50.583°E  / 26.217; 50.583 Official languages Arabic Ethnic groups (2010 [1] ) 50.7% Arab (46% Bahrani, 4.7% other Arabs) 45.5% Asian 1% European 1.2% Others Religion Islam Demonym(s) Bahraini Government Unitary constitutional monarchy • King Hamad bin Isa Al Khalifa • Crown Prince Salman bin Hamad bin Isa Al Khalifa • Prime Minister Khalifa bin Salman Al Khalifa Legislature National Assembly • Upper house Consultative Council • Lower house Council of Representatives Independence • Declared Independence [2] 14 August 1971 • from United Kingdom [1] 15 August 1971 • Admitted to the United Nations 21 September 1971 • Kingdom of Bahrain 14 February 2002
Read more

Postfix configuration issue with fips on centos 7; mailgun relay

Image
Clash Royale CLAN TAG #URR8PPP up vote 0 down vote favorite I am trying to setup postfix to relay all mail generated on the local machine via SMTP to a mailgun relay. I have used the mailgun relay before with success on an ubuntu server, but I am migrating to a Centos 7 server which I will be running in FIPS mode. There error log is below, slightly sanitized. I have a small enough network that I choose to have each machine reach out to mailgun individually (this the loopback-only, 127.0.0.0/8 restrictions) and no firewall open port allowing smtp in to the machine. I assume the FIPS mode (and with it disabling of MD5) is causing problems, but I don't know how to overcome it or if it is even possible for tls_fprint to use some supported hash such as sha256 or sha512. However, the relay=none is slightly concerning since I have relayhost set, but perhaps that is because the smtp process is failing? Any help would be appreciated! postconf -n: alias_database = hash:/etc/alia
Read more