mjhjmtu

Because I've heard that it's a very bad idea to install packages through pip instead of through a package manager, I'm trying to identify all the packages that I've installed through pip that are also readily available in pacman's official repositories. Not all of them are (I understand that I can explore AUR or package them myself, but for right now I'm trying to stay within the official repositories), so I'm leaving some of them installed through pip for now, and migrating the rest to pacman.

My (admittedly primitive) workflow to do this has been:

1. 
   pip freeze | less; choose a package to migrate (essentially arbitrary)


2. 
   pacman -Ss python-packagename (If unfound, abort and retry with a different package)


3. sudo pip uninstall packagename


4. sudo pacman -S python-packagename


5. If pacman reports file conflicts, repeat steps 2 through 4 for all dependent packages before reattempting step 5 on the target package.

This works, insofar as I believe I'm successfully now managing the package through pacman - pacman -Ql python-packagename lists files that appear correct, and anecdotally I've seen packages updated through pacman as well, indicating that pacman has taken over responsibility for updating the packages.

The problem, though, is that pip freeze still lists the packages, even though they've been explicitly uninstalled through pip.

As an example, both pip freeze | grep pylint and pacman -Qqe | grep pylint indicate that the package is installed. I can successfully uninstall the package through either sudo pacman -Rs python-pylint or through sudo pip uninstall pylint.

If I do so - uninstall the package either through pip or through pacman - the package disappears from pip freeze's list, and pacman -Qqe no longer finds anything. If I then reinstall the package through pacman -S, the package then reappears in pip freeze. Reinstalling the package through pip does not prompt it to reappear in pacman -Qqe.

Uninstalling a package from pip by running sudo pip uninstall packagename, then reinstalling it with the --user flag (i.e. pip install --user packagename) causes sudo pip freeze not to display the package, but pip freeze to continue to display it. Reinstalling the package through pacman causes it to reappear on both lists.

I do not believe this question to be a duplicate of the one at this link; in that question, the user was unable to remove packages from pip freeze by uninstalling them and the problem was tracked to multiple different pip binaries.

In this case, uninstalling a package through pip does remove it from pip freeze, and which pip and sudo which pip have the same output: /usr/bin/pip. Additionally, sudo pip freeze and pip freeze produce the same output.

However, the test proposed by the accepted answer to the question linked above exhibits what I believe to be strange behavior. (At the time of running the below test, pylint had been uninstalled from pip and reinstalled through pacman.)

$ python -c 'import pkg_name' &> /dev/null && echo installed || echo not installed
Using config file /home/[my user name]/.pylintrc
not installed

The package is indeed installed - not only does it print the config file line, but I can use the package as normal (pylint foo.py). Wouldn't I then expect the test to print installed?

To get to my actual questions about this issue:

• What can I do to ascertain which packages are actually installed only through pip, and which are "installed" through both pip and pacman?


• What can be done to correct the broader problem: pip tracking packages installed through pacman?

I wrote a python script (gist link) that checks all python packages whether they were installed with pacman.

installation

Download the script and make it executable:

$ wget https://gist.github.com/hoefling/314565368a66c308b4d7d407a3028cb7/raw/7b81553fa0a84b92a90fbaa0746482d0ec18516c/pip-query -O pip-query
$ chmod +x pip-query

usage

Here's an example calling it in a base/archlinux docker container where I have installed python-pip and python-wheel:

$ ./pip-query
Package Version Owner
---------- ------- -----------------
appdirs 1.4.3 python-appdirs
packaging 17.1 python-packaging
pip 9.0.1 python-pip
pyparsing 2.2.0 python-pyparsing
setuptools 39.2.0 python-setuptools
six 1.11.0 python-six
wheel 0.31.1 python-wheel

The output resembles the output of pip list --format=columns, but with an additional column Owner that prints the name of the system package that owns the python package.

Let's test with a package installed directly with pip:

$ sudo pip install tqdm
...
$ ./pip-query
Package Version Owner
---------- ------- -----------------
appdirs 1.4.3 python-appdirs
packaging 17.1 python-packaging
pip 9.0.1 python-pip
pyparsing 2.2.0 python-pyparsing
setuptools 39.2.0 python-setuptools
six 1.11.0 python-six
tqdm 4.23.4
wheel 0.31.1 python-wheel

The Owner column has an empty cell in tqdm row, this indicates tqdm was not installed by pacman. Uninstall tqdm with pip:

$ sudo pip uninstall -y tqdm

and install it with pacman:

$ pacman -S python-tqdm

permanent installation

If you want, place pip-query file somewhere in your PATH, preferably something local like $HOME/.local/bin to call it like a proper executable with

$ pip-query

adapting for other package managers

It should be pretty easy to adapt the script for using with other package managers if you want to. For example, if I change ['pacman', '-Qqo', file] to ['qfile', '-q', file], the script will work on Gentoo without any further modification*. If your package manager doesn't support pure package name printing, extract the relevant info from the out string with regex or whatever.

batch reinstalling packages with pacman

You can modify pip-query, or follow the Unix philosophy and write some bash command that does reinstalling for you. However, you have to deal with multiple suggestions made by pacman (as in the example below) and implement the case when nothing was suggested. Here is my humble attempt (not an expert in bash):

$ ./pip-query | tail -n +3 | while read line ; do split=($line); 
> if [ "$#split[@]" -eq "2" ]; then pkgname=$split[0]; 
> echo -e "Suggestion: pip uninstall -y $pkgname && pacman -S $(pacman -Sspq $pkgname | tr 'n' ' ')"; 
> fi; done
Suggestion: pip uninstall -y tqdm && pacman -S python-tqdm python2-tqdm

Enhance this and you have an automated solution for reinstall task.

_{*: assuming app-portage/portage-utils is emerged.}