mjhjmtu

I'm working in a team, which develops a c++ (ROS) project.

For some reason, we haven't a good git management.

We have several git branches. To compile the project, I have to git clone the codes from each branch and rearrange the structure of directories.

First, I mkdir -p /home/me/repo, then I git clone the codes from each branch and put all of them into /home/me/repo.

Now I need to rearrange the structure, here is what I've done:

#!/bin/sh

mkdir -p /home/me/project/src
cd /home/me/project/src
catkin_init_workspace # command of ROS to initialize a workspace

cp -r /home/me/repo/robot_dev/control .
cp -r /home/me/repo/robot_dev/control_algo .
cp -r /home/me/repo/sim/third_party .
cp -r /home/me/repo/planning .
cp -r /home/me/repo/robot_dev/cognition/hdmap .

cd ..
catkin_make # command of ROS to compile the project

I created such a script to compile the project and it worked. As you can see, I simply copied and rearranged some directories and compiled.

Now I'm thinking that cp -r is not a good idea because it took too much time. I want to use ln -s to do the same thing. So I wrote another script as below:

#!/bin/sh

mkdir -p /home/me/project2/src
cd /home/me/project2/src
catkin_init_workspace # command of ROS to initialize a workspace

ln -s /home/me/repo/robot_dev/control control
ln -s /home/me/repo/robot_dev/control_algo control_algo
ln -s /home/me/repo/sim/third_party third_party
ln -s /home/me/repo/planning planning
ln -s /home/me/repo/robot_dev/cognition/hdmap hdmap

cd ..
catkin_make # command of ROS to compile the project

However, I got an error:

CMake Error at /opt/ros/kinetic/share/catkin/cmake/catkin_package.cmake:297 (message):
 catkin_package() absolute include dir
 '/home/me/project2/src/hdmap/../third_party/ubuntu1604/opencv-3.3.1/include'

I've checked cd /home/me/project2/src/hdmap/../third_party/ubuntu1604/opencv-3.3.1/include, it does exist.

Is there some reasons why cp -r can compile but ln -s can't?

Accessing .. doesn't really work as you expect when symlinks are involved...

And when you try that in bash, bash tries to be "helpful" and fixes it for you, so the problem does not become aparent.

But, in short, when you go to /home/me/project2/src/hdmap/../third_party, the kernel will first resolve the symlink of "hdmap", getting to /home/me/repo/robot_dev/cognition/hdmap, then look up .. which means the parent directory of that hdmap directory, so /home/me/repo/robot_dev/cognition and then it will try to find a third_party in there.

Considering /home/me/repo/robot_dev/cognition/third_party does not exist (or, if it does, it's not the same as /home/me/repo/sim/third_party, which is what you want), you get a file not found error.

Bash keeps a $PWD variable with the path stored as a string, and so it can help "resolve" the .. references in the shell itself before it passes the kernel a path... That way, it will hide these details from you.

You can disable that behavior using set -P (or set -o physical) in bash. (See bash man page for more details.)

In order to help you with your underlying problem... Perhaps a decent solution is to use cp -rl to copy the trees. The -l option to cp creates hardlinks. That should not be a problem in this case, particularly as I imagine you're not expecting to modify the files. It will still have to traverse the data structure and create each object individually, but it will not have to create any contents...

If you're working on a modern filesystem (such as Btrfs), you can also try cp -r --reflink which creates a CoW (copy-on-write) copy. It's essentially a hardlink, but slightly better since there's no connection between the two names, they're just sharing blocks on the backend, touching one of the files will actually fork them into two separate files. (But I imagine hardlinks are probably good enough for you.)

Perhaps there are some tricks you could do with git, in order to just expose the directory you need in each step... So you can actually clone the parts you need... But that might be harder to accomplish or to maintain... Hopefully cp -rl will be enough for you!

I hope this helps!