mjhjmtu

Sometimes you need to unmount a filesystem or detach a loop device but it is busy because of open file descriptors, perhaps because of a smb server process.

To force the unmount, you can kill the offending process (or try kill -SIGTERM), but that would close the smb connection (even though some of the files it has open do not need to be closed).

A hacky way to force a process to close a given file descriptor is described here using gdb to call close(fd).
This seems dangerous, however. What if the closed descriptor is recycled? The process might use the old stored descriptor not realizing it now refers to a totally different file.

I have an idea, but don't know what kind of flaws it has: using gdb, open /dev/null with O_WRONLY (edit: an comment suggested O_PATH as a better alternative), then dup2 to close the offending file descriptor and reuse its descriptor for /dev/null. This way any reads or writes to the file descriptor will fail.

Like this:

sudo gdb -p 234532
(gdb) set $dummy_fd = open("/dev/null", 0x200000) // O_PATH
(gdb) p dup2($dummy_fd, offending_fd)
(gdb) p close($dummy_fd)
(gdb) detach
(gdb) quit

What could go wrong?

Fiddling with a process with gdb is almost never safe though may be
necessary if there's some emergency and the process needs to stay open
and all the risks and code involved is understood.

Most often I would simply terminate the process, though some cases may
be different and could depend on the environment, who owns the
relevant systems and process involved, what the process is doing,
whether there is documentation on "okay to kill it" or "no, contact
so-and-so first", etc. These details may need to be worked out in a
post-mortem meeting once the dust settles. If there is a planned
migration it would be good in advance to check whether any processes
have problematic file descriptors open so those can be dealt with in a
non-emergency setting (cron jobs or other scheduled tasks that run
only in the wee hours when migrations may be done are easily missed if
you check only during daytime hours).

Write-only versus Read versus Read-Write

Your idea to reopen the file descriptor O_WRONLY is problematic as not
all file descriptors are write-only. John Viega and Matt Messier take a
more nuanced approach in the "Secure Programming Cookbook for C and C++"
book and handle standard input differently than standard out and
standard error (p. 25, "Managing File Descriptors Safely"):

static int open_devnull(int fd) 
 FILE *f = 0;

 if (!fd) f = freopen(_PATH_DEVNULL, "rb", stdin);
 else if (fd == 1) f = freopen(_PATH_DEVNULL, "wb", stdout);
 else if (fd == 2) f = freopen(_PATH_DEVNULL, "wb", stderr);
 return (f && fileno(f) == fd);

In the gdb case the descriptor (or also FILE * handle) would need to
be checked whether it is read-only or read-write or write-only and an
appropriate replacement opened on /dev/null. If not, a once read-only
handle that is now write-only will cause needless errors should the
process attempt to read from that.

What Could Go Wrong?

How exactly a process behaves when its file descriptors (and likely also
FILE * handles) are fiddled behind the scenes will depend on the
process and will vary from "no big deal" should that descriptor never be
used to "nightmare mode" where there is now a corrupt file somewhere due
to unflushed data, no file-was-properly-closed indicator, or some other
unanticipated problem.

For FILE * handles the addition of a fflush(3) call before closing
the handle may help, or may cause double buffering or some other issue;
this is one of the several hazards of making random calls in gdb
without knowing exactly what the source code does and expects. Software
may also have additional layers of complexity built on top of fd
descriptors or the FILE * handles that may also need to be dealt with.
Monkey patching the code could turn into a monkey wrench easily enough.

Summary

Sending a process a standard terminate signal should give it a chance
to properly close out resources, same as when a system shuts down
normally. Fiddling with a process with gdb will likely not properly
close things out, and could make the situation very much worse.

open /dev/null with O_WRONLY, then dup2 to close the offending file descriptor and reuse it's descriptor for /dev/null. This way any reads or writes to the file descriptor will fail.

If you dup a descriptor to /dev/null, any writes will not fail, but succeed, and the reads will succeed and return 0 (eof).

This may or may not be what you want.

On linux, you can also open a file with flags = 3 (O_WRONLY|O_RDWR aka O_NOACCESS) which will cause any read or write to fail with EBADF.

The file will only be available for ioctls -- which brings up a danger not talked about in the other answer and comments: reads and writes are not the only operations done on file descriptors. (what about lseek or ftruncate?).

Update:

I found something better than the undocumented O_WRONLY|O_RDWR: O_PATH = 010000000 / 0x200000. According to the open(2) manpage:

O_PATH (since Linux 2.6.39)
 Obtain a file descriptor that can be used for two purposes: to
 indicate a location in the filesystem tree and to perform opera-
 tions that act purely at the file descriptor level. The file
 itself is not opened, and other file operations (e.g., read(2),
 write(2), fchmod(2), fchown(2), fgetxattr(2), mmap(2)) fail with
 the error EBADF.

 The following operations can be performed on the resulting file
 descriptor:

 * close(2); fchdir(2) (since Linux 3.5); fstat(2) (since Linux
 3.6).

 * Duplicating the file descriptor (dup(2), fcntl(2) F_DUPFD,
 etc.).