mjhjmtu

I use wget's built-in spider mode as a convenience sometimes to quickly check a local site for broken links. This morning I turned its attention to a production site that we'd just put major changes on, and it's coming up with 3 broken links, but it seems impossible to tell where they are! (It only says what they're linking to and there's no straightforward way of relating that alone back to a page.)

The options I'm currently using are wget -r -nv --spider http://www.domain.com/ -o /path/to/log.txt. Does anyone know of an option I'm overlooking, a way to read the output, or even a simple substitute for this command that will also let me know what file the links appear in (and ideally a line #)?

You should be able to watch the web server logs, in conjunction with the wget run. Look for the 404's in the log file and pull the referrer field. That will tell you the page that contains the broken link.

It should then just be a matter of examining that page for the offending link.

A good way (not involving the webserver logs) is to use the --debug flag and grep for ^Referer:

On the command line:

wget -r -nv --spider http://www.domain.com/ 2>&1 | egrep -A 1 '(^---response end---$|^--[0-9]4-[0-9]2-[0-9]2|^[0-9]4-[0-9]2-[0-9]2 ERROR|^Referer:|^Remote file does not)'

You can do similar grepping on your log. Caveat: some wget are not compiled with the support for --debug