mjhjmtu

I need to have the string directly following/pbs.twimg.com/profile_images/echoed out. This is for a search tool I have created with a bit of help, and everyone recommends sed, but I never really use sed so I don't understand it well. But here is the script:

read -r Username ;
wget -q -O - https://twitter.com/"$Username" | 
 sed -n '/data-screen-name=.'"$Username"'".*data-user-id=/I 
 s/^.*data-screen-name=.'"$Username"'".*data-user-id="([0-9]*)".*$/1/Ip;q'

This script works perfectly fine for most accounts, but if the account is private then it will fail. If it searches instead for/pbs.twimg.com/profile_images/then only the ID will come up and it doesn't come up with different variables like it does in the original script (that is why it selects the row with data-screem-name= I can't use the twitter API because I'm exporting this to people who wouldn't even know how to go about obtaining the API and so I'm trying to make it as simple as possible for the user.

I have looked through several SO/SE posts and I don't believe this is a duplicate, so sorry if it is.

EDIT----
This doesn't work on private profiles because the line that it goes to is not inside the HTML. After going through further I noticed that /pbs.twimg.com/profile_images/ gives the ID of the user and is on both private accounts and open accounts.

Sample data:
Right now, if I were to run this on my account it would work and you would see my ID:

Username="thematrix1o1"
717835108540030976

But if I run it on a private account I will get no ID (because the line isn't there)

Username="touchmytweets"
.

(there is no dot in the report, it's just blank)

Here is the image with what I need it to find http://imgur.com/Yp8Okx7
As you can see, her ID is: 726618076633030656
Small Sample Report
ile_background_color":"C0DEED","profile_background_image_url":"http://abs.twimg.com/images/themes/theme1/bg.png","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme1/bg.png","profile_background_tile":false,"profile_image_url":"http://pbs.twimg.com/profile_images/726618076633030656/wwYbLwbs_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/726618076633030656/wwYbLwbs_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/418265825/1463628965","profile_link_color":"0084B4","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"has_extended_profile":false,"default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":n

Your output contains any amount of random cruft, and one line you are interested in. Select that line, discard anything but the ID and print the result:

 sed -n '/profile_images/s/.*profile_images\/([0-9]+).*/1/p'

This could be made slightly more efficient by quitting immediately after that line has been processed.

In fact, that is pretty much exactly what the original code does. The only thing that has changed is the regular expression used.

When I'm confronted with needle-in-haystack work like this, I like to turn it into a line-oriented problem if I can. You might be able to do that with something like this:

$ sed -E 's:[0-9]+:n&n:g' filename 
| grep -F -A1 '/pbs.twimg.com/profile_images' | tail -1 

That puts any digit-string a line by itself, greps for your string, and prints the one after.

It's quite the hack; the right way to parse HTML is with an HTML parser. But it might get the job done for controlled inputs where you just need one string.

A slightly neater approach would use awk, definitely worth learning if you deal much with such things.