I want to remap my AltGR+Right_Windows but I can't figure out how. No matter what I try, pressing the right windows key emits the name keysym when right alt is pressed or not.

These are the contents of the xmodmap file I am testing right now. In my keyboard, 38 is the keycode for "a", 48 is the keycode for aportrophe/doublequote and 134 is the keycode for the right windows key.

keycode 38 = 1 2 3 4 5 6 7 8
keycode 48 = 1 2 3 4 5 6 7 8
keycode 134 = 1 2 3 4 5 6 7 8

When I press Right_Alt+apostrophe, it outputs 5 but Right_Alt+a and Right_Alt+Right_Windows give 1 (both when right alt is pressed and when it isn't) which is not what I wanted.

My current keyboard layout is us(alt-intl) and the right alt is apparently set up to act as AltGr / Iso_Level3_Shift. The output of xmodmap -pm is the following:

xmodmap: up to 4 keys per modifier, (keycodes in parentheses):

shift Shift_L (0x32), Shift_R (0x3e)
lock Caps_Lock (0x42)
control Control_L (0x25), Control_R (0x69)
mod1 Alt_L (0x40), Meta_L (0xcd)
mod2 Num_Lock (0x4d)
mod3 
mod4 Super_L (0x85), Super_R (0x86), Super_L (0xce), Hyper_L (0xcf)
mod5 ISO_Level3_Shift (0x5c), Mode_switch (0xcb)

Using xmodmap to configure individual key mappings

It is time to write down results of my own research.

I thought that I must have missed something in xmodmap and that it is just not very well documented and people are confused. But it turned out that X.Org design regarding XKB and xmodmap is just stupid.

Epic fail: xmodmap

You can use xmodmap to redefine existing mappings as long as those mappings actually exist in your original keyboard layout. In the case described in the question you cannot extend behavior of any keys to use AltGr. You can only change the AltGr keysyms for keycodes that are already using AltGr.

See also: http://blog.azundris.com/archives/193-X-treme-pain-XKB-vs-XModMap.html

Workaround: Mode_switch

This workaround is described in the answer by @Ned64. You can remap AltGr from ISO_Level3_Shift to Mode_switch.

I successfully used the following command line to remap AltGr.

xmodmap -e 'keycode 108 = Mode_switch'

The disadvantage is that it will break your current keyboard layout but you can recreate all the mappings one by one using xmodmap as @Ned64 already mentioned.

Workaround: Modified keyboard layout

I'm using us(cz_sk_de) as my keyboard layout and I tried to modify it by adding configuration for keys I wanted to extend.

key <AB08> [ comma, less, doublelowquotemark, leftdoublequotemark ] ;
key <AB09> [ period, greater, ellipsis, rightdoublequotemark ] ;

(In section xkb_symbols "cz_sk_de" of /usr/share/X11/xkb/symbols/us)

Just reset the keyboard layout to use the modified version.

setxkbmap 'us(cz_sk_de)'

Now you can (1) type Czech and English quotation marks and ellipses using combinations of ,, ., Shift and AltGr keys and/or (2) remap those keys using xmodmap now that they are defined.

The main disadvantage is that setxkbmap doesn't seem to support arbitrary locations for keyboard layouts and therefore you need to write to system configuration instead of your home directory.

Conclusion

This seems to be an example of overengineered and bad design of X.Org where even such a trivial thing as mapping a key code and a combination of modifiers into a symbol turns out to be a problem. The tools don't seem to provide a reasonable way to just change individual key mappings in user configuration without any side effects.

Step one: Make AltGr usable

I found out that AltGr-modified Keys only work on my system if I also remap the AltGr-Key to Mode_switch, like this:

xmodmap -e "keycode 108 = Mode_switch Mode_switch Mode_switch Mode_switch"

If you have a different keyboard you may need to replace the 108 with your code, check by typing

xmodmap -pke | grep Alt_R

then use that number. (The key is normally called Alt_R on systems I know.)

Alternatively, this may also work if Alt_R is not yet assigned to something else:

xmodmap -e "keysym Alt_R = Mode_switch Mode_switch Mode_switch Mode_switch"

Your AltGr modifier will now serve position 3 (with Shift: 4) within your modifier list.

Step two: Assign keys as required

You can assign AltGr-variants for any key of you keyboard you wish. If you have other keys which do not correspond to the desired code now, you need to re-assign these just like your other keys (the ones you tested).

Step three: Convert this into a script

Now just take your xmodmap commands and write all (but only) modifications to your default keyboard into a file - that is faster than loading a complete keymap. Start this script once after each login and whenever the keymap gets reset by some process.

I have experimented with this a lot over the years, under Solaris, IRIX and Linux, and to my knowledge this is the only solution to get AltGr working - and it has worked well for me for more than 20 years.

Apparently, xmodmap does not read the default AltGR as it should, it may be limited to specific keyboard layouts, maybe it's all non-US layouts, either way, the real problem is t hat while you can replace your keyboard layout, some people might not really want to do that as they need to use special characters on the layout they are used to (like scandinavian keyboard layouts which have characters like ðþæöáúíóåů and so on) that would be a pain to configure for other keyboard layouts.

You can run

xmodmap -e "keycode 108 = Mode_switch"

To rebind AltGR to something xmodmap can work with, and indeed, the 3rd assignment of a keycoad will be output when you press AltGR now, however it will completely break the original keyboard layout as no AltGR keybindings will work anymore with the AltGR key.

I have found two workarounds that do not break my existing keyboard layout and does not involve switching layouts, one is to bind Mode_switch to another key.

The ideal key for this is Super_R which is right next to AltGR, the keycode for it is 134. However not all keyboards have them and in my case on a laptop I do not have it either, so I decided to bind it to the last remaining key that I practically never use, Right Control.

xmodmap -e "keycode 105 = Mode_switch"

I have an american physical keyboard layout, but european layout configuration, and the less/greater/bar key (also known as ISO_Backslash, although that is not recognized by xmodmap itself) usually next to Z on such keyboards is missing. I bound that to Menu (135) but it could serve as an alternative to R_Ctrl and Super_R

This is a dirty fix I suppose.

The second workaround took a bit longer to figure out. Using sxhkd (xbindkeys alternative)

If you add this to ~/.config/sxhkd/sxhkdrc:

~ISO_Level3_Shift
 xdotool key Mode_switch

Your AltGR will function as normal, but when it is pressed, Mode_switch gets toggled (kinda like caps lock, but for Mode_switch) and any xmodmap keys that use it will be locked into AltGR mode until you press AltGR again to untoggle it. It should be a simple matter to make a script to untoggle it automatically when the key is released but my attempts to do so were quite buggy, to say the least.