Whenever we discuss the key of a piece, we usually describe it as either a major or minor key. And then, based on the intervals of the major or minor scale and the root note we define, we can construct the sharps and flats of that key.

When we play a mode of a major or minor scale, for example the mixolydian mode of the major scale, what key are we in?

Consider the mixolydian mode of the C major scale.
C D E F G A Bb C

If we played a song in this scale, the song would have the same key signature as the F minor scale, (though it would have C as the tonic).

So what key are we in if we play C mixolydian? Are we now in F minor?

If I were to rephrase this question more broadly, it would be this:

If keys of songs defined in terms of the major and minor scales, how do we define our key when we play scales other than the major and minor scales?

It would be F major rather than F minor, as there's an A in C Mixolydian, not A♭.

The reason for mentioning any letter name is to tell what the root, or 'home' is.

So, in, say, C Mixolydian, that's what we're in. The root is C, but the notes are not from C major, or C minor (any of them!) but C Mixolydian.

It just happens that diatonically, all those notes (C D E F G A Bb) are also those from F major. There's only one key sig. for each major key, and they're usually stated at the beginning of each line to remind us of what they are, to save using accidentals instead. They also tell us , indirectly, what the key is likely to be. Although with just the Bb, it may not be F major, it could be D minor - or even C Mixolydian!

Scales and keys are directly related, but not synonymous. We are aware of this when considering the minor key and scales associated. In say C minor, with its 3♭ key sig., there are the notes C D E♭ F G A♭ A♮ B♭ B♮ that can and are all used - they do not make up a single minor scale. They're an amalgam of several different minor scales - all belonging to the key C minor.

So, back to the question. Using the notes of C Mixolydian - we are in C Mixolydian, but a preferable key sig. would be one flat for the written piece; we're not really in F. (and certainly not F minor!!). It could also be said that we're in the Mixolydian mode of F, just to confuse some! And - the defining feature of major keys is the major third, leaving that of minor keys containing a minor third - both from the root note, of course.

'The Mixolydian mode of the scale of C' starts on G. 'C Mixolydian' starts on C, and is a mode of the scale of F.

There is a connection between C Mixolydian and F major. (A pretty strong connection - they use the same notes!) But if we are 'in F major', F is the root. In C Mixoldian C is the root. We've left the world of 'keys'.

In real-world music, it's often very fluid. b7 notes (and bVII chords) co-exist with nat7 notes (and V chords). Is a 'Blues in C' in C Mixolydian (because of the C7), or C major (because of the G7)? Or something else (because of the F7)? Surely it doesn't keep switching key/mode? :-) Probably better to let it be in C major, with some non-diatonic notes.

If a piece is squarely in C mixolydian, the most precise thing you can say is that it's in C mixolydian. If speaking casually, you can say that it's 'in C' as long as the person listening to you is going to interpret that as "the root note of the piece is C", and not "the piece is C major" - because a piece in C mixolydian is not "in C major".

When it comes to what key signature you should use to write a piece in C mixolydian, you will find differing opinions as to what's clearer. Some would suggest using a key signature with one flat, and marking the score "C mixolydian". Others would suggest simply marking each B flat with accidentals.

If keys of songs defined in terms of the major and minor scales, how do we define our key when we play scales other than the major and minor scales?

Tonalities of songs are not only defined in terms of the major and minor scales; I would simply suggest avoiding using the word 'key' where it may lead to confusion. The major/minor key system can often only be an approximation, and restricting yourself to speaking only in terms of the major/minor key system may cause you to lose information about the music you are discussing.

A key signature defines two things: what modifications should be made to notes by default, and a direction as to what key the music is in. Basically, the choice for what key signature you use is simply a balancing act between those two factors.

Let me give some examples:

• 
  C Minor, with 3 flats (Bb, Eb, Ab). The choice is between writing it with no flats/sharps in the key signature, and writing the
  accidental for every single 3rd/6th/7th. On one hand, the signature
  makes it clear its in C... but on the other hand, you're having to
  accidental a whole lot of notes. It's clear that the second factor
  is a lot more of a hindrance than the first. So, instead, C minor is
  commonly written with 3 flats in the key - and it's up to the person
  reading the music to understand/grasp that the three flats don't
  signify that the music is in Eb major. It's a downside (which
  musicians have to mentally account for)... but the downside is more
  than outweighed by not having to have every other note accidental'ed.


• 
  C Mixolydian, with 1 flat (Bb). Now the choice is: do I add a Bb to the key signature? Adding the Bb makes it appear that the song's
  key is actually F. But the upside is that I don't have to accidental
  every B in the score. In that case... the first factor almost
  certainly outweighs the second: nobody would bat an eye at seeing all
  the B's flatted, but it might cause confusion when the base of the
  music wasn't an F. (As a former jazz musician, I can definitely say
  that I would much rather read a flatted seventh accidental than
  getting confused as to the chord structure or key for a song.)

As a general rule of thumb, you should probably score the music with a key of either a major or minor key - not because you can't do something else, but because the confusion of a base/key will probably outweigh having to parse accidentals more than usual.

When we play the C mixolydian mode, are we still in the key of C?

Maybe. Maybe not.

The problem with the question is thinking a collection of tones C D E F G A Bb C is the only thing needed to answer the question.

In a very simplistic way you can say this collection of tones fits the key signatures F major and D minor.

Of course it would also fit 1 of 7 possible modes - C mixolydian, D aeolian, E locrian... etc.

Tonality is fluid so you could also say those tones match C major with a temporary tonicization of the subdominant. In fact analysis discussion can reflect that. People may refer to things like "Phrygian flavor in minor" regarding a N6 chord or "subdominant inflection" when a lowered seventh in used in a major key without necessarily identifying a key change.

...how do we define our key...

You need to identify the tonic before you can answer the question.

Also, listing the notes in alphabetic order starting on C doesn't answer the question. That scale fits nicely over chords like Dm7, C, C7, F, etc.

Yes.

If you are using borrowed chords or modal interchange, and C is the tonal center, then yes, the key is still C. If you are strictly using C Mixolydian, but it still resolves to C, then it is a modal song or modal music, like a Gregorian Chant and the key is C Mixolydian. Even though the notes are the same as F major, it is not F major if the song resolves to C.