I'm writing about a setting where building technology is quite advanced, but elevators have not yet been invented, so the limiting factor on height is ergonomic.

What would be the maximum practical height for a building where the people going up the stairs are expected to be pretty fit, and where it's deemed acceptable to need to stop for a rest on the way up?

The Problem

A large part of the problem is how big do you want to build without an elevator of some sort.

The earliest elevators were made in the third century BC. The basic technology is simple with a platform below a hoist. If you are going to be building something more than a few stories you will have a hard carrying everything up and down stairs or ladders and eventually will just throw a rope down and pull stuff up.

Once you are pulling stuff up on a rope its a short step to a loop to stick your foot in and then a board to stand on.

Gravy on the whole scheme is a pulley at the top so the lifting can be done from the bottom.

The elevators may not look like modern elevators but once you start building high, it is almost inevitable. You need to get things up to build, and moving 100kg of person up and down in order to move 50kg of concrete up just sucks.

Pretty much once you have a way of lifting stuff up, people will try to ride it.

The Solution

The only reasons elevators wouldn't be ubiquitous in tall buildings would be political or social.

Fitness Fetish

This society has a strong focus on physical fitness where there is a strong shame associated with not moving under your own power.

Historic Accidents

Elevators were invented but a tragic accident in the past instilled a cultural bias against them

Taxes

If elevators are highly taxed, then there won't be many of them except for the very rich where they would almost be a status symbol.

Practical Concerns

According to http://urbansome.com/how-many-calories-does-walking-up-the-stairs-burn/ humans burn 10 calories climbing up one floor. This means that climbing up a 20 story building means you will burn a tenth of your daily calories.

New York City used to be limited to 8 story buildings due to hydraulic concerns, they couldn't get the water up to the higher floors through pressure alone. Most of these didn't have elevators.

Of course there is also this modern building in Spain.
http://www.spiegel.de/international/europe/architect-forgets-elevator-for-47-floor-skyscraper-in-spain-a-916082.html

A final loophole

If tall building without elevators are normal, then there will be a strong desire to tie buildings together at higher floors with elevated walkways. Commuting from the 13th floor to the 12th of a building a mile away isn't so bad if every building in between is connected at the 10th floor. There is less energy expended walking the extra mile than up the 10 extra floors.

Any practical (i.e. within the troposphere) height, if there is a primary reason for people to do something on highest floors. It would be part of the duty (or pilgrimage) to ascend the tower over the course of multiple days, toiling up the supplies.

15-20 stories high, if it's a general purpose building and it should be accessed by wide population - including frail, but high-ranking members of the society. Any more than that, and carrying up a high-level bureaucrat on the stairs would became an impractical task.

I've walked up several tall structures - the tallest may have been Ulm Minster; at 161 m approximately the same height as 50 storey building. I'm middle-aged and quite unfit - I found it pretty tiring, needing to stop for a rest a couple of times.