Your players are new to the game (and new to your game). Unless you've told them what to expect, how could they know what your game is like? Getting stuck investigating small details or trying to get their way in through the door could plausibly be what your game is about. I think you should be more active in leading the game towards the direction you'd like to see it play out in.

When they got to the castle, they spend about half an hour (real time, not in-game time) banging the door waiting for it to be opened.

The solution to this problem is simple in principle: don't let them do this. You are not obliged to let the players try banging on the door indefinitely long in real time. Consider the following example:

GM: You stand before the heavy, wooden castle door. It's eerily silent.

Player 1: I knock on the door.

GM: You rap on the door, but hear no response, no footsteps, nothing from the inside.

Player 1: I knock on the door again, then.

GM: Still no response.

Player 2: I bash the door too.

GM: Still no response.

et cetera...

This can go on for arbitrarily long unless you or the players stop it. Do something like this instead:

Player 1: I knock on the door.

GM: you rap on the door, but hear no response, no footsteps, nothing from the inside.

Player 1: I knock on the door again, then.

GM: You spend a while knocking on the door, but getting no response, it becomes apparent that if there's anyone in there, they're not going to open the door for you.

If you want to be really explicit about it --- which I tend to recommend especially to new players and GMs, add something like:

GM: You're going to have to find some other way inside.

The important trick is that you don't let your players spend time on anything boring, and that you pass the long, boring time their characters spend on knocking the door with a sentence or two. Repeated attempts at mundane tasks is something you don't have to focus on, and indeed shouldn't --- time spent in-universe doesn't have to be mirrored in real time.

Similarly, your example about the hole in ceiling shouldn't take an hour under any circumstances: the first time a player attempts to grapple themselves there, you can come up with a reason why they can't go there:

GM: The hole is just large enough to let a bit of light in, but there's no way you can fit through it.

Or, to take an alternate path:

GM: You climb up the hole and notice, to your disappointment, that it leads only through a crack in a rock outside the castle, surrounded by thickets that obscured it from you before. This clearly isn't the right way in.

Whatever you do, don't make your players try something and roll for minutes. If the task is something where failure wouldn't be interesting and they could just try again, let them succeed without a roll. If it's something that can't work out, tell them it's impossible after the first try.

A few words on railroading and linearity

You say your players seem to think of the game as a simple, linear story, but to be honest, that sounds like a pretty good assessment. You have a castle where you've planned a lot of content that's accessible only through the caverns where you've also planned a lot of content, and you're upset because the players aren't engaging with your content. You are running a railroaded campaign.

A common misconception is that railroading is bad, period. It's not. Railroading is fine, as its own genre, but you need to be open to your players about what kind of a game this is and need to have clear cues for your players to follow in order for it to work. Without clear directions, your players are still as railroaded as before, but will spend time stumbling, feeling their way in through the invisible railroad instead of having fun advancing along it.

A few words about storytelling

Even after repeatedly failing to do what they think they should do, they just keep going, until I tell them to try something else, reminding them that not everything in the world is there just to be in their story.

What you're saying is contrary to the usual principles of RPGs and storytelling. You're making a story with the players: everything in the world exists for that story and the players' (including you, the GM) enjoyment, and not for anything else. Isn't that why the whole world you're playing in exists?

Different genres of fiction handle different narrative tropes differently, but one fairly major principle that works its way across genres is that of Chekhov's Gun: if there's a gun hanging on the wall in a play, it is there to be fired. More generally, if a work of fiction describes something, the reader/viewer/listener/player will assume it is going to be important and is usually absolutely right.

This is not necessarily true, nor necessarily realistic (for groups who prefer that sort of thing). However, it is so prevalent in normal storytelling that it's essential that you make sure your players and you are on the same page regarding how it is to be applied, because your players are going to assume it holds, consciously or not.

Think about it: you went through the trouble describing an appealing hole in the ceiling and told them they were under the castle they were supposed to infiltrate --- it seems obvious to anyone who hasn't heard your side of the story that the hole is of vital importance!

The Same Page Tool & Talking with your players

Finally, check out the handy Same Page Tool which gives you some things to discuss with your players regarding the type of game you want to play. Note that it's not a survey, nor a list of things to vote on, it's a discussion aid. It might help you clarify what kind of a game you want to play, overall.

You should also never hesitate to talk with your players, off-game, if you're not having fun with the way the game is playing out, and encourage them to be open with you in a similar manner. Table time is too valuable to spend on rolling for futile grappling hook attempts or other fumbling.

Players

The players are new and possibly overwhelmed by having too many options. Thankfully, this is easily solved by talking to them and reiterating that the game you play is more open world.

Make sure to ask the players what they would feel would be a good prompt for them to make new decisions.

You can encourage them to do just that by describing thing in more detail. The castle seemed to be in one piece and standing just fine. So, instead the door should have had arrows in it, scorched marks, and there should be crows everywhere. The walls look impenetrable but there's a massive crack in one as if a sally was done -- clue for caves, assault, and of course another mean to get in.

GM

You are doing a little of rail roading: You planed the castle exploration, getting into the caves, and interesting encounters. Then the players did not follow your plan. That is your fault for having a plan.

A fix is instead of saying "no", say "yes, and …" or "yes, but … ", or "yes. How? …". This leads the players to feel (rightly) that they have agency in the world. It allows you, as the GM, to add complications to ensure that the story moves in the right direction.

So, in your example: they spend time trying to climb up. Great, let them. Now, they are in the castle, ready to explore it. However, while climbing they made such a racket that the orcs heard them. Cue: Drums! Drums in the Deep!.

As for clues, remember that you know what things are clues and what are not. To the players, all your say have equal value. Therefore, if you say something like "fluff, fluff, clue, fluff" all your players hear "thing, thing, thing, thing". Which one is important?

If they players do something weird, it is quite alright to question it. For example, the players knock on the door. You could say "You knock. After a few heart beats, you hear bunch of birds take off from within the castle. As you look up, crows stream out towards the sky. Something wet drops on X's face. It is a drop of blood, probably carried by a crow overhead. There is no response from the castle. What do you do?"

When dealing with new players there are three key things that I'd use to help them understand that the game is open to options and ideas rather than following a single course and encourage them to think for themselves and get ideas rather than just follow a course.

Talk to them out of game

Getting them to understand what and how a game works out of character, either during the game or before it to help them understand how things work - for new players who are used to games with defined rule sets (like Settlers of Catan) or stories where there are only a single path this can be a bit of a confusing concept; you can help this by prompting them out of character when they first played and laying the options out for them clearly in out of character comments, they also may not be listening or remembering other options.

You're already telling them this after they've been failing for a bit, but try to lay the options out for them so they realise there are paths earlier.

GM: "You bang on the door and it's very loud, you're pretty sure that no one could have missed that noise who was on the other side. Now players, just to remind you; there's a low balcony that you could possibly climb up to if you were athletic, the door itself could be battered down with an axe if you had a mind and a lot of time, and there's also that grate down at the bottom of the moat that could be crawled past, there are other possibilities, if you can think of something you can try it."

Spoon-feeding ideas and options can help them, and then you can move into playing it through descriptions...

Use visual cues

The options above can then be woven into descriptions rather than telling the players openly what their options are as they get more experienced, describe other doors, loose bricks, low windows and so on - lowering the amount of OOC info you give so that they'll hopefully start picking up ideas from your description, remind them to listen to the descriptions and repeat if necessary - it can be a bit tedious at first, but they're learning so you need to be patient :)

Sidekick helpers

I also find it helpful if players have a familiar, sidekick, or any sort of animal (my MERP group had a Rohan warhorse that was the smartest person in the group) that can help feed these ideas or quest for ideas to them without breaking into OOC.

GM: "You've hammered at the door to the keep for ten minutes now without any help. Siggy the Warhorse has wandered off and is chomping down at a large patch of tasty green grass right next to a large crack in the wall about thirty feet away, looks like it might be wide enough to squeeze through."

New Rule: One Scene, One Try

There are several exhaustive answers here, but I wish to focus on a particular issue I often encounter:

they spent almost all of the remainder of the session trying to get a grappling-hook style thing up through the little hole and then climb out.

Did it go something like this?

DM: You see a small opening in the top of the cave with light shining through.

Player 1: my character takes out his grappling hook and 50ft of rope and throws it up at the opening.

DM: Ok. Make a roll.

Player 1 rolls dice.

DM: Your grappling hook soars through the air but misses the opening.

Player 2: Wait wait wait. My character picks up the grappling hook and tries.

Player 2 rolls dice.

DM: The grappling hook again soars through the air, misses the opening and strikes the cave wall with a loud clank.

Player 1: Let's try again, but this time I have my character assist the other character.

Player 2 rolls dice and with assistance bonus.

DM: The grappling hook misses. Again.

Player 3: What if we try it from on top of this rock over here?

Player 4: Spider climb could help...
etc..

In a dice based game this play style is guaranteed to produce:

1. A high enough dice roll to tackle any problem (i.e. a natural 20)


2. Waste an inordinate amount of time on small tasks.

If you have players that fall into this play style, you need to be more firm with not allowing continuous retries. I prefer the one scene one try rule: the players, collectively, get one try per scene* and must abide by the results. The above example would play out like:

DM: You see a small opening in the top of the cave with light shining through.

Player 1: my character takes out his grappling hook and 50ft of rope and throws it up at the opening.

DM: Hold on. The 'one try' rule applies to this scene. Take five minutes and discuss the best way you can work together on this roll.

Players discuss strategies for five minutes.

DM: Ok time is up. What is the plan?

Player 1: Player 3 will have their character cast spider climb on my character. They then walk as far up the side of the cave as possible while carrying the grappling hook. If I am unable to reach the opening, my character will attempt to throw the hook at the opening. Player 2's character will give me an assistance bonus by helping me aim the throw and keep the other end of the rope from tangling.

DM: Great! If everyone agrees with this plan, cast spider climb and make a roll.

Player 1 rolls dice.

DM: Here's what happens..

*A scene is defined as a part of the game that is larger than an 'encounter' but smaller than a 'session'. A night spent drinking at a tavern, a visit to an abandoned castle, a walk through a creepy forest are all examples of what I call scenes. A single session may have several scenes. From the description of your session, I would say the visiting of the castle is a scene that includes the cave. The characters could not just leave the cave and go back in and expect a retry (it's the same "scene"). However, if they left and traveled into the forest only to be chased by owl bears back into the cave, that would be a new scene and merit a retry. As with most things in tabletop RP, the DM will need to use their judgement when a scene ends for the purposes of re-rolls.