You have a similar problem with your government regulating magic as any government has with regulating weapons. Even restricting education as suggested by @L. Dutch does not help on the long run, as studied mages can train their own students outside classical circles and not shackle them with an oath or provide an unmagical minion with bound artefacts or support from a familiar.

You will either require an oath without any loopholes which is incredibly hard to design. Torturing people or experimenting on them without magic, then healing them magically would completely fulfill the "only good magic" policy, yet it is truly destructive.

Prohibiting exploitation and missuse is almost impossible. Thus, you will require a magic police force, mainly relying on scrying magic to detect any missconduct. Then, you will likely need experts in all fields to investigate or counteract magic missdeeds.

All ritual circle sessions must be registered

Since every spell requires a ritual circle and takes "30 minutes to several hours depending on the spell," magic is not something one can do spontaneously or unknowingly.

Think of it like registering a flight plan. Whether you pilot a commercial airliner or you own a tiny plane, you need to create and submit a flight plan before every flight. It's not hard to do. The fact that one flight might be solo to the next county and another might have 3 passengers to 3 states over is fine. You just create a new one each time.

Same with magic. Just a quick report with the names of the participants, the primary spellworker, and the type and purpose of the spell.

• It's illegal to perform magic without registering the circle first.




• It's illegal to lie about the participants (last minute changes are okay as long as you amend the report within 24 hours).




• It's illegal to do certain spells and therefore illegal to lie about the kind of spell you're doing.

This doesn't immediately solve the problem of tracking down illegal magic, but it makes it worlds easier. Any unreported ritual circle is immediately suspect and investigated. Some will just have mild fines for forgetting to file, but others will turn out to have been illegal. Any report with a discrepancy gets investigated. And if there is a crime report later, it's easier to track down who might have been involved.

Other ways to figure out what's happening could include (depending on how you've set up the world):

• Ritual circles put out a detectable energy for X distance that an instrument can pick up. If X is big enough, then you can set up detection stations and triangulate location, then cross match with the filed magic plans.




• Ritual circles put out a detectable energy for X distance that a trained person can pick up. This would be similar to having police patrols looking for criminal activity, or could be used in surveillance when suspecting someone of illegal spells.




• Ritual circles leave a detectable energy or mark on the ground/floor of the location where they occurred. It may or may not be possible to determine number of participants and/or length of time since the spell was cast. It may or may not be possible to determine the type of spell.




• When someone has participated in a ritual circle, there is residue on their hands (or elsewhere), just like there is gunpowder residue on someone after firing a gun.

Unless you go all Harry Potter (the government immediately knows when an underage wizard performs any amount of magic outside of Hogwarts, but we never know how), your state will have to have detectives and police officers and courts to determine the illegal use of magic. You build a case using the evidence and testimony of victims or witnesses.

Create a professional order of the authorized mages, and allow the profession only to those who are member of the order. Membership of the order comes with the oath to perform only legal magic.

In this way the government, whenever some magic is performed, can first check if the performer was or not allowed to perform magic at all, and then if the performed magic was legal or not.

The order can also develop means to detect when magic is performed, to act in a more proactive way with law enforcement.

The state could have all magic users marked with a sigil when they enter the education system, and that sigil would flare when forbidden magics were being cast, and alert the authorities or, perhaps, a singular organisation that handles regulations for magics.

Alternatively, you could have the state propagate the use of familiars (created through Transmogrifcation spells) and use them as a form of silent observers for any signs of forbidden magic. Of course the public wouldn't be aware the familiars are spies or radars, and the state could use propaganda to normalise the use of familiars from an early point in its history.

3 Keywords here:

Scrying, Protection And Transmutation

You basically need 2 things:

1) A law than forbidds certain types of magic/spells

2) A group designed to find and enforce that legal restrictions.

For all of that you could have something like Magical Minority Report police force, they can:

1) Have a main tower/base or several, where they continuosly scan the world/nation/provice/state/wherever in search of infractors using Scrying magic.

2) Have in this main tower/base (or the several) some Protection spells maintained over the territory than absolutly prevents or at the very least resist (and hence, delay) the execution of any one of this forbidden magics.

3) Have a group of Transmutation mages than basically alter space-time to allow the fast-travel of strike teams through portals to the places where this rituals are being executed.

How can the state regulate magic and track the use of illegal spells when they are conducted by a mage?

The same way they did in the Harry Potter books and movies when He Who Should Be Named overtook the Ministry of Magic. Use this:

Scrying Spells – These are spells designed to allow a user to perceive in ways that go beyond his fve senses.

College of Mages

Not everybody can be mages. In order to be one of them you must to complete several exams and show them that you are a trustworthy person. By this way, in order to be a magician you must to not cast illegal magic, otherwise, you lose your profession forever, maybe also your life.

Academical Lock - Transmogrifcation

When a new magician enters into the magical academy (or if magic is innate when they earn the magic permission) the magical police insert a transmogrifcation spell in their bodies. These spells forbid casters to use forbidden spells, either by will manipulation, intense pain, sickness or instant death.

Oracle of Magic - Scrying

Magic is like sunk water. Sunk water is motionless, static, powerless. When a magician(s) cast a spell they touch this water causing fluctuation or waves in the cosmos. Oracles of the magical police are able to detect this fluctuation using powerful and advanced scrying spells. Analysing the vibrations of that waves, they are able to determine the type of spell and if it's forbidden or not.

Additions

You modify these ideas. For example, if you don't have academies nor need permission for the government to cast magic you may build an immense tower in the capital of the kingdom which is always casting a powerful AOE (Area of Effect) spell in the entire kingdom forbidden caster to do dangerous magic. Or if you don't like having a powerful oracle maybe each mage, when entering into the academy or gain their permission, the police insert in their bodies a scrying spell... which in addition it could also track other things...

Personally, I like the idea of having an immense tower which cast an enormous spell to detect every casted spell in the whole kingdom, and if one of them is forbidden, they automatically respond to casting magic or teleporting the magical police. Take into account that this tower has to be really cold or full o water if we follow your another question.

I have an interesting idea that I believe no one has mentioned (although we all seem to have the same idea of Scrying) which is to cast your tracking spell on the spells that are illegal. Given the following two definitions:

• Scrying Spells – These are spells designed to allow a user to perceive in ways that go beyond his fve senses.


• Protection Spells – These are spells designed to ward a user, object, or location against a variety of possible harms

It seems reasonable to me that you could ward the actual spell, or if a spell isn't an object in the eyes of the Orlone maybe you could ward the combination of certain sigils which would produce the spell (this could also interestingly add false positives where someone casts a legal or newly created spell which contains the tracked combination of sigils but produces a different effect). By warding the spell and that using Scrying to detect when that ward is broken or triggered I imagine your government could have a good idea of when illegal spells are used, without having to use enormous amounts of energy to monitor all uses or the Orlone or restricting themselves to a specific area.

The state casts a detection spell (would be categorized as a Scrying spell) over a big area (like the whole country) that detects illegal sigil sequences. Then they react based on the informations from the spell (location, the type of the spell cast, etc). Could be powered by something to keep it up for an indefinite time.

I suppose Orgone has a fabric over space-time. When someone casts a spell, there will be a temporary deficit of it, and some seconds are needed so that it can flow back from neighbouring areas. So any use of magic disrupts the fabric of Orgone.

This disruption, while relatively subtle, can be detected by some power magic users, including mages employed by some sort of government agency. It can also be deduced by skill - although not necessarily with 100% probability - what kind of spell it was.

1. 
   

   Create a climate of fear surrounding magic use and magic users.

   
   
   

   a. Make a big public show of sending troops to "less advanced" countries to help quell "magic insurgencies".

   
   
   

   b. Encourage your media outlets to disproportionately report on crimes committed by magic users, or where magic use is suspected be sure to down play any other possibilities.

   
   
   

   c. When a magic user is accused of a crime, the media should always treat them as an adult with a history of bad behaviour, regardless the facts. When nomajs commit crimes have them treated as innocents, caught up by bad circumstances.

   
   


2. 
   

   Make magic users a lower class than nomajs

   
   
   

   a. Magic users should be seen as savages. They attack their own. They seem to have a lack of self control. Look how many magic users there are in prison, in proportion to nomajs.

   
   
   

   b. Magic users are a drain on resources. They are so destructive, even accidentally. Immigrant magic users use up the welfare system. They come here and take all the jobs. You have to tighten the borders to stop the bad ones getting in.

   
   
   

   c. Most magic users who have jobs can only seem to get low paying jobs. They don't have the wherewithal to be socially mobile. It's their own fault for the state they are in.

   
   
   

   d. Magic users are zoned into particular living areas. For their own protection, or course.

   
   



3. Reward people for reporting anything magic related that seems suspicious.

In my opinion,tracking illegal magic is almost mission impossible by your description.And a most efficiency way I can have in mind to prevent illegal spell from being performed,is make sure a high-level wizard can get a good job with high salary.

I find that in the real world, it's actually pretty hard to track the apply of illegal technology.Take a example,If I synthesize some explosive by lawful material and make it into a fully armed bomb,No body would know till it explode.physically, making a bomb or synthesizing some drug it's not much different from cooking a dinner or change a light bulb.difference happen after those actions.

Tracking illegal magic would be even harder in you faction because unlike technology which require material and equipment to apply (though those material and equipment may be obtained easily and legally),perform a magic only require Orgone and knowledge,and it doesn't sounds like having a way to know what those Orgone were used for from a distance.

Though it's hard to track illegal tech,our world still hold up well,because most people that could apply those tech, can have a good job using those skills.they don't have much motivation to do such dangerous thing.for same reason,I think if the government in a
magical world want to regulate magic,it at least should low unemployment rate among wizard down.throw more money in basic magic research sounds like a good idea!