mjhjmtu

Suppose your team writes a software system that is running fine.

One day one of the engineers mistakenly runs some SQL queries that change some of the DB data, then forgets about it.

After some time you discover the corrupted/erroneous data and everyone scratches their heads as to which part of the code caused this and why, to no avail. Meanwhile the project manager insists that we find the part of the code that caused it.

How do you deal with this?

It is obvious no project manager will invest an infinite amount of time into such a problem. What they want is to prevent happening the same situation again.

To achieve this goal, even if one cannot find the root cause of such a failure, it is often possible to take some measures for

• detecting such kind of failure earlier in case they reoccure


• making it less likely the same failure will happen again


• making the system more robust against the specific kind of inconsistency

For example, more detailed logging, more finegrained error handling, immediate error signaling could help to prevent the same error to strike again, or to find the root cause. If your system allows to add database triggers, maybe it is possible to add a trigger which forbids the inconsistency to be introduced in the first place.

Think of what the appropriate kind of action might be applicable in your situation, and suggest this to the team, I am sure your project manager will be pleased.

A production database should have full access logging and role based access controls. Thus you should have hard evidence as to WHO did WHAT WHEN to the database thus moving the attention from the code to poor operational security.

1. Explain to your project manager that you think the most likely cause is manual database access.


2. If they still want you to look for the code that caused this, go and have another look at the code.


3. Come back in a couple of hours (or some other appropriate time) and say you can't find any code which would have caused this, therefore you still believe the most likely cause is manual database access.


4. If they still want you to look for the code, ask how much time they would like you to spend on this. Subtly remind them that you won't be working on feature X, bug Y or enhancement Z while you're doing this.


5. Spend as much time as they ask. If you still think the most likely cause is manual database access, tell them this.


6. If they still want you to look for the code, escalate the issue as this has clearly become an unproductive use of your team's time.

You may also want to consider if you should add in an extra processes to reduce the likelihood of manual database access causing this kind of issue in future.