mjhjmtu

In machine learning, people usually refer to the "performance of a model" or "performance of an optimizer". What is the exact definition of "performance"? What would be the "performance of an optimizer"?

I know that there are ways of measuring how far away current predictions of machine learning models are from the expected ones: for example, you can measure the accuracy of a model, perplexity, etc. Is this what the performance of a ML model refers to? Is it a name to refer to any way of measuring the correctness of the predictions (or, in general, outputs) of the model? Or is performance actually the time it takes to perform the prediction? Or something else?

In the absence of any specific clarifying context, "performance" is just a synonym for "quality."

The sentence "I want a model that performs better" is essentially the same as the sentence "I want a higher-quality model." Readers understand that the speaker is not satisfied with how well the model solves some particular problem, but the reader does not know, precisely, what about the model is dissatisfactory. Does the model predict too many false positives? Or false negatives? Does it predict incorrect classes for images that have a tilted horizon, or are taken on cloudy days? Understanding what about the model needs improvement would require further, specific elaboration.

Likewise, if someone says that Adam has better performance than another optimizer, they're making a claim that Adam does better at some task, which they would have to specify for it to be possible to assess the truthfulness of the claim. One way to assesses performance of an optimizer is how many iterations it takes to reach some neighborhood around a minimum; another, which is particular to machine learning classifiers, is how well the solutions obtained by an optimizer generalize to out-of-sample data.

Performance does not englobe a formal definition, it is mostly dependent on, but not limited to:

• the context of the model you are trying to implement


• the metrics you are using to evaluate the model's output


• the objective you are pursuing

The use of one metric or another will depend whether you are trying to predict a continuous or a discrete variable. Some of these are: Accuracy, Recall, Precision, R2, F-Measure, Mean Square Error, etc.

To make this clear, say for instance you are working on a credit card fraud machine learning algorithm, where you want to predict the number of fraud transactions. To evaluate how well the algorithm works: a) understand the context b) understand the metrics that are applicable to the problem

a) We are dealing with a classification problem; a transaction can be fraud or not (target is a discrete variable). We will be most-likely facing a highly imbalanced dataset since most of the credit card transactions are non-fraudulent.

b) Since it is a classification problem, we can discard all the metrics associated to continuous variables (for example, R2 and MSE). Moreover, due to the possible imbalance, we can also discard Accuracy metric. This leaves us with two metrics: Recall and Precision. We know by theory that these two metrics present a trade-off (if I increase one, the other decreases). Recall would help detect the major possible transactions that were fraud. Precision would help with avoiding misclassifying frauds.

Concluding, how well our it works, after considering the previous things, will ultimately depend on what is our objective:

• Is our goal to detect the highest amount of fraudulent transactions
  as possible? If it is, our metric to evaluate the model's
  performance, and further improvements, will be Recall.


• Is our goal to avoid classifying a transaction as non-fraud when it
  was a fraud? If it is, our metric to evaluate the model's
  performance, and further improvements, will be Precision.

Hopefully this provides you a better insight.

As the other answer correctly points out, there is no universal definition or measurement of performance of a machine learning model. Rather, performance metrics are highly dependent on the domain and ultimate purpose of the model being built. Performance of an ML model is just "how good" it does at a particular task, but the definition of "good" can take many forms. A "good" model could be one that predicts well, one that trains quickly, one that finds a robust solution, or any combination of the above.

For example, an algorithm used for a medical screening test should be highly sensitive - we want to catch all possible cases of a disease, at the cost of misdiagnosing some people who aren't actually sick. These individuals can go on for further tests that may optimize other metrics like positive predictive value, indicating that a positive test result is likely a result of actually having the disease. Depending on the purpose of the test, we may want to put more weight on true positives/negatives at the cost of errors on the other side.

Performance can also be a function of the error measure used. Suppose your classifier outputs values on a continuous scale which are then thresholded for a binary classification. Do you care only if points are on the correct side of the boundary (accuracy measure)? Or do you care how badly you missed on the misclassified points (RMSE)? There is no universal best way to optimize performance.