mjhjmtu

I have two sets of data, namely blue and yellow. I manually added a point 8, -3 to the blue data.

sampledata[center_] := BlockRandom[SeedRandom[123]; RandomVariate[MultinormalDistribution[center, IdentityMatrix[2]], 200]];
clusters1 = sampledata /@ 9, 0, -9, 0;
clusters1[[2]] = Append[clusters1[[2]], 8, -3];

plot1 = ListPlot[clusters1, PlotStyle -> Darker@Yellow, Blue];
plot2 = Plot[0.2 - 0.375*x, x, -12, 12, PlotStyle -> Red];
Show[plot1, plot2]

As you can see, the two sets are linearly separable. Thus the SVM algorithm should be able to separate all points by using just a linear kernel. Now I try below:-

c3 = Classify[<|Yellow -> clusters1[[1]], Blue -> clusters1[[2]]|>, Method -> "SupportVectorMachine", "KernelType" -> "Linear"]
Show[Plot3D[c3[x, y, "Probability" -> Yellow], c3[x, y, "Probability" -> Blue], x, -15, 15, y, -4, 4, Exclusions -> None], ListPointPlot3D[Map[Append[#, 1] &, clusters1, 2], PlotStyle -> Yellow, Blue]]

As you can see, SVM failed to separate the points. The blue point 8, -3 is now located in the yellow region. Why would SVM be failed to separate the linearly separable points?

Many thanks!

It's not explicitly documented, but I think Mathematica is using the C-SVM variant, where a regularization parameter C basically says how "expensive" mislabeled training samples are, compared to the size of the margin. So in your case, SVM will perfer a larger margin between the yellow and the blue points. In 99% of the cases, this kind of robust behavor is exactly what you want.

If you don't want that, you can play with the (undocumented) SoftMarginParameter option. Think of this as the cost of mislabeling a training sample, compared to getting a larger separation margin:

c3 = Classify[<|Yellow -> clusters1[[1]], Blue -> clusters1[[2]]|>, 
 Method -> "SupportVectorMachine", "KernelType" -> "Linear", 
 "SoftMarginParameter" -> 1000000]

Show[ContourPlot[
 c3[x, y, "Probability" -> Yellow], x, -15, 15, y, -4, 4, 
 AspectRatio -> Automatic], plot1]

Now all samples are classified "correctly", but if you tried the classifier on new data, it will likely perform worse, as some of the dots are much closer to the margin