The M-ary support vector machine (SVM) is introduced as a nonparameter nonlinear phase noise (NLPN) mitigation approach for the coherent optical systems. The NLPN tolerance of the system can be improved by using the M-ary SVM to conduct nonlinear det