Thursday, April 25, 2013

A Good Approximation for L0 Norm Regularization ?

In machine learning, Regularization are usually used to prevent over fitting. Different regularization are used subject to different purpose. Usually, L2 norm and L1 norm are used.
L2 norm is fully derivable, but converge slowly
L1 norm is only broken on 0, and faster than L2.
L0 norm are thought to be NP-complete.
the formular for L0 is 
$y=\sum _{n=0}^N1\left(W_n\right)$
where 1() is a indicate function, whether Wn is non-zero
Not only it is broken on 0, the derivative of other point is 0, which is not able to be used as regularization.
Think about the function below:
$y=-e^{-b\left|x\right|}+1$
where b is a parameter, the bigger b is the thinner the pitch is.
Approximation of 1()
This would be a good approximation of 1(), as it's only broken on 0 and the derivative of other points are
$y=be^{-b\left|x\right|}\sign \left(x\right)$
It seems like a Reverse of an Impulse.



No comments:

Post a Comment