I am a student of a University of Computer Science, and in my project have put me a tasks and is one me is returning crazy

Initially they gave one, that given a cloud me of points, is needed that my program gives back the convex polygon, and I already did it, with a mathematical solution, but they have put me other than it has to me throwing of the hair

For these points:

Must be something similar to this:

For example for an ear:

I must give this result (so so): (of course with more points the image its better)

I have investigated and I have not given with the solution, I do not see any mathematical algorithm that it solves this, if somebody can help me would thank for much.

Best Regard