Point in a polygon algorithm
When you say "points of polygon", I am assuming you are referring to vertices.

I don't think what you have is the fastest. A simple improvement to this could be to divide your matrix in to a grid of p x p cells, where p is a parameter, and classify each grid-cell as completely inside or completely outside of the polygon. This can be done by checking if all the vertices of the grid-cell are inside the polygon. This way, large regions inside grid cells can be directly labeled as inside or outside, without needing to test every point inside the cell against all the edges of the polygon. For the grid-cells that are not completely inside the polygon, you just go by what you are doing. The optimal choice of p depends on the size of polygons you may encounter.

More details about the grid method here (with comparisons to other strategies):
http://erich.realtimerendering.com/ptinpoly/

There could be better ones. I just suggested a simple improvement, which works well in practice.