The point is, it is just a matter of perspective. While the one who envies, well, envies; the one who is envied probably views his position a matter-of-factly, and would not think he is being envied.
My grass will always look greener to you; your grass will always look greener to me.
We are all green with envy. But we fail to remember, whether your grass or mine, grass is just, well, green.