When I first came to NYC, I had a tough time knowing when a cab was available. I see tourists dealing with this all the time. There's no point in standing there with your arm raised. It's much better to wait until you see one that's available and then raise your arm.
This cab is available:
This one is occupied (notice its sign is not at all lit up- sorry it's blurry- these cars move fast):
And this guy is off duty:
Some other random info but hopefully useful if you're visiting:
Taxi cabs cannot refuse to take you somewhere if they're available. It happens from time to time that they do, and you see that number at the top of the cab, in the light box? That's written along the side of the cab too, so you can report them should you want.
The only cabs allowed to pick up people randomly are yellow cabs. There are car services that people can call to take them places, but they are not allowed to randomly offer rides. They charge more for one, and also, unlike yellow cabs they don't take credit cards. These guys are often referred to as gypsy cabs, and they have no problem offering you a ride randomly. They also trick people freshly landed into using their "service" and despite all the signs warning against using anything other than a yellow cab, people fall for it and end up getting taken for a ride- literally.
There is also apparently one yellow cab that has a TV show called Cash Cab, and I've been told that it's a trivia type show. If I got into that cab, I would get right back out and catch the next one. I don't worry too much about that though, the statistical odds of ending up in that cab are very low, given the number of yellow cabs in Manhattan (something like 13,000).