On current versions of Android (I'm not sure when this was added, but this is available in Android KitKat), you can go to Location Settings and it will show a list of apps under Recent Location Requests.
You can then kill those off one by one until the GPS notification goes away. Sadly, you can't kill them directly from Location Settings. You'll have to either go to Settings -> Apps, or find them in your recent tasks.
It would be nice if there were a way to actually see what is using the GPS, as sometimes it seems to be some sort of Google service that doesn't show up as an app. (So, even killing everything listed doesn't help, then GPS turns off randomly a while later.)