When you say "No" to someone, you're also saying "No" to everyone they've ever worked with. A friend of mine who is into playing bass, told me a story about his growth as a musician. At one point, when he had a lot going on in his life, he felt like he couldn't take on a few gigs that he wanted to.
"Someone I hadn't worked with before asked me to play bass on a gig that I was interested in, and I almost said No," he told me, "but then I realized that if I say no, I'm not only saying no to him, but I'm saying no to everybody he would have recommended me to."
As it turns out, the first gig was a blast, and more importantly, he ended up being able to say "Yes" to some really awesome gigs as a result of saying "Yes" to the first gig. Keep in mind that, for the other gigs, he wouldn't even have been asked to play if he had said "No" to the first one.
Every "No" you see becomes a thousand more that you don't.