Does this phrase bother you when uttered by people who you know have money?
What about "I just can't afford that right now." Or "It's not in the budget."
It's just another way of saying "I am not going to spend money on that right now." And we all have our reasons. Maybe we can't go out to lunch because we really are broke. Maybe we have stretched ourselves a little thin and can't buy lunch because we have to pay rent. Maybe we are careful with our budgets and don't have the money in the "lunch" category.
For the most part, it doesn't bother me. But I have a friend who is constantly complaining that she's broke and she's not. She revealed her financial situation to me (in general terms) and she pays all her bills and has paid off her credit card. She's got spending money. Spending money that she spends. She'll tell me on Tuesday that she's broke and then on Wednesday, go and drop $200 on new clothes. It's her choice as to what she buys, but I'm sorry, that doesn't qualify as broke.
What REALLY grates is that sometimes, someone will offer to buy her lunch for her. I would happily do that for a coworker or a friend if I knew she was tight on cash and wanted her to join us. But I think it's wrong to claim you have no money and take advantage of someone else's generosity.
Do these phrases bug you? Would you prefer that someone just say "It's just not in the budget" rather than "I have no money." Do you care?