Yes, altruism decreases your fitness. But you don't need maximal fitness to pass your genes forward; you only need a certain, finite amount. Beyond that, you can afford altruistic behaviours.