If a person punches you and you punch back to defend yourself against the person attacking you again, it is self defense. If you punch somebody 3 hours after they punched you, that constitutes as assault. At that point you are no longer defending yourself as there was no percieved threat or harm at that point.