You both have made mistakes and still think that the relationship is worth trying to save.
Sometimes affairs and wrongdoings are hard to put in the past while you are trying to rebuild and that is understandable.
The best advice that I can give is marriage counseling for both of you. None of us can go back and change things that we have done in the past. We can only accept the things that we cannot change and change the things that we can. The hardest part is knowing the difference... and a professional can help you both.
As far as your friends are concerned, you do not owe them any explanations. Just tell them that you both have made mistakes and are trying to work things out and would appreciate their support. If they continue to knock down what you are trying to rebuild, then it's time to look for new friends.