The authority to forgive sins belongs only to God. However, Jesus claimed to have this power (Mark 2:5-7) and He is now passing it along to his church. Notice however, that He gives this authority only after giving them His Holy Spirit. It is only under the direction of the Holy Spirit that sins may be forgiven or retained.

Mark 2:5-7  When Jesus saw their faith, he said to the paralytic, “Son, your sins are forgiven.”  Now some teachers of the law were sitting there, thinking to themselves,  “Why does this fellow talk like that? He’s blaspheming! Who can forgive sins but God alone?”

This authority was given to the church as a whole and not to individuals. Though individuals sometimes exercised it in its most dramatic form (Acts 5:1-11, 13:6-12) under the direction of the Holy Spirit, they were acting on behalf of the church and not performing a ‘ministry.’ No individual man has the authority to forgive the sins of men. An example of the church as a whole exercising this authority is found in (2 Cor 2:5-11).

Acts 5:1-11  Now a man named Ananias, together with his wife Sapphira, also sold a piece of property.  With his wife’s full knowledge he kept back part of the money for himself, but brought the rest and put it at the apostles’ feet.  Then Peter said, “Ananias, how is it that Satan has so filled your heart that you have lied to the Holy Spirit and have kept for yourself some of the money you received for the land?  Didn’t it belong to you before it was sold? And after it was sold, wasn’t the money at your disposal? What made you think of doing such a thing? You have not lied to men but to God.”  When Ananias heard this, he fell down and died. And great fear seized all who heard what had happened.  Then the young men came forward, wrapped up his body, and carried him out and buried him. 
About three hours later his wife came in, not knowing what had happened. 
Peter asked her, “Tell me, is this the price you and Ananias got for the land?” “Yes,” she said, “that is the price.”  Peter said to her, “How could you agree to test the Spirit of the Lord? Look! The feet of the men who buried your husband are at the door, and they will carry you out also.”  At that moment she fell down at his feet and died. Then the young men came in and, finding her dead, carried her out and buried her beside her husband.
Great fear seized the whole church and all who heard about these events.

Acts 13:6-12  They travelled through the whole island until they came to Paphos. There they met a Jewish sorcerer and false prophet named Bar-Jesus,  who was an attendant of the proconsul, Sergius Paulus. The proconsul, an intelligent man, sent for Barnabas and Saul because he wanted to hear the word of God.  But Elymas the sorcerer (for that is what his name means) opposed them and tried to turn the proconsul from the faith.  Then Saul, who was also called Paul, filled with the Holy Spirit, looked straight at Elymas and said, “You are a child of the devil and an enemy of everything that is right! You are full of all kinds of deceit and trickery. Will you never stop perverting the right ways of the Lord?  Now the hand of the Lord is against you. You are going to be blind, and for a time you will be unable to see the light of the sun.” Immediately mist and darkness came over him, and he groped about, seeking someone to lead him by the hand.  When the proconsul saw what had happened, he believed, for he was amazed at the teaching about the Lord.

2 Corinthians 2:5-11  If anyone has caused grief, he has not so much grieved me as he has grieved all of you, to some extent—not to put it too severely.  The punishment inflicted on him by the majority is sufficient for him.  Now instead, you ought to forgive and comfort him, so that he will not be overwhelmed by excessive sorrow.  I urge you, therefore, to reaffirm your love for him.  The reason I wrote to you was to see if you would stand the test and be obedient in everything.  If you forgive anyone, I also forgive him. And what I have forgiven—if there was anything to forgive—I have forgiven in the sight of Christ for your sake,  in order that Satan might not outwit us. For we are not unaware of his schemes.

Actually, the verbs “are forgiven” and “are retained” are both in the perfect tense indicating an already-completed action. If the church is really following the Holy Spirit in this, it will only be declaring the forgiveness or retention of sins that have already been forgiven or retained in heaven.

The exception to this is when someone sins against you. Then you have the authority to forgive them and ask God to not retain guilt against them (Luke 23:33-34, Acts 7:59-60).

Luke 23:33-34  When they came to the place called the Skull, there they crucified him, along with the criminals—one on his right, the other on his left.  Jesus said, “Father, forgive them, for they do not know what they are doing.” And they divided up his clothes by casting lots.

Acts 7:59-60   While they were stoning him, Stephen prayed, “Lord Jesus, receive my spirit.” Then he fell on his knees and cried out, “Lord, do not hold this sin against them.” When he had said this, he fell asleep.

Even more specifically, there was a Rabbinical metaphor of binding and loosing by teaching and proclaiming that was adapted by Jesus in (Matt 16:19, 18:18) and this is a similar use. The church’s primary function along the lines of forgiving and retaining guilt for sins is the declaration of that forgiveness under gospel terms (Acts 2:38, 3:19, 10:43). The world needs to hear this message of good news which we have!

Matthew 16:19  I will give you the keys of the kingdom of heaven; whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven.”

Matthew 18:18  “I tell you the truth, whatever you bind on earth will be bound in heaven, and whatever you loose on earth will be loosed in heaven.

Acts 2:38  Peter replied, “Repent and be baptised, every one of you, in the name of Jesus Christ for the forgiveness of your sins. And you will receive the gift of the Holy Spirit.

Acts 3:19  Repent, then, and turn to God, so that your sins may be wiped out, that times of refreshing may come from the Lord,

Acts 10:43  All the prophets testify about him that everyone who believes in him receives forgiveness of sins through his name.”