Native speakers correct me if I'm wrong. But as far as I know if you say 'I went to the church' that means that you didn't go to a religious service, while if you say 'I went to church' means you went to a religious service.
The same happens with other words such as prison, school, university, hospital ... You say 'He went to the prison to visit his brother' but 'The thief went to prison'. In the first example he is not a prisoner, just a visitor.
Another example: He went to hospital / He went to the hospital. In the first example he is ill or injured so he is in hospital, while in the other one he is not ill, he is there as a visitor.