Hi Tunde.
There are several ways to get started recording. It depends on your technological "comfort zone", how complex you want your recordings to be, and of course, budget.
If you're doing a solo act, maybe a piano or guitar plus vocal. you can get by with something simple like the Tascam Portastudios, or the Zoom recorders. If you have a reasonably current computer, you can use that with a basic audio interface. Add a few microphones, and recording software (like ProTools First, Reaper, or Cakewalk) and you can be recording.
If you want to record a full choir, with a group, live in the church, you may need a more extensive setup.
Look through the forums here, and read for a bit before you start throwing money on the counter! Lots of people were in your same situation.
Good luck