Building choirs for a congregation new to music can be very difficult at first. Many people are excited to have music added to the liturgy at all and would like to just participate from their seats for a while. It's can be an uphill climb for sure but don't be disheartened. As people become comfortable with the repertoire they will be willing to sing more.
On top of that, you never know how many people are interested in joining your musical efforts if you don't ask. There are a bunch of different ways to find willing participants but the opportunity has to be available to them. People will probably not come to you on their own.
1. Make sure announcements are made when you want to try to bring new members in. Be honest with people and make sure they know what is expected of them when joining.
2. Have recruitment nights. Invite anyone who might be interested and just have a good time with them and the music. Share some simple songs with them and help them sing together. Give them the experience that they might have being a member of the group.
These are just a couple of ideas and there are so many more depending on the size and feel of your parish. Have patience and keep it up. People don't want to join something that isn't good already. So in the mean time, make the music the best it can possibly be and realize that people will come when they are ready.
You may also need to hire extra musicians if you can afford it. It can create a group that people will want to join.
It isn't easy, so stick with it.