There's no single answer to this question. It all depends on what you as a caster consider 'decent'. It also depends heavily on your schedule, your pre-existing community interactions, what time of day you cast and a hundred other factors - not all of which you will have control over.
Depending on all of these factors, building a solid community can take anywhere from a few months to a few years. The best advice given by streamers is often the most difficult to act upon. Don't look at your numbers, at least not while you're on cast. Fill silences and give potential new community members something to interact with or think about, both chatter and lurker.
If you are going to look at your numbers per cast to track growth, there is a rough guideline you can aim for. Best case scenario, you should aim to retain approximately 1% of your follower number as your viewership. So if you have 100 followers, you should only really be looking to get 1 viewer. If you're ahead of that 1% then you're ahead of the game. Keep with that momentum and build, making sure to shape your community as you go. If you're not quite making this yet, don't be disheartened. Don't be afraid to shake things up and change your cast to alter those factors, see if you can achieve your goal another way/time.
I will say that there is one thing you should make sure you keep on top of if you are looking to build a regular following. To reach your goal, ideally you should be keeping schedule consistency at the top of your list. The more people know when to come catch you, the more they'll do so. Think of your favourite TV show (past or present). You always knew it was on at X time on X day. Make your stream the same kind of unmissable entertainment.