Firstly, after analytics and mesurement, you need to understand that there are three core reasons to open-source software:
* to solicit improvements (see Linux)
* to facilitate adoption (through implementation transparency, see OpenVPN and TrueCrypt)
* for personal reasons (to brag or to support political agenda, see libevent or IO language)
These can be mixed and matched, but it typically helps to understand WHY you are open-sourcing. That's a first step.
Second step, if you want to make some online money, the main thing is determining your business model. You can make money off the open-source either via the support or via dual-licensing.
A. Support model does not really scale, because in order to earn twice the online money, you have put a double effort. It is also more of a sales task, which you may or may not have an inclination or an ability to so.
You've got several choices:
1) Sell Training
Write books, online training, seminars, whatever, and sell that as an adjunct to your open source project. Of course, those can be open source as well.
2) Sell Customizations
Offer to develop online custom features or just consult on deployment. Some of those may be rolled in to a future version of the existing package if that makes sense.
3) Sell Support
Get people to buy support for the package and offer telephone/email support for issues. If the application is critical to a business, they may pay to have support on hand.
4) Sell access to the code under non-GPL license
Some applications are release GPL, but offer the option of paying to get a closed source commercial license.
5) Split the package in to open and commercial packages.
Bundle the basic system as open source and then have add-ons that are commercial. This is sort of getting them hooked on the free version and then hoping they grow in to needing the features of the add-ons.
B. Dual-licensing is a way to go for online money making, but it implies that the code is non-trivial, solid and mature. Otherwise it does not make any sense for a 3rf party to become dependent on something that's not quite ready with an uncertain future. This automatically implies that you should not be open-sourcing the code that needs work.
Keep in mind that there are online money making ways for open source projects because it's often possible to find someone willing to purchase the open source projects as is from you.