The value of Ratings and the trouble with it
When we started to develop our Open Source Directory we were taking an Enterprise perspective from start on. We wanted to develop a tool that helps CIOs and IT decision makers to easily find and pre-select open source technology. For this purpose we defined our rating system. We went one step further with the online version and opened the platform for EOS users to also enter their rating. In an ideal world this “user rating” would follow the same rules and guidelines as the “Optaros rating”. Both the “Optaros rating” and the “user rating” are trying to assess the “enterprise readiness”. We defined 5 levels of “enterprise readiness”:
4 stars:
Product/projects matches or is superior to best - proprietary/closed source - available and widely used products. It has proven to be a standard in a specific category that you can’t pass.
3 stars:
Product/project is mature, fulfills the important requirements and is supported well, it is enterprise ready and conforms to typically found needs/requirements in enterprise production environments
2 stars:
The right tool for the many situations, more investigations needed, not top league yet. Weaknesses may be driven by technology or lack of functionality or support. A proof of concept is recommended.
1 star:
Not recommended (yet) to be used immediately in a broad and enterprise wide context, other than early proofs of concept or deployment in a controlled way (e.g. as component that understood by the developer)
0 star:
Not recommended to be used in an enterprise, probably not worth further investigation. These technologies are not shown in the directory at all.
When looking at the “user ratings” people often seem to have only one objective, bumping up the technology to 4 stars. But it takes a lot to reach the 4-star-level in reality. With this we would like to ask people doing ratings to stick to the criteria defined and make a reasonable judgement.
We are pleased to announce new relationships with payment service providers that are friendly to the open source community. If you are one of our EOS Directory fans and need atm, pos systems, credit card processing, or business funding, click here for more information. Good news is they are proud to always answer the phone!
This entry was posted by Bruno von Rotz on Friday, July 13th, 2007 at 8:18 am and is filed under Making of EOS. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.






July 13th, 2007 at 8:26 am
I do absolutely agree! Only with sticking to the criterias the “user rating” will be meaningful. I wonder whether asking the people doing the rating to back up their decision might help. But, good work, go on with it. EOS is a very useful resource for me.