Ranking mechanisms - ideas/comments?
As this time of year is back, the Google Summer of Code 2008 has just started. This year I will be working on improving last year's work. Therefore, I would like to start this discussion and hope to get feedback from you on what to improve, what are your thoughts/general opinions about both ranking meters (entries ranking and users ranking), or what additional features you would like to have.
Entries ranking system.
I have checked discussion from last year, and it seems that you mentioned two additional features: include meter for entry ranking in list of owned objects and include two modes: browsing and ranking to prevent bias. The latter feature was proposed by Wkbj79:
I would like to propose a compromise to this issue of whether we see ratings or not. It would be nice if we had a "browse mode" in which users can see ratings but cannot rate entries, and a "rate mode" in which the ratings are hidden and users can rate entries.
Do you have any additional ideas?
Users ranking system.
As you probably know, one of meters for objects that you see in encyclopedia shows user ranking. I know that some of you were disappointed with the quality of ranking that is produced (or with that users were classified to "medium" confidence instead of "high" etc.). Therefore, I would like to improve this feature. As you might remember, current algorithm uses HITS method to calculate the rankings. You can obtain more details after clicking on the meters. To evaluate this method, I have prepared a list of top-600 users found by HITS. It is available at http://www.mathcs.emory.edu/~pjurczy/users.htm. I think that the biggest problem is with decoding score value to meaningful text label (do not focus on score value so much, as this score is HITS value, rather focus on position in ranking that users have). I have looked at the list and observed that the current decoding thresholds that decode values to text labels can be too strict. Therefore, in the ranking I am also proposing new thresholds. Also, what do you think about the ranking? Is the order of users corresponding to your feelings about the community?
The last point I would like to rise is overall evaluation of the ranking systems. Do you have any ideas for evaluation of usefulness? One method we can use now for users ranking is just to take the current ranking of users with points and check the correlation of this ranking with the HITS ranking I am calculating - any additional ideas? Do you think we should run an evaluation survey? If yes, what kind of questions you would ask there?
Thanks a lot for all input!
- Planetary Bugs
- Research Topics
- LaTeX help
- Math Comptetitions
- Math History
- Math Humor
- PlanetMath Comments
- PlanetMath System Updates and News
- PlanetMath help
- Strategic Communications Development
- The Math Pub
- Testing messages (ignore)
- Other useful stuff