Fork me on GitHub
Math for the people, by the people.

User login

The integration of users ranking/object ratings with PM

Primary tabs

The integration of users ranking/object ratings with PM

As soon we are going to implement users trust management/rating features to the PM, we would like to briefly go over our vision of this part of the PM portal. Any feedback or questions are welcomed.

The object view will be modified so that it contains two small icons which represent:

1. Ranking of user according to HITS algorithm (value 0 – 1) who is owner of this object – this can be understood as user quality measure
2. Current rank of the entry according to ongoing object ranking – as there will be a way of rating objects, we will present this information.

Specifically, we will implement two icons placed in the title bar of object. Icons will represent small bars, the higher bar, the better entry owner/object is. Even though icons will be presented in title bar of an object, they will be small enough not to change its current size. Below the object, one will have an option to rank it. The ranking will be in the form of ‘stars’ – user will have an option of assigning 1 – 5 stars for different aspects of object. We will ask similar rating questions as in survey which we did in the past: clarity, correctness, pedagogy, precise and language. Rating will implement aging. It means that, every time an edit is made to the object, value of past ratings will be decreased (probably by 5%-10%). Such a mechanism is required as quality of objects can change as they are edited.

When searching, users will define their default scoring functions for results. Currently, results are scored by TF-IDF measure. Users will have an option of defining the scoring function as function of TF-IDF score, users ranking (HITS) and object rating (stars). The final scoring function will be thus a linear combination of the three above. After given user defines such a scoring function, results presented in response to his query will be ranked using this new scoring function.

Pawel


Good point here I guess. But still, we have to think about some way of dealing with votes aging. Simply, when object is edited, its quality can either decrease or increase. Then, if we weighted old votes the same as most recent, it will not be a good idea either. On the other hand, if entry is not good, it will not have high ratings. The only way for making object better is to correct errors - it gives a chance for improving rating at the cost of older votes - sounds fair, doesn't it?
You can also take a look at wider context - we are implementing user trust/quality measure. There, if you don't make corrections to your objects which are suggested, you will not get good score. Therefore, even though aging will somehow discourage people from making corrections, I think other arguments will encourage them to do so.
One more thing. We might differentiate between type of corrections. Then, major changes would decrease old ratings more than minor changes. But those are just details of the implementation. I wanted to show you a big picture of the idea.

Pawel

I think it would work in favor of your concerns.

If the entry is not so good, and is getting gradually improved, then presumably the old ratings will reflect a lower quality and new ratings will reflect a higher one.

Then, the linear falloff of rating weight in calculating the overall rating will correctly give bias to the higher ratings.

In the general case, ratings which are more recent will simply have higher weight, and ratings which are older will have lower weight. There will be no built-in bias towards a particular score or another.

apk

> Rating will implement aging. It means that, every time an edit is made
> to the object, value of past ratings will be decreased (probably by
> 5%-10%). Such a mechanism is required as quality of objects can change
> as they are edited.

If I understand correctly, that doesn't sound like a great idea. Most of the times, entries are edited to improve their content, not to decrease the quality. A system that reduces the rating of an entry whenever it's edited discourages the owner from modifying it, which is probably the opposite of what we would want.
And, specifically, I wonder how (if so) do you plan to differentiate the correction of a typo from a major modification, and how 10 consecutive corrections of minor typos affect the rating.

Subscribe to Comments for "The integration of users ranking/object ratings with PM"