How can we determine an appropriate rate of decay for our scoring algorithms?

Question

We build scoring algorithms for different customers in an enterprise environment. Typically they tend to measure most goals in either weekly, monthly, or quarterly intervals. We use a scoring algorithm to score individuals based on performance relative to goals and other metrics.

We assume that the rate of decay should vary based on the goal interval they primarily use. Meaning, a company focused on weekly goals should have a faster decay rate than one focused on quarterly goals.

What's a good way to determine how quickly to begin the decay process based on past performance relative to metric interval length?

Answer 1

The game League of Legends utilised an Elo Rating System which incorporated within it a decay system. The decay worked on a user score and reduced it dependent on both user performance and time. Here, user score was an indicator of skill level and was used in another formula to calibrate the actual scores they received from other in-game activities. So, while this system may not be directly applicable in a customer engagement scenario, its worth looking into first.

The system calculated score based on this formula:

Ra_Old = Ra_New + K(Sa-Ea)

The sytem had various category of players such as:

• Bronze: Between 0 and 1149 (Team: 0-1249) (Top 100%)
• Silver: Between 1150 and 1499 (Team: 1250-1449) (Top 68%-13%) Majority of Active Player Base
• Gold: Between 1500 and 1849 (Team: 1450-1649) (Top 13%-1.5%)
• Platinum: Between 1850 and 2199 (Team: 1650-1849) (Top 1.5%-0.1%)
• Diamond: 2200 and above (Team: 1850+) (Top 0.1%)

and varying decay rates based on certain player properties:

• Elo decayed at a rate of 50 Elo for Diamonds, 35 Elo for Platinums, 25 Elo for Golds, 10 Elo for Silver, and 0 Elo for Bronze for every 4 consecutive weeks of inactivity and every 7 days thereafter.
• For normal rating, inactivity was defined as no activity in any queue.
• For ranked rating, inactivity was defined as no activity in the specific queue
• Ranked decay only applied to people who were ranked above 1400 rating.
• The decay timer was reset after a game was played in that specific queue.

The major changes that will be required in the case of performance scoring will be:

• Less aggressive scoring than in this system
• Decay rates calibrated to maintain a bell curve distribution centred depending on the level of competition you wish to achieve.
• A bell curve spread such that the leaders are not out of reach of the majority of players as that may end up demotivating the users.