Firestone: Online Idle RPG – How to Calculate the Probability that the Game is Wrong

How to Calculate

There is a tool from mathematical statistics called the chi-squared test (see Wiki).

What you do is compare your observed statistics with the assumed probability distribution from the game and calculate the probability that they match.

Example: Assume the game tells you that an event occurs with 5% probability. However, after 100 observations the event didn’t occur a single time. What is the probability for that?

We have two categories in the chi-squared test:

• Category 1: The event occurs. The null hypothesis says this should happen 5 times out of 100.
• Category 2: The event doesn’t occur. The null hypothesis says this should happen 95 times out of 100.

The chi-squared value is calculated as: (#ocurrences – #expected)^2 / #expected summed over all categories. In our example:

chi-squared = (0 - 5)^2 / 5 + (100 - 95)^2 / 95 = 25/5 + 25/95 = 5.26

Now how probable is this? You look up the probability in the table of p-values for the chi-squared distribution (see Wiki). In our case we have 1 degree of freedom, so we look it up in the first row of that table. A chi-squared value of 5.26 lies between 3.84 and 6.63, so the probability of that happening is somewhere between 1% and 5%.