Regressions

One of the primary uses of mozci is to help detect which tasks and/or tests (if any) a push has regressed. Since we do not run all tasks on every push and because of other factors like intermittents, this problem is more difficult than it first appears. In fact mozci can make very few guarantees and so has to rely on probabilistic guesses.

This page will help explain how regressions are calculated by introducing concepts one at a time.

Definitions

There are currently two different vectors of regression that mozci can check for: label and group.

  • label - is a task label (e.g test-linux1804-64/debug-mochitest-e10s-1)
  • group - is a grouping of tests, typically a manifest (e.g dom/indexedDB/test/mochitest.ini).
  • runnable is the unique label identifying a set of tasks, or the unique group identifying a set of tests.
  • classification - an annotation that Sheriffs apply to tasks manually. It is also known as “starring” because it puts a little asterisk next to the task in Treeherder.

Runnable Summary

Thanks to retriggers, each runnable can run multiple times on the same push. The collection of labels or groups of the same type that ran on a push is called a runnable summary. For instance, if all the runnables on a push passed, then the status of the runnable summary is also PASS. Likewise if they all failed. If at least one instance of a runnable passes, and at least one instance of a runnable failed, then the runnable summary is said to be intermittent.

The GroupSummary class implements this logic for groups and the LabelSummary implements the logic for labels. Both classes inherit from the RunnableSummary abstract base class.

All instances of RunnableSummary have an overall status and an overall classification.

Candidate Regression

A candidate regression is a runnable which meets the following criteria:

  • At least one instance of this runnable failed on target push (i.e, the status of the runnable summary is either FAIL or INTERMITTENT)
  • The overall classification of the runnable summary is either unclassified, or fixed by commit. This means runnables classified as a known intermittent are not candidate regressions.
  • For runnables classified fixed by commit, the referenced backout backs out the target push and not some other one.

OR

  • The runnable ran on a child push (up to MAX_DEPTH pushes away), and is classified fixed by commit.
  • The classification references a backout that backs out the target push.

Candidate regressions are the set of all runnables that could possibly be a regression of this push. This does not mean that they are regressions. Just that they could be.

The set of candidate regressions can be obtained by calling Push.get_candidate_regressions().

Regression

A regression is a candidate regression that additionally satisfies the following criteria:

  • The candidate regression is not marked as a regression of any parent pushes up to MAX_DEPTH pushes away.
  • The condition total_distance <= MAX_DEPTH is satisfied. This condition is explained in more detail below.

Note

Distance Calculation

The total_distance is the number of parent pushes we need to go back to see the runnable plus the number of child pushes we need to go forward to see the runnable. A total_distance of 0 means the runnable ran on the actual target push.

The total_distance can be modified in certain scenarios:

  1. The push was not backed out => total distance is doubled.
  2. The runnable was intermittent => total distance is doubled.
  3. The runnable was marked as fixed by commit referencing a backout that backs out the target push => total distance is 0 even if it didn’t run on the target push.

These modifications help us deal with (un)certainty in special easy to detect circumstances. The first two make a candidate regression less likely to be treated as a regression, while the third guarantees it.

Regressions can be obtained by calling Push.get_regressions().

Likely Regressions

A likely regression is a regression whose associated total_distance is 0. In other words, we are as sure as we can be that these are regressions.

Likely regressions can be obtained by calling Push.get_likely_regressions().

Possible Regressions

A possible regression is a regression whose associated total_distance is above 0. In other words, it could be a regression, or it could be regressed from one of its parent pushes. We aren’t sure. The higher the total_distance the less sure we are.

Possible regressions can be obtained by calling Push.get_possible_regressions().

Note

Candidate regressions that aren’t also possible regressions could still technically be real regressions. Mozci just thinks the likelihood is so low they aren’t worth counting.