While the newest, all-encompassing, iteration of the Office for Students’ Teaching Excellence Framework (TEF) is still out for consultation (stage two closes on 2 January 2027) in releasing data and a draft rubric for the exercise OfS must surely know that it has kicked off numerous “shadow TEF” iterations up and down the sector.

With potential endpoints including recruitment caps and the removal of recourse to strategic funding, institutions would be foolish not to be wargaming potential awards based on the current release of data. Findings will inform decisions on recruitment, on portfolio design, and on the size and shape of university activity. Universities prefer to be ahead of these consequences in order to avoid them, and the earlier information is available, the better.

That said – it is not possible to model the entire proposed TEF. Although it has slipped away from the post-Pearce review enhancement focused ideal the exercise is still not as mechanistic as earlier attempts. Provider and student submissions will make a meaningful contribution to the two awards (remember Student Experience, and Student Outcomes, with no “overall” award) and this remains unknowable.

A part of the purpose of the provider statements is to address perceived deficiencies in the data. Here a close examination of the individual indicators on each side can help target explanatory effort. As with all TEFs, much of the data refers to cohorts who have long since departed from the university: and it is very likely that any issues have already been identified and remedied within the normal course of institution-level quality assurance and enhancements. Sometimes the statement will involve pointing to what has already happened – or is in process – and providing data that demonstrates that these efforts are working.

Walking through the data

I’ve gone over the broad design of the exercise and some immediate issues I spotted elsewhere. This article is going to go deeper into what we are looking at and why. I will focus on the TEF element, though the baseline review works in a similar way and could be seen as a mini-TEF that can only award bronze or “needs improvement” ratings.

There are six metrics that apply to Student Experience, (SE) and five that apply to Student Outcomes (SO).

Student experience

In SE, each metric (one of six National Student Survey thematic totals) is weighted equally. In total these add up to 50 per cent of the overall SE rating, with the student and provider statements contributing the other half. All of the SE metrics are compared with a benchmark, generated using the likelihood of particular groups of students (based on intersections of personal characteristics) to respond positively to an NSS question (specifically question 14, though it would really have been better to use the overall rating had OfS not inadvertently scrapped it). It is the difference between the benchmark and the actuality (the indicator, which is based on a four-year average of NSS scores) that is what the process bases quality judgements on.

Rather than the old positive and negative flags, we deal with four initial interpretations for each indicator. “Outstanding” sits at the top, with what we might call “excellent” (described as “high quality or high rates of success), below; “satisfactory” is the minimum, and “unsatisfactory” one at the bottom you want to avoid.

Broadly speaking, these shake out as follows as a starting point:

• Outstanding: indicator materially (>2.5 percentage points) above benchmark
• Excellent: indicator broadly in line (between 2.5 per cent above and 2.5 per cent below) with benchmark
• Satisfactory: indicator materially but not substantially (between 2.5 percentage points below and 5 percentage points) below the benchmark
• Unsatisfactory: indicator substantially (more than five percentage points) below the benchmark

I should add that these are also subject to an analysis of the quality of statistical confidence available concerning the distribution of students/graduates across the benchmark.

Student outcomes

Over in Student Outcomes world we can split the metrics between two familiar friends – continuation and completion – and a basket of three metrics that make up the judgement on post-study experience. Continuation and completion are constructed from multiple years of data in the old familiar way, and are the only metrics to retain a B3-style numeric threshold (which have shifted since we last saw them, and is now joined now by a “higher threshold” five percentage points higher, and a “very high” threshold set at ) alongside a benchmark.

Here things are a little more complex, but bear with me:

• Performance is unsatisfactory if you are materially (2.5 percentage points or more) below your benchmark, and your indicator is below the (lower) numeric threshold.
• You get a satisfactory if you are broadly in line with or materially above your benchmark (anything above 2.5 percentage points below) and below the numeric threshold; or if you are below the higher threshold and and materially (more than 2.5 percentage points) below your benchmark
• To be rated excellent you indicator needs to be above the minimum threshold and broadly in line with your benchmark (anything more than 2.5 percentage points below); or you can be materially (more than 2.5 percentage points) below your benchmark if you are above the higher threshold.
• And what about outstanding? Materially (more than 2.5 percentage points) above your benchmark and above the numeric threshold; or inline (or better) with your benchmark and above the “very high threshold” (of 95 per cent).

The post-study experience metrics are progression (the Graduate Outcomes survey “good destination”/graduate job measure familiar from current B3 and TEF), earnings (based on LEO data up to three years after graduation, compared with a national median for 25-29 year olds and benchmarked via geographic salary data linked to where your graduates currently live), and use of skills (the Graduate Outcomes question). There are no thresholds here – these metrics are assessed against benchmarks in the same way as the NSS ones.

For SO, the statements are a little less important, and rather than having a certain proportion of the overall award devoted to them we are instead encouraged to explain and caveat the data sources. To give the obvious example that we all know, Norland University will score poorly on progression because its graduates don’t get “graduate jobs” – the statement will be where (once again) Norland explains this.

Overall awards

As above, there is an award for each aspect but no “overall” award, and it is not possible to reliably construct the awards from the metrics on student experience as the statements count for such a lot of the judgement.

For SE, all of the criteria are rated equally and considered collectively, but where more than one of the criteria are judged to be unsatisfactory it would be most likely that you get a “requires improvement” award. You could still end up there with just one unsatisfactory, but assessors will make a call on the “severity, scale, and significance” of the problem.

For SO, we are told that to get a gold you would need at least two “outstanding” criteria, of which one needs to be a post-study metric. And if one or more criteria are “unsatisfactory” you are looking at a “requires improvement”.

On both aspects there are many other assessment criteria: more recent evidence is generally seen as more important than older data, and assessors will take a deeper interest in split metrics (referring to particular groups of student or provision characteristics) where an overall criteria rating applies to less than 80 per cent of all students. Your award arrives with an assessment report – and appeals are only available to those rated bronze or “requires improvement” where new data is available or there has been a procedural irregularity.

A presentation of the data and process

What I’ve built below attempts to capture the process as described above, and offer an indication of what (based on the supplied logic) criteria and award judgements might be. This is, of course, only one part of the process – we can’t model the assessment process or reliably guess how submissions might tip the scale (we can get an indication of what the split metrics look like, which might help).

Start with choosing a provider, and setting population options along the next row – note that to make both aspects work you need to choose both “all undergraduates” (which affects student experience metrics), and one other (which affects student outcome metrics).

Below that on the purple row are the split indicators – by default set to overall which gives you what will form the bulk of the main score. If you want to get deeper change the “split indicator type” first and then the “split indicator label” – don’t panic if the chart goes blank in the middle. I highly recommend looking at subject areas, and at partnerships (the latter even includes named partners in most cases!)

At an award level not every judgement is available based on the current guidance as I read it – the proposals simply aren’t as mechanistic as we have seen previously. In many cases I can only say you are at bronze or above

What’s missing is any hardcoded test of statistical significance – I’ve included the standard deviation and the number of students/graduates in each indicator (the denominator) to allow you to spot when this might occur. For this reason, the tool marks harsher than OfS probably will (it’s nice to see OfS like this sometimes).

I hope this is all of some help – and is a bit more user-friendly than the OfS version. If you spot any problems please do email me on [email protected].