Using AI to analyse 30,000 school policies across 1,000 school websites

As a school governor, one of my least favourite moments is the moment when someone has to volunteer to check compliance of the school website. Everyone tries to make themselves as invisible as possible. As one of the original creators of GovernorHub, I tended to be extra-visible: “James, can’t you just write a computer program to do it for us!?”. The answer was a resounding no, for a thousand reasons that basically amounted to “every school does their website differently, names things differently, talks differently”. Not going anywhere near it. No. Way.

But when ChatGPT burst onto the scene a year and a bit ago, one of our core insights was that suddenly previously inaccessible data sources, written in natural, human language, were available for computers to analyse. Or put another way, the pesky school website, in all its different forms, was suddenly open for business!

We decided to dip our toes into the water by doing a review of the policies on a website. What policies are there, when are they due for renewal, that kind of thing. Should be easy! And yes, doing schools one-by-one was quite easy. Doing it for thousands turned out to be hard. I won’t bore you with the details, but a year later, I feel like we’ve started to crack it. The mythical full website compliance check feels within our grasp.

We’ve now done policy reviews for over 1,000 schools, and Robin has “read” just shy of 30,000 policy documents! To mark this milestone, Joe and I decided to go rooting around in the data generated to see if we could uncover anything interesting. I feel compelled to put in a huge caveat here that this is not science! We’ve made no effort to make sure our 1,035 schools are representative. In fact, they’re definitely not. They’re nearly all academies for starters. And nearly all of the MATs that those academies are in have some relationship with The Key Group. But I still think there are some interesting insights to be gleaned. OK, with that off my chest… let’s go!

Our dataset of schools

Here’s how those schools are geographically distributed:

How many policies do schools publish?

It averages out to just under 30 per school. Here’s how that’s distributed, separated out by primary / secondary:

Yes, there is at least one school out there with 175 policies published on their website. No, I will not name them. But aside from that, schools tend to publish between 15 and 60 policies on their website, and it doesn’t much matter whether you’re a primary or secondary school - you still publish, and have to manage, lots of policies.

What policies are schools publishing?

What policies are schools publishing? Here’s a table of the top 20, ordered by prevalence across the websites:

It’s reassuring to see that every school we’ve checked has a Child protection policy. Of the policies that academies are required to put on their website, the most common missing one is Whistle-blowing procedures.

School policies around flexible working and mobile phones have been in the news recently. We’d normally expect those to be folded into other policies, but it’s interesting to note that 34 schools (almost all Primary) have published an explicit Flexible working policy and 111 have an explicit Mobile phones and devices policy.

We’d like to dive into this in a bit more detail - but I think it’ll have to wait for a future post!

How often are policies overdue review?

After categorising the policies, we had 19,061 documents that we were able to extract review dates from. 32% were overdue for a review. Here’s the breakdown for the most common policies:

Again, it’s great to see that schools are on top of their Child protection policies! Remote learning is an interesting one - the fact that nearly 2/3 are overdue leads me to wonder about the legacy of all the remote learning experience we gained during the pandemic.

If we compare primary vs secondary schools we get:

Secondary schools, in general, have fewer overdue policies than Primary schools. We looked at this from a few different angles but I think the following is the best reflection as to what is going on:

Roughly: the larger the school, the more on top of policies you are. One interpretation of this is that (as we saw earlier) the policy burden is more-or-less the same, no matter what size of school, and smaller schools will have fewer resources to throw at managing it. But another aspect could be that, on the ground, policies matter more for larger schools than smaller ones, and these numbers reflect that!

How often are policies reviewed?

We can also look at the gap between the last review date and the next review date to analyse how often policies are reviewed. Here’s the distribution of review cycles across our policies.

There’s not a lot to say on this. Yes, a couple of the policies really are on a 12 year cycle, we checked (one Animals on Site policy and an Online Safety policy). Nearly all policies are on 1 year cycles. We haven’t dug into this in depth, but it makes me wonder whether they all need to be. If not, shifting to longer review cycles could ease the administrative burden.

Does Ofsted have an effect?

Finally, onto Ofsted. Do schools with a higher Ofsted rating have fewer overdue policies?

It does look like “Outstanding” schools are more on top of their policies, but there’s nothing really separating “Good” and “Requires Improvement”. We can also look at the time since the last Ofsted inspection. Does this matter? From my years as a Governor, I’d expect schools to be more on top of their policies in and around the Ofsted window, and less so when there’s a few years of safety between them and Ofsted. Here’s the percentage of overdue policies based on schools that have had an inspection in the last four years:

There’s a distinct hump - fewer overdue policies close to the Ofsted inspection (and closer to a presumed one at around the 4 year mark). If we define being “in the window” as within 6 months of an inspection, or after 3.5 years of the last inspection then we find:

The difference isn’t huge, but it seems my instincts were right! I don’t know whether this points to Ofsted sharpening up schools, or whether it points to policies being viewed as an unnecessary tick-box exercise. And I’m not about to wade into that debate…

Next steps

We’ve just scratched the surface here. But we don’t think this kind of analysis has been done before (it would have been incredibly time consuming in the recent past), and hopefully there’s a lot more to come.

Robin is a system that can extract this kind of data en-masse and we’re only beginning to get to grips with the possibilities! There’s a treasure trove of information on school websites about how they operate - data that we’ve never had access to at a national level before. For now, our sights are set on both going more broad than just policies (hello, website compliance…), and also going deeper into the policies themselves, to analyse and report on what is actually said in them.

If you want to get this sort of policy insight across your MAT, including the finer details of what policies are where, then Robin’s policy review may be just the thing for you. You can check out our demo here, and feel free to get in touch with us at [email protected]. We’d love to chat.