On GOV.UK Pay, we're evaluating options to improve our monitoring and observability capabilities which help us ensure GOV.UK Pay is efficient and reliable.

In this post we show how to set up a new AWS feature called CloudWatch cross-account observability using Terraform.

GOV.UK Pay runs on AWS and like many organisations, it has multiple AWS accounts to separate our test, staging and live environments, as well as separating our deployment and testing infrastructure. We use a combination of Cloudwatch for our AWS generated Infrastructure metrics and a SaaS product to aggregate our application metrics. One of the downsides with this setup is that engineers need to log into multiple AWS accounts and the SaaS UI to view metrics, making it difficult to correlate behaviour across environments. We're considering ways to improve our monitoring systems and a recently launched AWS feature called "cross-account observability" looked like it could help fix this problem. It allows a single AWS account to access instrumentation data from multiple AWS accounts. The potential benefits of this are: 

• it gives all technical staff on the team access to CloudWatch monitoring in one place, removing the need to have production access in order to view production metrics
• it avoids the need to log in and out of different accounts to view metrics
• we can potentially codify all CloudWatch alarms and dashboards in one place which could make for simpler code organisation
• we could make correlation and contextualisation of data easier
• there's no extra cost for logs and metrics — traces can be shared with one monitoring account at no cost, but will incur costs if shared with additional monitoring accounts — this will provide cost savings for us as our monitoring SaaS product pulls data from the CloudWatch API which comes at a cost

When exploring CloudWatch cross-account observability, we wanted to configure everything using Infrastructure as Code using Terraform and avoid having to manually configure ("click ops") anything at all. Unfortunately, all the examples found on the internet used the click ops method, and there is sparse documentation on how to configure it in Terraform. Thankfully, we were able to configure everything in Terraform and thought it would be good to share our solution with the world!

Configure a monitoring account

A source account is an individual AWS account that generates observability data for the resources in it. A monitoring account is the account that can view and interact with observability data generated from source accounts. Configuring the monitoring account requires us to configure a Sink and a Sink policy.

This is quite simple (note you’ll need to supply the source account IDs):


variable "source_account_ids" {
type = list(string)
}

resource "aws_oam_sink" "monitoring_account_oam_sink" {
name = "ExampleSink"
}

resource "aws_oam_sink_policy" "monitoring_account_oam_sink_policy" {
sink_identifier = aws_oam_sink.monitoring_account_oam_sink.id

// See https://docs.aws.amazon.com/OAM/latest/APIReference/API_PutSinkPolicy.html for examples
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = ["oam:CreateLink", "oam:UpdateLink"]
Effect = "Allow"
Resource = "*"
Principal = {
"AWS" = var.source_account_ids
}
Condition = {
"ForAllValues:StringEquals" = {
"oam:ResourceTypes" = ["AWS::CloudWatch::Metric"]
}
}
}
]
})
}

Once applied, you should be able to verify in the AWS console that the monitoring account has been enabled:

Configure source account(s)

For each source account you’ll need to:

• create a link to the monitoring account
• enable the AWSServiceRoleForCloudWatchCrossAccount role — which is created by CloudWatch when the monitoring account is enabled — in the monitoring account to assume the CloudWatch-CrossAccountSharingRole role in the source account.

The terraform configuration for this:


variable "monitoring_account_sink_identifier" {
type = string
}

resource "aws_oam_link" "source_account_oam_link" {
label_template = "$AccountName"
resource_types = ["AWS::CloudWatch::Metric"]
sink_identifier = var.monitoring_account_sink_identifier
tags = {
Env = "test"
}
}

locals {
names = [
"CloudWatchReadOnlyAccess",
"CloudWatchAutomaticDashboardsAccess",
"AWSXrayReadOnlyAccess"
]
}

data "aws_iam_policy" "policy" {
for_each = toset(local.names)
name = each.value
}

variable "monitoring_account_id" {
type = string
}

locals {
policy = {
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Principal" : {
"AWS" : var.monitoring_account_id
},
"Action" : "sts:AssumeRole"
}
]
}
}

resource "aws_iam_role" "aws-cloudwatch-metrics-role" {
name = "CloudWatch-CrossAccountSharingRole"

assume_role_policy = jsonencode(local.policy)
}

resource "aws_iam_role_policy_attachment" "policy" {
for_each = toset(local.names)

policy_arn = data.aws_iam_policy.policy[each.key].arn
role = aws_iam_role.aws-cloudwatch-metrics-role.name
}

Once the above has been applied we can now go into our monitoring account and enter a source account number to see metrics from the source account:

Viewing a cross account metric

With the above set up, we were able to view cross account metrics on our self-hosted grafana dashboard on our monitoring account:

Here we are viewing a CodeBuild metric from the “test-12” environment (which is in the “test” account) from our monitoring account. 

Conclusion

We were able to collate metrics from AWS resources from multiple accounts (in the same region) into one designated monitoring account. We were able to configure this completely using our infrastructure-as-code tool, Terraform. Although we focused specifically on metrics here, this should work for logs and traces too. Although we have yet to make a decision on the solution, the benefits to be had from this AWS feature seem to be compelling at the moment.

It is worth mentioning the following caveats:

• terraform version 1.x is required at minimum
• terraform AWS Provider 4.62.0 is required at minimum
• propagating metrics across different regions and accounts is not possible at the moment — for example, we couldn’t link metrics in eu-west-2 for one of our source accounts to our monitoring account in eu-west-1.

Hi there! I joined GDS back in December 2022 and I’m really pleased to be writing my first blog post. I came to GDS because both the organisation and I share a passion for driving positive impact for citizens and I’m so happy to have seen so much evidence of that happening here in the last 6 months.

I’m responsible for GDS’s digital service platform products and my teams exist to serve the needs of government users, providing cost-effective, time saving digital solutions that mean there’s no need to reinvent the wheel, go to market through lengthy and costly procurement exercises, or manage the risks of building from scratch in-house.

Creating common platforms means understanding the issues faced right across government services, and to this end we are constantly learning and sharing with government teams both in the UK and around the world. We take our learnings and apply them directly to our product development process, so as to create solutions which meet the needs of many, up to and including becoming Digital Public Goods. Additionally, we put a huge amount of effort into the maintenance and operations of our product suite, making sure they remain secure, accessible and available to our users.

In celebration of all this, I am so pleased to share more about how we’re helping government teams transform services and give our readers an update on what’s happening next.

What’s been delivered

• GOV.UK Pay has onboarded 163 services in the last 12 months, including the Forestry Commission and NHS Business Authority, enabling them to take payments through online digital services — in total, we saw 23m payments processed and £1.3bn cash value handled over the last financial year
• GOV.UK Pay also launched a new Recurring Card Payments function - initially with Kent County Council and with more public services to follow - which enables local residents to save their payment details for use in ongoing regular payments
• there are over 1,300 organisations as part of nearly 7,000 public services using GOV.UK Notify and 99.99% of all messages are sent within 10 seconds — 86% of our users said in our latest survey that they were satisfied or very satisfied with our product and this is a real testament to our continued and robust service performance
• GOV.UK Design System released GOV.UK Frontend v4.7.0, which includes a new Exit this page component to help vulnerable users quickly exit a page or service
• we released Version 13 of the GOV.UK Prototype Kit, which makes it easier to update the kit, create and make changes to prototypes — we’ve also simplified the installation and tutorial to help users get started
• we supported delivery of Emergency Alerts, a UK government service that warns you if there’s a danger to life nearby
• we recently published a new site that brings all of our products together in one place to make it easier for government users to find what they need from us

What we’re doing next

With all digital products, it’s important to continue developing them based on user needs, even when they are well established like GOV.UK Pay, GOV.UK Notify or GOV.UK Design System.

GOV.UK Pay already offers Apple Pay and Google Pay to central government digital services, but soon we’re releasing the same mobile wallet payment types to local authority services and this will be a great benefit to people who are paying for government services on the go, like paying for Clean Air Zone charges.

Later this year we’ll also be investigating how GOV.UK Pay might offer open banking, which means that people will have the option to pay for services conveniently using their own banking app. 

GOV.UK Notify is also launching the ability to add attachments and QR codes to letters, which will enable more government services to use the product to communicate to citizens.

It’s also right for us to call out developments with the accessibility of services. GOV.UK Design System are reviewing all components and patterns in line with the new WCAG 2.2 guidelines, updating these now so that government users will have less to do when the new guidelines are published later in the year. You can find out more about WCAG 2.2 on our YouTube channel.

Product development is also important for our emerging products, like GOV.UK Forms, which is in private beta. We’ve already made some great strides in transforming services, working in partnership with teams like the Redundancy Payments Service in the Insolvency Service. 

We’re currently testing GOV.UK Forms with a limited number of other government teams to help us understand how the product is working and what issues we need to address before we make it more widely available. DWP’s Access to Work Plus referral is one of the services now using GOV.UK Forms and provides support for people with high in-work support needs. 

There’s more to come from all of our teams really soon and you can read about our strategy for GOV.UK Notify and how we’re understanding more about users of the GOV.UK Prototype Kit in our recent blog posts. 

The future of our common platforms

Ultimately, we want to be the first choice for building government digital services and it’s my vision for the future that any team in government, regardless of their technical skills, can deliver usable, accessible digital services that integrate seamlessly with GOV.UK, so our new mission is to integrate our products to enable that to happen.

You’ll be hearing more about it from us soon, so please stay tuned for our next update.

I wanted to update you on an aspect of our work in GOV.UK One Login that we haven’t really talked about before, our drive to make it one of the most sustainable government digital services in operation. 

As part of our work to design the solution, we started to take a real interest in how different technical and style choices have a direct impact on carbon emissions. And when we really started to look at it in detail we realised how much work has been done in this space and yet how hard it still was to get a definitive ‘how to’ guide from anywhere if you want to start to calculate the carbon footprint of your digital service. 

My deepest thanks to colleagues in Defra and our partnering organisations who have invested time (and patience) with us, helping us to understand the issues, the tools and principles we need to put into practice to be able to both measure a footprint but also to evaluate the effect of any future changes we might implement as a result. 

How we started our green journey

OK, some technical bits, for those of you that already know this, please bear with me. Calculating your emissions or your sustainability is as much art as science and in order to make any tangible progress you have to draw a ring around the things you’re trying to measure and influence. 

For us this meant focussing on the greening of digital services not greening by digital services. Sounds similar right? They’re actually quite different. We've decided to focus on how we can make the digital tech we’re building as sustainable as possible, rather than trying to calculate the emissions saved by people being able to complete things online rather than perhaps taking a bus to complete a face to face interaction. 

We’ve defined our ‘digital service’ as a combination of data centres and hosting, networks and end user devices, both those used by us to build and run the service but also those used by citizens when they access it. 

Then we set about trying to measure our emissions and started to run experiments in areas to see whether changes to a design, or implementation can reduce our carbon emissions. 

It’s taken us nearly four months to get a baseline! We built an app in less time than that! Which goes to show how hard it’s been to get data, and tooling and figure out how to come up with numbers that are robust. We think our numbers are probably under estimates and we’ll continue to refine them. 

Understanding our digital emissions 

We think One Login’s annual digital emissions are currently roughly ~200 tonnes of CO2e but this will grow as the user base grows, likely from hundreds to thousands of tonnes of CO2e. It is not easy to compare One Login’s carbon emissions to other web services as they are driven by a number of factors. However there are a number of other data points that we can use to compare One Login to other services.

Two of these data points are website page weight and mobile app size that indicate the amount of data transferred when using One Login (on the web or the app). The amount of data transfer impacts the amount of energy used to interact with the service, therefore impacting the overall carbon emissions attributed to citizens for using the service. Although it’s not as simple as saying lower data transfer leads to lower emissions, it is a useful data point to estimate the emissions of web services and to compare One Login to others.  

One Login has relatively low page weights and app sizes as the comparison to other services below shows. Of course One Login has a different use case to these sites, however this still illustrates that we think we aren’t doing too badly.

Page weight gathered through developer tools.

Based on app sizes from Apple App Store.

So now we have a baseline, we are starting to build out a prioritised roadmap of activities to reduce our carbon emissions and put in place better tools to monitor emissions on an ongoing basis. Some examples of activities we are considering to reduce One Login’s emissions include: hosting assets (e.g. images and fonts) on shared domain so the user only has to download the assets to their device once, ensuring all our computing workloads scale dynamically, removing some unused cloud assets and implementing better data classification and use policies to minimise unnecessary data storage.  

Next we’re rolling onto our neighbours in GOV.UK to run a similar exercise on GOV.UK itself. So exciting! Let’s hope we can shave a couple of months off our previous measurement time. 

Future proofing green IT

As the SRO for GOV.UK One Login I’m really passionate about understanding how we can make sustainability one of the factors all government digital teams consider when they’re making design or technical choices. In GOV.UK One Login we have the glimmer of an idea that in order to do this teams will need a set of simple tools and processes they can apply and as we collaborate with GOV.UK teams on calculating their footprint we’ll be adding to our lessons learned around how to measure things effectively and efficiently. 

I’m hoping we’ll have much more to share on our journey into green digital over the coming months and years but for now… if you’re out there and you’re reading this and you’ve done work in this area and have tips, tools, how to guides or ideas about how to help please do get in touch with the Green IT Team so we can share ideas and move forwards together. 

Natalie Jones 

Core to GDS's mission for GOV.UK One Login is our commitment to make sure as many people as possible can easily prove their identity to access government services. 

We are therefore excited to announce that we are now working with the Post Office to run an in-person identity check in their branches, for people who need additional support to use GOV.UK One Login.

The in-person service has been created to assist people who want to use GOV.UK One Login, but are unable to use the app or browser journey for identity verification. We’ve designed this route specifically for users without smartphones, or those who have a low level of confidence in using a digital platform to input their document details. Offering an in-person service allows for a wider range of people to access vital public services online. Once having proved their identity people will, in the future, be able to reuse this to access services across government.

The service is currently in private beta which means that together with the Post Office we are testing the route and making improvements. Once these changes have been made and tested, the service will be fully available to the public. 

How this will work

A user accessing the in-person service to help them prove their identity will go through the following steps:

1. Provide basic details to create a GOV.UK One Login account online.
2. If they need to prove their identity for the service they wish to access and don’t have a smartphone to access the app or need support to utilise the digital user journey, they can prove their identity using the in-person service.
3. They can choose a Post Office branch local to them and visit it, taking along a photo ID document, which could be:
   a) A UK passport
   b) A non-UK passport (with or without a chip)
   c) A UK or EU driving licence
   d) An EEA national identity card
   e) A biometric residence permit
4. Have their ID document scanned by the postmaster, and a photograph taken.
5. Following processing with the Post Office's document checking partner, Yoti, the user will be notified an outcome is available for them via email, and that they can now access the service they began their journey with.
   

We’re aware this process does begin and end online, which may not work for everyone. However, for those who lack confidence or don’t have a smartphone, this in-person option may offer all the help they need to prove their identity with GOV.UK One Login. 

The in-person identity checking also allows people to make use of a wider range of identity documents, beyond driving licences, passports and biometric residence permits. This grows the number of people who can prove their identity using GOV.UK One Login. 

The GDS and Post Office partnership

There are many benefits to partnering with the Post Office on this service. The Post Office is a well known institution and many people in the UK use their services regularly. They have a selection of branches offering in-person verification already in place for other government and non-government services. Most people in the UK are within 10 miles of a branch that is offering this new in-person identity check for GOV.UK One Login.

Postmasters are trusted and often well-known figures in their local community. They can support those individuals who are unable to use the app or browser journey for verification through the in-person service.

Once initiated online using the above steps, during the visit to the Post Office staff will be happy to help with the identity verification process. 

What's coming next

We are continuing to build out our offering so as many people as possible can access our government services using GOV.UK One Login. Which is why we are currently procuring a service where people can get support over the phone or by email when using the GOV.UK One Login service. 

GOV.UK Pay is the payments platform for the UK public sector. Our vision is to:

• make it easy for individuals and businesses to pay the public sector
• make it cheap and easy for the public sector to take and manage payments

September 2023 marks 7 years since we took our first live payment. Reaching such a milestone birthday gave us a great opportunity to reflect on how far we’ve come and decide on what’s next for Pay. 

Here's what we're doing to deliver our vision.

Our overall aim

Our aim is to support more central government and local authority teams to take payments for their services digitally. This allows them to automate their financial processes, reduces burden on their teams, and saves government time and money. This aim closely aligns with GDS’s strategy.

The 6 things we want to do

We’ve defined 6 strategic objectives as part of Pay’s product strategy. These objectives describe what we need to do for Pay over the next few years and why it’s important.

1. Diversify payment types

What: We want to expand from card payments into other inbound payment types

Why: Ten years ago, mobile wallets (like Apple Pay) did not exist, but now our users expect them as standard. This rate of change in the fintech industry is likely to increase and we’ll see our users’ expectations change too. We want to offer paying users with the most convenient ways to pay now and in the future

How: 

In June, we released recurring card payments to enable our users to make payments in instalments without having to manually input their card details every time.

We already offer Apple Pay and Google Pay for our central government services. This month, we’ll be releasing the same mobile wallet payment types to our local authority services too.

Later this year, we’ll be investigating how Pay might offer open banking, so that users can pay for public services via their banking app.

2. Enhance paying users' experience

What: We want to make Pay more usable and accessible for all our paying users. 

Why: Paying for public sector services should be hassle free. It's important that the widest range of users are able to complete their payments as quickly as possible.

How: 

We’re making updates to the payment journey, so that the content presented to paying users is clearer and easier to understand.

3. Improve public sector users' experience of managing card payments

What: We want to improve the Pay admin tool and documentation for our public sector users

Why: So that public sector users can manage transactions and refunds quickly and focus their energy on delivering great services

How: 

In June, we released webhooks so that services can receive automatic updates after payment events have taken place.

We’re reviewing and updating content in our tech docs to make it easier to use and understand.

4. Run a resilient, scalable, secure and cost efficient service

What: We want to continue to provide a robust, reliable and secure payments platform for all our users

Why: We need to keep all sensitive information secure in line with legislation. We want Pay to continue to perform successfully as our number of users and services grows

How: 

We’ll continue to provide responsive 24/7 support and improve our support processes.

We’ll continue to maintain our Payment Card Industry compliance.

5. Make it easier for the Pay team to improve the product

What: We want to create the right conditions so that our team can do their jobs with as little friction as possible

Why: Reducing pain points for our software engineers will help our team to work more efficiently. This should shorten product release cycles, enabling our team to add value for our users more quickly.

How: 

We’re improving the visibility of logs and metrics for our whole team, so everyone has access to the data they need to make decisions.

We’ve built up some technical debt and we want to address it. Over the next 12 months, we’ll be investing time in improving the quality of our code by breaking down known pieces of work and feeding them into our team’s backlogs.

6. Improve our knowledge of procurement and suppliers

What: We want to build up our team’s knowledge about procurement processes in government, and learn about payments suppliers in the fintech market 

Why: So that we can make sure we work with payment suppliers that offer the best value for money for our services 

How: 

We’ll be growing our team’s commercial capability through training and partnering with commercial experts across government.

What we're not doing

As well as describing what’s in our strategy, it’s equally important to describe what’s not in Pay’s scope and why. 

The main areas Pay will not be exploring are outbound payments, Direct Debit and PayPal.

Outbound payments

Pay provides inbound payments, enabling businesses and people to pay for UK public sector services. Outbound payments allow government and local authorities to make payments to businesses and people. Pay will not be offering outbound payment types as the approach required is vastly different from inbound payments and the public sector has other existing mechanisms to manage outbound payments.

Direct Debit

Direct Debit was not originally designed with digital payments in mind. This makes it difficult for Pay to integrate with and offer Direct Debit to users. Instead, we’re prioritising other types of recurring payment (like recurring card payments) so we can still offer ways for our users to make repeat payments while limiting our technical overheads.

PayPal

We’re prioritising the mobile wallet payment types that have the lowest fees for our services, while still meeting the needs of our paying users (Apple Pay and Google Pay wallets). PayPal is not a mobile wallet payment that we plan to offer in the immediate future.

What now?

We’re using the 6 strategic objectives in our product strategy to drive decisions about our product roadmap and to prioritise our backlogs. We’ll be working towards these objectives over the next few years. 

Here at GDS, we’re making it easier for departments to build better digital services. GOV.UK Forms is an online form builder which can be used to make accessible and easy to use digital forms for GOV.UK. It saves time for departments that are processing form submissions, and time for users that are filling in forms.

In just a matter of minutes, government teams can replace PDF and other document-based forms with digital forms which all users can access and are legally compliant with the Public Sector Bodies Accessibility Regulations 2018. 

Best of all, there’s no need for any technical knowledge and it’s completely free!

GOV.UK Forms is currently in private beta, where we’re testing the product out with a small number of teams that own forms. This is to help us steadily understand how our product is working and what issues we need to address. After that, we’ll move into public beta, where we’ll open up GOV.UK Forms for any central government department to use freely.

We wanted to share what we’ve been up to since our last update, and what we’re doing to open up private beta for a wider pool of users - a phase that we’re calling ‘Early Access’. This will allow departments with forms that only require current features to start building them, and also allow any government user to try out our product. It will also help us to ramp up our overall user numbers and test the stability of our platform before public beta.

What we’ve done

In October 2022, the Insolvency Service was the first government organisation to publish a form with us. This was a relatively simple form as we only supported quite basic features at the time. The vast majority of document-based forms require more complex features to turn these into digital forms, including multiple options, declaration statements and more.

Routing to skip questions that aren’t relevant

One of these features is routing, or skippable questions. If you think of any form that you might have to fill out, for example at the GP or for an application, the chances are that it will say ‘If you answer ‘No’, skip to question 13’ or similar. This is a really core requirement for form building, and we needed to find a way to build this so that people who aren’t digital professionals could understand how to set up this kind of logic for their form.

We created a new journey for form builders to add this to questions that ask for one answer to be selected from a list of options. They can specify which answer should route people to a future question and which question they should go to. The person filling out the form will then skip questions they don’t need to answer - saving them time and saving processing time too. You can see how this appears for form builders in the screenshot below:

A screenshot of a page informing a form creator that if a user answers ‘Yes’ to a question, they will be taken to a further question.

We know this is not going to cover all routing needs though - in the future, we’d like to look at building branching (two separate sets of questions depending on the answer), and later down the line we might be able to expand our routing so you can add routes to more than one answer, and also routes based on a combination of answers.

Making a draft of a live form

At the start of 2023, if one of our users created a form, made it ‘live’ (meaning it can now be put on GOV.UK), and then later on wanted to edit their live form, any change made would be updated on the live form immediately. This would cause issues as form builders may be making multiple changes, or change their mind about what they want to edit. And each time this would happen, people that are filling in the form would be at risk of losing their progress.

So what we did is we made live forms uneditable - instead, if you want to make changes to a live form, you would create a new draft copy of that form. Then you could make all the edits you want, and only make that new version live when you’re happy with all the changes. This new version would automatically replace the existing live form - meaning this change only happens once, and affects a much smaller number of people filling in the form.

Detailed guidance

Our form builder already allows users to add hint text to a question, such as ‘Enter your name as it appears in your passport’. But sometimes on a form there are questions that need a bit more information for someone to answer - for example, specific guidance that they may need to refer to, or industry codes that need to be defined so that the right code is entered.

To do this, in September we released a feature called ‘detailed guidance’ that will allow this more detailed information to be provided to the person filling in the form. Here’s what it will look like for people filling out an example form:

A screenshot of a question that provides more information about adjustments for interviews that can be offered, before asking the user whether they need adjustments.

What we still need to do

User management and self-service accounts

Right now we use GOV.UK Signon to allow users to sign into our product and use it. However, each time that a new user wants access, our team has to set that account up. We also can’t give people custom permission levels based on what we want them to be allowed to do (and not do). Signon is currently making improvements to make it more self-serviceable, but this wouldn’t have been ready for us to start expanding.

So last year we decided that we needed to move off of Signon and use Auth0 (also used by Ministry of Justice’s MOJ Forms) for authenticating users, and bring the permission controls into the GOV.UK Forms product itself. We also wanted to create a system so that users could create their own trial accounts without our involvement (self-service), try out the product, and then be easily upgraded to the next permission level in order to make their forms live.

Doing this work is one of our key challenges to adding many more users onto the platform, and we’ve designed an easy flow to allow quick access to the product, whilst also keeping enough control of who can make live forms whilst we’re still in private beta. We also want to ensure we’re following good governance practices and ensure Memorandum of Understandings (MoUs) are agreed before somebody from a department is able to make a form live.

Other features

Before we kick off Early Access, there are some other features that we’re working on to implement, including:

• an analytics page for each form that shows basic analytics such as submission numbers and completion rates
• email confirmations of submission delivery, so that form completers have a record that their submission was received by the department
• an updated product page with a more detailed public roadmap, guidance on the product, and an easier way to send the team support requests

Launch of Early Access

Once we’ve achieved all these things, GOV.UK Forms will be ready for launching into our ‘Early Access’ stage within private beta, with a public beta launch planned for the first half of 2024.

We are planning to start our Early Access period in November 2023, at which point central government departments can start trying out GOV.UK Forms to see what they can make - and if they meet the criteria, we’ll be providing access to make those forms live. This will open up in public beta when departments can provide their own editor access to form creators. As with any agile development process, timescales can shift depending on what we find out along the way, and priorities.

Before we move to public beta, we have some much-requested features that we’ll be working on, including:

• uploading file attachments
• saving form completion progress and returning to where you left off
• paying for services within the form

We’re not currently supporting organisations outside of central government, such as local councils or NHS and Police, but we’re hoping that we can see this scope expand later in 2024 and beyond.

We launched GOV.UK Forms’ private beta back in April 2022 and we’re now ready to move into our next phase - ‘Early Access’. 

So far, we’ve been working closely with 11 private beta partners and over 60% have already published forms. We’ve published 17 forms and we’ve got a pretty healthy pipeline of more forms being worked on. Our remaining partners are all getting closer to publishing their first forms which is hugely encouraging too. This means we’ve helped to solve some common problems experienced by lots of form owners and form processors. 

Quotes from partners

This is happening at ridiculous speed, it’s a fantastic opportunity. I’d still be struggling to get a PDF through!

- Dan M, Animal and Plant Health Authority

And we feel that now is a good time to release the value we’ve created and allow more teams to use the platform.

What does ‘Early Access’ mean?

During this phase, anyone who works in central government and has a gov.uk email address can try out using GOV.UK Forms by creating a trial account. They can learn about the platform and decide if this is going to be useful for them to build forms now. Trial accounts won’t allow users to create live forms but we’ll be considering requests to upgrade accounts, based on our criteria below. 

This means giving access to our new form builder to more colleagues working in government. We’re doing this because we think we’ve got a great platform and lots of teams will benefit from using GOV.UK Forms, even at this early stage.

We want to do this in a managed way, one that helps us to test out new and existing features in a live environment and to scale up confidently. 

Users can ask to upgrade their trial account and gain full access to the platform if:

• they’re confident they can create forms using the features available at the moment
• their organisation is ok with form responses being emailed individually to a team email address (we’re planning to support other ways of receiving form submissions but they’re not available yet)
• forms created using GOV.UK Forms will only be published on the GOV.UK website - not on other websites or for other uses such as internal forms or surveys
• the forms they make get no more than 10,000 submissions a year. We’ll be looking to expand this in the public beta phase of the roll out.
• they’re happy give us feedback and to take part in user research

It’s important to state that even with an upgraded account, users will not be able to publish any forms on GOV.UK themselves. To make a form findable on the website, form creators will need to work with their departmental publishing team to add a link and make any other content changes required. 

We’re still in private beta and have lots to learn. Early access will help us to do this in a scalable way, releasing value to our users and tying this to our roadmap. We’ll be inviting more and more users to access the platform as we develop more features. 

What GOV.UK Forms currently offers

We’re building a brand new platform which means we have a growing, but still limited, set of features right now. You can ask questions for names and addresses, or for longer text answers, as well as asking for dates and answers which require a number. You can also create questions which ask a person filling in a form to select options from a list. You can even allow a user to skip future questions based on a previous answer. 

Another really great feature we’ve recently added allows you to include detailed guidance, where typical hint text does not provide enough context or information to help people answer a question. It allows you to add guidance, as well as formatting, like bullet points, subheadings and links.  

You can choose how to format certain questions such as, whether to ask for a name in one box or split this into first name and surname. We support autofill, allowing a user’s browser to automatically suggest information they’ve provided on other websites, for example for names, addresses and dates of birth. 

We’ve just released a new feature to send confirmation emails to people who have filled out a form using an integration with GOV.UK Notify. Another cool feature allows you to see some simple form metrics, including weekly completion rates and the number of form submissions. This helps form owners to manage and to iterate forms, making them better and easier to fill out.

What happens next?

Over the next few months we’ll be developing our functionality in order to move into public beta where anyone working in central government will be able to use the product in a self-service way. We are hoping to move into public beta in the spring. 

Before we move into public beta, we will integrate with GOV.UK Pay so people filling out forms can make payments using a payment link. We will also add functionality that allows users to include additional answers to the same question, a common pattern used on around 35% of government forms. For example, if a question asks for an address history for the past 5 years, users will be able to add multiple addresses.

We will iterate our sign-up journey and account management process to make it fully self-service. We will also complete some technical work to make sure our platform can scale for the increased demand. 

Why we think this is a big deal and how you can get involved

Our aim is to design an easy to use, self-service tool which has an accessible interface, built with common GOV.UK Design System components. In this way, we think the new platform is a crucial tool to help us deliver better government services. 

It’s our vision that all forms on GOV.UK will be accessible, easy to use and quick to process. This isn’t an easy goal - there are almost 10,000 document-based forms, most of which need to be transformed. GOV.UK Forms is making this vision real! Our product team has done a stellar job of creating research-backed solutions to real user problems, releasing value for the civil service and UK society. This is transformation and this team is on fire!

Amanda Dahl, Deputy Director, Digital Service Platforms, GDS

We’re already working closely with a number of teams to plan how they will improve their forms and help shape our product, both now and as we release more features in the coming months. If you work in central government, we’d love to hear from you, please do get in touch.

You can read more about the features we already have, and what we're working on next, on our product site.

Early Access is our big idea for scaling up! We hope you’re interested in working with us, learning more and helping to shape the development of future form features. We can only do this with your help, so please share with colleagues and any civil servants who may be interested in our form builder too.  

We’ll also be hosting a webinar to discuss the new form builder and what’s coming up in January 2024. Details will be on our product site.

We recently reduced CSS size and improved performance across GOV.UK pages by moving away from bundling all CSS into a single stylesheet served to all pages to serving smaller stylesheets containing only the necessary styles for each page. There have been reductions of up to 40% in CSS size on some pages, accompanied by incremental improvements in timing metrics across many pages. These improvements could lead to a slightly faster experience for users accessing GOV.UK through older or low end devices or for people on slower connections.

Previously

GOV.UK is made up of many different applications. Previously, each application served a single stylesheet, a set of CSS rules that define the layout and appearance of a page, made up of many different types of styles bundled together. Styles can come from our component library (accordion, button, notice and more) or they may apply to specific pages within the application. Styles were served to all application pages. Bundles could sometimes include as many as 30 different style imports.

In addition, a separate stylesheet which includes commonly used styles like layout or typography styles is served from an application called static.

Bundling is a common HTTP/1 performance optimization technique where resources cannot be loaded in parallel and too many sequential network requests can take a long time to download, so it makes sense to bundle styles together to reduce network requests.

Problems

Concatenating multiple stylesheet assets into one bundle has its downsides.

Firstly, users might have downloaded CSS for unvisited pages. For example, the homepage previously loaded CSS for 24 components yet, aside from commonly used styles (included in the static stylesheet), only required styles for 5 components. Therefore, a homepage visit required 17 kilobytes of unused CSS to be downloaded.

Also, the application stylesheet cache was easily invalidated for small changes, which meant the browser had to download the application stylesheet again. For example, consider a small style change to the contents list component styles in our component library to change bottom margin spacing from 20px to 30px.

This small change would require the entire application stylesheet (27.7 kB gzipped) to be redownloaded. This is an important consideration when our component library is frequently updated.

Lastly, bundling all CSS doesn’t take advantage of the multiplexing features of HTTP/2 where multiple assets can be requested at the same time making it possible to serve many smaller files. HTTP/2 was switched on for GOV.UK in 2020.

How we solved these problems

To solve these problems, we created a new asset helper in the govuk-publishing-components gem. The asset helper is used to create a list of stylesheets required for rendering a page, it includes helper methods to add required stylesheets to the list and the `render_component_stylesheets` method to output the list of stylesheets as stylesheet link tags in the <head> of the document.

The asset helper ensures that all stylesheets in the list are unique. For example, if several button components are rendered on a page, only one button stylesheet is loaded.

The asset helper runs in the background for all applications that use GOV.UK Publishing Components, but implementing the individual loading of stylesheets feature is optional.

Once we had the asset helper available, the next step was to implement the individual loading of stylesheets feature in our rendering applications. We decided to do this one application at a time, starting with the `frontend` rendering app (which renders the home page, the register to vote and tax your vehicle services and more). This approach allowed us to monitor the impact of the change, giving us time to fix any issues before implementing in the next rendering application.

We also created detailed documentation for setting up individual component CSS loading in a rendering application. This includes step-by-step guidance, links to example pull requests and known issues.

Performance gains

Using SpeedCurve to test various pages, across various devices and connection speeds, before and after the changes, we’ve seen a site-wide reduction in CSS size together with performance improvements when measured against key performance indicators.

CSS size has reduced by 40% (dropping from 42 kB to 25 kB) on the GOV.UK home page along with improvements to timing metrics including Start Render (the time from the start of the initial navigation until the first non-white content is painted to the browser display), Largest Contentful Paint (​​a Core Web Vital metric which measures loading performance) and Last Painted Hero (a metric that shows you when the last critical content is painted).

On an emulated Mobile 3G device:

• Start Render - 4% improvement from 2.3s to 2.2s
• Largest Contentful Paint - 7% improvement from 2.53s to 2.35s
• Last Painted Hero - 7% change from 3s to 2.8s

Likewise, for COVID-19 pages, there’s been a 27% reduction in CSS (from 44 kB to 32 kB) and similar performance improvements.

• Start Render - 4% improvement from 2.3s to 2.2s
• Largest Contentful Paint - 4% improvement from 2.41s to 2.32s
• Last Painted Hero - 6% improvement from 3s to 2.81s

Such improvements will mostly be imperceptible to users on high end devices or faster connections but could make a difference for users on older or low end devices (GOV.UK analytics show a sizeable number of users using Galaxy A12, A13 and earlier models) or for people on slower connections - for some pages the improvements bring load times in under 3 seconds (statistics indicate that 40% of visitors will leave a website if it takes longer than three seconds to load).

Whilst these size reductions are encouraging, arguably the main advantage is in caching improvements. In the earlier example, a small spacing update to one of the application component styles forced the browser to redownload the entire application stylesheet. Now the same change means the client only downloads the updated component CSS (1.1 K gzipped). All other stylesheets are loaded from cache.

Other improvements

Whilst implementing the asset helper we identified other optimisation opportunities including finding and removing redundant code. We removed 32 stylesheets across 7 applications and more than 600 lines of CSS by removing unused CSS - styles that are no longer used anywhere - and also removing redundant code - custom helpers or component styles that are now available in the component library or design system.

Negatives

Loading lots of individual stylesheets doesn’t always lead to performance improvements. For example, on detailed guide pages e.g. rates and thresholds for employers 2023 to 2024 which use lots of different components, users can download approximately 16 different stylesheets. Whilst CSS size is still lower than previously, performance indicators are adversely affected with time metrics slightly worse than previously.

Also, compressing files with small amounts of text doesn’t always produce smaller file sizes (or leads to hardly any size reduction) since there are fewer instances of repetition to be identified. It’s even possible to see larger file sizes in some cases.

However, the performance improvements across the site outweigh the downsides seen in these particular edge cases.

What’s next

For some pages, where many individual stylesheets are served, performance metrics are slightly worse. We might look to group together some component styles, commonly served to the same pages, into smaller bundles to achieve better compression savings and reduce their size.

Also, we’re investigating loading JavaScript scripts individually, though this is more complex and will require changes to our component library to make use of ES6 modules.

Finally, future enhancements might include preloading assets based on predicted user journeys or lazy loading ‘below the fold’ component stylesheets such as footer and feedback component CSS; both positioned at the bottom of each page.

This is the first time I have written a guest blog for Government Digital Service (GDS), so I’m really pleased to be able to tell you more about my team and the work we have done with GDS and the GOV.UK One Login programme. You might be wondering why my team is called the FAT Badgers… We were originally called the Honey Badgers (due to being fearless and highly intelligent) but then we redesigned Find Apprenticeship Training (FAT) so rebadged ourselves the FAT Badgers.

In October 2022, our team in the Department for Education began the work to switch over to GOV.UK One Login for a range of our services. One Login will deliver a single consistent way for users to sign in and prove their identity to all government services, so being an early adopter was an exciting yet slightly daunting task! We knew we were in safe hands as services such as Disclosure and Barring Service basic check (DBS), Apply for a vehicle operator licence (DVSA), and Register to be a Social Worker service (SWE) had already joined GOV.UK One Login.

Why we became early adopters

The Department for Education has a number of services which use different sign in methods. For example, the same people who use our ‘Find an apprenticeship service may use ‘Apprenticeship service account' however they have different portals to sign in. 

Therefore, being a part of GOV.UK One Login increases our usability. We recognised that by adopting GOV.UK One Login our users would have a simpler process to register and return to our service. Our users include people like sole traders and small employers who really need their interactions with the government to be simple and fast, given they could be multitasking and how busy they can be.

Other important benefits of onboarding to GOV.UK One Login included:

• increased account security through two-factor authentication (reducing fraud and misuse of our service)
• secure two-factor authentication options
• less duplication across government in terms of log in capabilities 

How we started the onboarding process

The delivery team began the complex task of transitioning around a dozen separate sub-services to a new method of user authentication. The onboarding process involved working with the team in GDS to sign a memorandum of understanding, reviewing user journeys, planning our communications, and testing the service. After months of hard development work, our services migrated over.

Our top tips of switching to GOV.UK One Login 

Work in the open

Working in the open allows both teams to work at pace. We were regularly working to tight deadlines to ensure that there were solutions in place for all sign in processes. To improve ways of working we scheduled weekly meetings and set up shared Slack channels, letting us discuss issues together and unblock any stuck development tickets. 

Working in an open way also helps teams that are planning their future switch to GOV.UK One Login. Our work was produced on public repositories so other teams can see how we integrated GOV.UK One Login and use our code, available on github, as a starting point.

This is particularly useful as we were one of the first GOV.UK services that use Microsoft .NET to implement GOV.UK One Login. We initially had some development difficulties as a lot of our services were using outdated Microsoft framework versions which did not support the updated security packages that were required for the integration to work. Now GDS can use our work as an example to support other teams which is a win-win for everyone.

User-centred design work is required

Though switching to GOV.UK One Login clearly involves development and resources for internal testing, this kind of project is an example of how each role in a service team is important. 

For instance, our user-centred design team had to change the service to follow the GOV.UK One Login design recommendations. The design work included:

• changing in-service settings screens to link to the GOV.UK One Login account management page
• re-writing GOV.UK Notify emails that mentioned previous ways of signing in
• requesting content changes to GOV.UK content so users understood they now needed a GOV.UK One Login to use the service

Throughout this experience, we've discovered that unexpected tasks can pop up now and then. It's crucial to leave some room for flexibility in your planning. We believe this is a valuable lesson that other service teams should keep in mind when making a switch.

Think about communications early

We worked closely with our colleagues in Service Engagement to ensure that users knew the switch was coming and understood what actions they had to take. We did this early: our first email was sent out to users over 4 months before the actual migration date. Service Engagement colleagues communicated regularly with internal employer facing teams, giving them key information to be able to support employers they work with.

As we were making a large change to the service, we also made sure we kept the National Apprenticeship Helpdesk in the loop. We provided guidance and documentation, so they could offer support to users.

Switching to GOV.UK One Login is worth it

Changing how users sign into a service could have been a daunting task, but by switching to GOV.UK One Login, we’ve implemented a future proof sign-in that improves our user journey. We found through careful planning of the migration, and regular communication with users, there was minimal impact on our support teams. 

We couldn’t have achieved this successful migration without support from colleagues across GDS and DfE.

We’re making it easier for departments to build digital services here at GDS and we’ve built a tool to make it easy to create online forms for GOV.UK without any specialist digital skills. GOV.UK Forms makes creating an online, accessible form as easy as creating a document.

We’ve been testing forms created using our tool with the public, and found they much preferred online forms to PDFs and other document-based versions. We learned some interesting things from users about why they find online forms easier to use.

Usability testing

We’ve been doing lots of testing with the immediate users of GOV.UK Forms - civil servants who create forms. But we wanted to check in with the people who end up using those forms - the members of the public who need to submit information to government.

We ran some user research sessions with members of the public to understand their experiences of using government forms. We asked them to complete a range of simple forms and share their thinking as they did so. Some of these were document-based forms in PDF or Word format and some were online forms created using GOV.UK Forms.

Clear, trusted design

There was strong support for the online forms across the sample group we tested, with everyone stating they preferred the forms created using GOV.UK Forms. GOV.UK Forms uses the GOV.UK Design System, and our participants commented that they recognised the visual style as being a government site and something that they could trust. 

I do like that familiarity every time I come to a form, and it’s something I’ve begun to kind of notice when I'm on the government website.

Forms using the GOV.UK Design System all look the same, increasing their sense of trust and familiarity.

Checking your answers

Online forms created with GOV.UK Forms all end with a simple 'Check your answers’ page. Our participants said they liked this page showing them their information and giving them the chance to go back and change anything they weren't happy with. 

Once I’ve finished, it also just means it's a lot easier to check as well, rather than having to look down through every section. You can be moving your eyes around quite a large area, whereas with this it's all nice and straight down the page.

Automatically completing personal information

When completing online forms, people were able to use ‘autofill’ to complete common information, such as names and addresses, that was saved in their browser.

One person gave more complete information because they were able to use previously saved information from the browser using autofill. 

And yeah, obviously, you know, I know it's optional, but because I could just click, I'm quite happy to do it because it's not taken me a lot of time.

It's not always perfect, and one person found a previous form had caused them to save some incorrect information in their autofill, but it does usually save people time, especially if they use assistive technology.

One thing per page

One of the most distinctive aspects of online forms on GOV.UK is the 'one thing per page' principle. The online forms we tested had one question per page, and people commented that this was welcome and made the form easier to complete.

They said that this principle allows them to focus on one thing at a time and not worry about navigating through the form. They simply answer the question that’s in front of them. 

You're not sort of presented with a long form straight away. It takes you through each section at a certain page, which I find helpful.

Being able to focus on one question at a time was also commented on as a positive by people with accessibility needs. We tested with people using a range of assistive technologies and this simple, focused layout was something many found helpful.

I'm not kind of bombarded or overwhelmed with loads of information at first looking at it. It's not making me look at it and think, you know, where do I start?

We also observed some people giving more accurate and complete information using the online forms.

Making forms quicker and easier

We've been building GOV.UK Forms to make it easier for civil servants to create interactive forms on GOV.UK. It's been good to confirm that these types of forms are also easier for members of the public to complete. 

Making forms easier and more accessible means people are more likely to submit accurate and complete information the first time around. This cuts down on the time both they and civil servants have to spend following up submissions.

It's quicker than the [PDF form] - and also, if I found out that I didn't put something in, it would alert me to it.

There will always be a place for document based forms where information needs to be provided in a situation where a hard copy works best, and to make sure people can access forms completely offline. However, using GOV.UK Forms can help government run services more efficiently, and can make it quicker and easier for people to access them. 

[The online form] says it's been submitted without much of a hoo-ha, which is absolutely wonderful.

GOV.UK Notify is hosted on the GOV.UK Platform as a Service (PaaS). The PaaS is being retired, so we are migrating all of our infrastructure into our own Amazon Web Services (AWS) account. This blog post explains how we migrated our PostgreSQL database with minimal downtime.

Migrating our database

The PaaS provides a database for us and we use it to store all of our data - from data about each notification we send to the content of the hundreds of thousands of templates service teams use to send those notifications. This is an AWS RDS PostgreSQL database and it lives in the PaaS’ AWS account. Our apps that run in the PaaS talk to this database. We are going to call this database our ‘source database’.

We needed to set up a new database in our own AWS account, and get all of our apps to talk to the new database. We are going to call this new database our ‘target database’.

Creating a new PostgreSQL database in our own AWS account is not too difficult. The hard part is transferring all of our data and getting our apps to use this new database, whilst incurring minimal downtime.

A bit more about our source database

Our source database is about 400GB in size. It has about 1.3 billion rows, 85 tables, 185 indexes and 120 foreign keys. It is PostgreSQL version 11.

On a usual weekday, we do somewhere in the region of 1,000 inserts or updates per second (sometimes much lower, sometimes much higher), plus a similar number of reads. 

GOV.UK Notify sends millions of important and timely notifications each day, from flood alerts to updating users about their passport applications . Every notification we send requires talking to our database. Therefore it’s important that we minimise any downtime.

AWS Database Migration Service

The PaaS team offered us the ability to migrate databases using AWS Database Migration Service (DMS). 

DMS is responsible for transferring data from our source database to our target database. It can be run in either the source or target AWS account.

DMS works by:

1. Copying across all of the data, table by table, up to a specific point in time. This is known as the ‘full load’ task.
2. Entering replication mode, where it ensures that all new transactions on the source database are replayed onto the target database, so that the 2 databases are in sync.

We would then be responsible for getting our apps to stop talking to the source database and start talking to the target database.

Database migration process

The database migration process was completed in several stages.

Setting up the DMS instance

In our case, the DMS instance was created in the source AWS account. We chose the source account because the PaaS team had already set up instances of DMS in their account and so were able to do this quickly and easily.

The DMS instance also needed to be given PostgreSQL credentials to talk to both the source and target database. 

The DMS instance and the target database live in different virtual private clouds (VPCs). With the help of the PaaS team, we set up VPC peering so that traffic from the DMS instance in the PaaS’s VPC could be routed directly to our VPC without the traffic going over the public internet.

Setting up our target database

We created our target RDS instance in our own AWS account. PostgresSQL version 11 was about to become unsupported, so we took this opportunity to upgrade our PostgreSQL version by making our new database PostgreSQL 15.

We then took a dump of the database schema for our source database using `pg_dump`. This gave us a file with the SQL commands to recreate our database schema.

From our database schema, we took the declarations for our tables and applied these to our target database.

We didn’t apply our foreign keys at this point because DMS’ full load process doesn’t try to copy across the data in an order that matches your foreign key constraints.

We didn’t create our primary keys or indexes at this point because this would massively slow down our full load task. Each individual insert would take longer; it would need to update our indexes and this would add up to a significant amount of time when inserting billions of rows. It was much quicker to first copy all of our data across and then add the indexes afterwards.

Full load

Once we had a target database with the tables created, we then started the DMS full load task. This copies across all the data that existed when we pressed the ‘start full load’ button. It doesn’t copy across any new data or updates that come in after this point. It took about 6 hours for the full load task to finish.

After the full load task completed, we applied the remainder of our source database schema file which adds our indexes and key constraints. Adding these took about 3 hours.

Replication

Once our full load task completed, the data in our target database matched the data from the source database at the point when we started the full load task. But many new inserts, updates and deletions had happened on our source database since then. And many more changes would keep coming in too.

To copy these new changes across, we then started the DMS ongoing replication (also known as change data capture) task. This reads all the transactions from our source database transaction log that were created after the full load task began and sends them to our target database. This ensures that our target database is in sync with our source database with, at most, a small amount of lag.

It only took a couple of hours for the replication process to catch up. At that point, we monitored the latency in the DMS replication process to make sure it could handle the number of changes happening to the source database and continued to stay in sync.

We ran the DMS replication process for about 10 days in the background, keeping everything in sync whilst we awaited the time for our apps to stop talking to the source database and start talking to the target database. We had announced this time to our users in advance and so had a set time already for the migration of traffic.

Preparing to migrate traffic

Several months ago we planned how we would stop our apps talking to our source database and get them using our target database.This was the process we used:

1. Stop all traffic from our apps to our source database. At this point we would enter a period of downtime where Notify was unavailable.
2. Ensure our replication had caught up so that all updates to our source database had been reflected on our target database.
3. Allow our apps to start talking to our target database. This would end our downtime.

It was important not to have some of our apps talking to our source database and the rest talking to our target database at the same time. If this happened any changes on our target database would not be reflected on our source database which would mean users would get inconsistent data.

We wrote a Python script for this process so it could be explicit, easily repeatable and much quicker than being done manually.  The quicker it could be done, the less downtime for users of Notify. Our target was less than 5 minutes of downtime. We ended up using this script at least 40 times during our various tests and practices beforehand.

We picked a Saturday evening for the migration. This is because it is one of our quietest times without us having to be awake in the middle of the night when we won’t be as alert.

Stopping traffic to our source database

Our script would stop all traffic to our source database by calling `pg_terminate_backend` on all the connections from our apps. This took less than a second. We also changed the password for the PostgreSQL user used by our apps, meaning that if the apps attempted to reconnect to our source database they would get an authentication error. 

Checking replication had caught up

DMS inserts some useful tables into our target database on the status of the replication which are updated every minute. These tables allow us to see how much lag there is between our target database and the source database. Our migration script would check these tables to make sure our target database was entirely caught up.

To be extra safe, after our apps had stopped talking to our source database, our migration script would write a single record to our source database and then wait to see that it safely arrived in our target database. This gave us extra certainty that all changes had been replicated.

Making a smooth swap of traffic

For our apps to connect to our database, they need to know the location of the database and also a username and password for a relevant PostgreSQL user. These are provided to our apps in an environment variable of the following format: 

SQLALCHEMY_DATABASE_URI = postgresql://original-username:original-password@random-identifier.eu-west-1.rds.amazonaws.com:5432

If we want our apps to connect to a different database, we need to update the username, password and location in the URI and redeploy our apps so this change takes effect. Redeploying our apps takes about 5 minutes. If we redeployed our apps as part of our migration script then this would mean an extra 5 minutes of downtime. To minimise downtime we made two changes in advance of our migration so that we could use a quick Domain Name System (DNS) change instead of redeploying our apps.

The first change was to create a user on both our source and target database that had the same username and password. This means that we don’t need to change the username or password provided to the apps during the migration.

The second change was to create a DNS record in AWS Route53 for `database.notifications.service.gov.uk` with a 1 second TTL (time to live). It had two records with weightings:

• 100% of DNS results were weighted to the source database location 
• 0% of DNS results were weighted to the target database location

We set our URI used by our apps to use our new username and password, and to use the new domain name for the location of our database.

SQLALCHEMY_DATABASE_URI = postgresql://shared-username:shared-password@database.notifications.service.gov.uk:5432

Now, when we wanted to swap the database that our apps would be pointing at, our migration script just needed to update the DNS weighting in AWS to 100% of results being sent to the target database location and wait 1 second for the TTL to expire. Then, when our apps next try to query our database they will be querying our target database.

What happened on the day

When we gathered on the evening of Saturday 4 November, we had set up our target database, the full load process had run and new transactions were being copied across. We checked and only had a couple of seconds lag between our target database and the source database. 

We then successfully ran our migration script so that our apps would stop talking to our source database and start talking to our new target database. During the migration there was a short period of downtime, roughly 11 seconds. This was much less than our 5 minute target so we were very pleased and so were our users.

What we learnt

We chose to use DMS because it was well supported by the GOV.UK PaaS and we could also get support from AWS. If we were doing a PostgreSQL to PostgreSQL database migration in the future, we would invest more time in trying alternative tools such as pglogical. DMS potentially added more complexity, and an unfamiliar replication process than what we may have found with other tools. This backs up what AWS say themselves on PostgreSQL to PostgreSQL migrations. 

What’s next for GOV.UK Notify’s migration to AWS

Now we’ve migrated our database, our next step is to migrate our apps. Sneak peek - we are moving them to AWS Elastic Container Service (ECS). We will blog about how this goes in the coming months.

GOV.UK Pay has grown every year since launching in 2016. 2023 has been our biggest year yet with 364 public sector organisations currently using Pay to take payments for 960 different services, an increase of 21%.

The Pay customer satisfaction survey is one of the ways that we get feedback from Pay users, ensuring that we continue to deliver for the needs of diverse organisations and services that use Pay. 

This survey allows us to collect quantitative data from a large group of people. We ask a range of different questions to understand how our public sector users feel about the different features available on Pay, and where we could improve. 

In the questions, users are asked to rank the benefits of Pay and to score how satisfied they are with different features on a Likert-type scale. 

In 2022 the survey was sent to 1,000 Pay public sector users who had logged in to the Pay admin tool at least five times in the last 12 months. We decided to widen the pool of people in 2023,  sending to public sector users who had logged in at least once in the last 12 months;  a total of 3,444 Pay users. 

Findings

We saw an increase in satisfaction for the majority of categories in 2023 compared to 2022. 

Overall, 88% of respondents said they are satisfied with Pay,  a 2.7% increase from 2022. 

Satisfaction with Pay’s 24/7 support increased from 72% in 2022 to 75% in 2023. Users said that the 24/7 email support is prompt and comprehensive, with one user from His Majesty's Passport Office stating:

I have always had a quick response when I have contacted GOV.UK Pay, and my queries are always dealt with to everyone's satisfaction.

The satisfaction levels for reconciliation increased by 17% from 2022 to 2023. The main theme for the high levels of satisfaction is the ease of use for reconciliation “Easy to use, simple to reconcile”.

Some of the main benefits users were satisfied with included: 

• Pay is easy to use for our customers
• Pay is easy to use for my work
• Pay is secure

In the 2022 survey, 50% of people told Pay that they wanted a pay-by-bank feature where customers can pay directly from their bank account. In response, we started a discovery in 2023 into how we can offer pay-by-bank.

46% of people surveyed in 2023 told us that they wanted to see an increase in the number of paying users that complete their payment. We are making improvements to the design of the payment journey to address this and these changes should be released soon.

We also know from both the 2022 and 2023 surveys that Pay users want to see improvements in the reporting options available—in 2022 users told us that they wanted more detailed and auto-generated reports. “Ad hoc reporting is very good, but daily reports should be system generated, not user-generated”. We are exploring how we can do this. In 2023, 30% of users surveyed wanted to reduce the time it takes to reconcile payments.

Some of our users also told us that they are not aware of new and existing Pay features, so we will be engaging with them to  ensure we advertise Pay’s features more regularly.

Next steps

We will combine the findings from other user feedback sources such as support requests, user research and analytical data to prioritise improvements that we want to make on Pay.

Here at GDS we’re focused on making digital government simpler, clearer and faster for everyone. As part of the early access phase of GOV⁠.⁠UK Forms we wanted to put this into practice by demonstrating just how quickly and easily you can create an online form.

With a sold out online audience, we hosted a webinar showcasing the service, current and future features, and showing how GOV⁠.⁠UK Forms is already making a difference across government.

GOV⁠.⁠UK Forms replaces the dependence on PDFs and document-based forms. In just a matter of minutes you can create a new online form that’s quick to process and incorporates lots of GOV.UK Design System components which are focused on accessibility. This makes it even easier for people to complete government forms. Best of all, there’s no need for any technical knowledge to create a form and it’s completely free to use!

Live Demonstration

During the webinar, we conducted a live demonstration to showcase the ease and efficiency of GOV⁠.⁠UK Forms in action.  In under 20 minutes a form was created, which included basic routing and some detailed guidance for the person completing the form.

The purpose of the example form created in the webinar was to capture feedback from the audience. It included questions such as name, date of birth, webinar attended and previous experience of form builders. The questions used a variety of methods to submit data, from text to buttons. Another feature on show was the ability to autofill answers, where you have information such as name and address already stored on your device. 

Providing questions that are clear and easy to understand are vital to help people complete a form once published. During the demonstration we also showed how the form builder provides helpful information in hint text, and prompts the person creating a form to include contact details for anyone needing additional support. 

Technical Overview

While you don’t need any technical knowledge to use GOV⁠.⁠UK Forms we’re also exploring more advanced options for departments and teams with more digital expertise. 

The next presentation in the webinar provided a brief technical overview of the service and work towards future opportunities. For instance, currently form submissions are emailed to a chosen inbox from which they can be processed. This is fine in lots of cases and, as demo-ed, is really easy to setup.  But we know departments have more complex needs for receiving and processing form data so we are learning and looking at how we might send data differently to support automated form processing.

This presentation also outlined how GOV⁠.⁠UK Forms follows the 14 cloud security principles from the National Cyber Security Centre.

User Feedback

User feedback is critical to the development of GOV⁠.⁠UK Forms and this formed an integral part of the webinar. John Ploughman, Head of Content Design at the Driver and Vehicle Standards Agency, described his first hand experience of GOV⁠.⁠UK Forms, emphasising its role in saving time and simplifying processes for DVSA teams: 

It'll make life easier for your colleagues and for your users [...] all of this is saving teams at DVSA time.

John Ploughman Head of Content Design at the Driver and Vehicle Standards Agency

Next Steps

Looking ahead, we're enhancing GOV⁠.⁠UK Forms with new features and capabilities. In our final presentation we shared upcoming developments, including the integration of payment links powered by GOV⁠.⁠UK Pay, which is a seamless solution for accepting payments through online forms. Additionally, improvements to user access management processes and the introduction of multi-answer questions will further enhance the platform's functionality, empowering departments to utilise GOV⁠.⁠UK Forms in a more self-service manner. The audience also heard how GOV⁠.⁠UK Forms will be available to the wider public sector in a later phase, once we have a scalable product for central government as this is our core mandate at GDS.

The webinar closed with a chance for audience members to put questions to the GOV.UK Forms team. Examples included:  

• are forms hosted by the service itself? The forms are hosted on GOV.UK Forms. There's nothing to install. You access GOV⁠.⁠UK Forms through your web browser. 
• will there be the functionality to add attachments if required? This is something we have included in our forthcoming features.

The questions shared and feedback to the presentations showed just how much interest there is in GOV.UK Forms. While we’re currently in the early access phase of the platform's development, look out for further webinars as more features are released.

GOV.UK One Login is the new straightforward, secure way to access a range of government digital services, from requesting a DBS check to signing a mortgage deed. In 2021, the Disclosure and Barring Service (DBS) became the first department to partner with GOV.UK One Login.

It's been over a year since we integrated the 'Request a Basic DBS Check' with DBS, so to mark the occasion we sat down with the Basic Digital Service Team in DBS to learn how the programme has changed their service. We discussed how they found partnering with GDS, onboarding to One Login, and what they learnt along the way.

Working with GOV.UK One Login

DBS helps employers make safer recruitment decisions. One of their key services, 'Request a Basic DBS Check', allows an individual to get a copy of their criminal record check. This check will only show convictions that are not 'spent' - for example, some types of caution will disappear after 3 months.

This helps employers decide whether applicants are suitable for a role, so a service which is accessible, secure and easy to use is a necessity.

After the sunsetting of GOV.UK Verify, GOV.UK One Login approached DBS to be a co-designer of a new identity checking solution. The DBS Product Manager reflected on that initial decision to partner with GOV.UK One Login:

It was important to understand what the 'Request a Basic DBS check' service would need from GOV.UK One Login, and GDS need to design a product that would be flexible enough to work for multiple services. It was useful to set this context early on as it helped both of us design the respective parts of the user journey.

Allan Robinson - Product Manager, Basic Digital Service Team, DBS

As a private beta partner, DBS provided invaluable insights into their specific service requirements. This approach not only allowed DBS to tailor their service but also offered One Login a unique perspective. By understanding DBS's objectives, we restructured our onboarding process for departments, assigning dedicated engagement managers to ensure streamlined communication and a personalised experience for each team.

Laying the foundations

From the initial co-design process through to private beta, and public beta delivery to ongoing developments, there was a constant exchange of ideas and feedback between DBS and the GOV.UK One Login programme. This level of openness laid the foundation for our success, as it allowed us to continually iterate and improve the service together.

I think being able to understand the roadmap, to be able to prepare for that - there is transparency if things don't go as planned. We continue to keep in touch on Slack etc - it's exactly what you would want in terms of the relationship. It's a big programme, but you still get that direct communication.

Allan Robinson - Product Manager, Basic Digital Service Team, DBS

Key to the GDS and DBS success was building strong relationships. DBS and GDS had regular communication through monthly meetings, facilitated by engagement leads. DBS and GDS also attended user-centred design (UCD) meetings/sessions to catch up whenever there were new designs to share. There were also playbacks and UR sessions to gain insight to design and iterate the service.

This combination of engagement also ensured that any concerns were addressed promptly and that our shared objectives stayed clear.

Celebrating success

The integration of GOV.UK One Login into the 'Request a Basic DBS Check' service has significantly improved user experience and efficiency. One of the most notable benefits is the increase in successful online verifications, which has led to a surge in user satisfaction. By leveraging the robust capabilities of One Login, users are now able to complete their DBS checks quickly and with greater ease.

The efficiency gains from reusing data through One Login have been remarkable. The average time to submit a basic DBS check has been reduced by six minutes. This translates to an astounding total user time saving of 1,814 days in 2023. The higher verification rates facilitated by One Login have also led to an increase in application volumes.

We're proud that the implementation of GOV.UK One Login into the 'Request a Basic DBS Check' service has driven substantial improvements in user satisfaction, from 81% to 88%, and increased operational efficiency. By reducing the time required for submissions, increasing successful verifications, and ensuring reliable communication through verified contact details, GOV.UK One Login has proven to be a pivotal enhancement in the DBS basic check process.

Looking to future work

GOV.UK One Login has been of great benefit to DBS users and the DBS after only 18 months of live service operation. We are looking forward to the new identity proving journeys and features that One Login has on its roadmap for 2024 and the positive impacts that they will have on our users.

We intend to take full advantage of the opportunities that they will provide for users with, for example, inclusivity challenges. We will continue to work very closely with One Login colleagues and build on our great relationship to ensure that the end-to-end service journey is as simple, easy and accessible as possible.

Allan Robinson - Product Manager, Basic Digital Service Team, DBS

The Government Digital Service's (GDS) mission is to design and protect the user experience of government - making it better for users and cheaper for the taxpayer. Over the last year, we've been asking ourselves some big questions. Like what's changed socially, technologically and economically since we were set up in 2011? We've challenged ourselves to test previous assumptions and technical decisions about how digital services needed to work, thinking critically about which aspects might need to change versus what should remain the same.

We've always believed that when you make things open, it makes things better. So when our counterparts from the Office of Management and Budget, part of the Executive Office of the President of the United States, invited us to have that very conversation with a group of global digital leaders, we were first in line.

Earlier this year, we swapped our UK base for the storied halls of the White House, and the iconic backdrop of Washington DC. Alongside colleagues from I.AI and the Central Digital and Data Office (CDDO), we sat down with leaders from the USA, Trinidad and Tobago, Cyprus, Canada, Germany, Iceland, the Netherlands, Azerbaijan, and Singapore for an inspiring (and intense!) three days of discussion, debate and storytelling. Read on for the key takeaways from our time in DC.

Our four main takeaways

Trust is everything

In an era of increased complexity and misinformation, users justifiably expect accurate guidance and information from government. They want to trust that services are designed around their needs, and be confident their data is suitably protected and responsibly used.

Frustrating or fragmented interactions can harm trust. The group agreed that adopting modern user-centric design principles is just one of the ways digital governments around the world can build and maintain trust among our users. By designing the user experience of government better, we have the opportunity to create more positive interactions between users and the state.

In the UK, we maintain trust through our unified digital brand, GOV.UK, supported by an ecosystem of other digital components including GOV.UK Pay, Notify and Forms. These platforms already offer a consistent, accessible experience for users who interact with our government, and drive efficiency for our taxpayers. We're now expanding our channel strategy to strengthen the trust they have in our services by meeting them in more places, and offering information and support in alternative formats that work best for them.

The role of connected data

Canada and Singapore kicked off the conversation highlighting their efforts to connect data around users in order to make services more efficient, and personalised. As you'd expect, privacy and security were at the core of the conversation, but so was data quality. As one US delegate summarised: "if you put garbage in, you'll get garbage out."

We know that data which is seamlessly integrated across departments could revolutionise government services - but only if we work together to achieve high levels of quality, privacy, security and user trust.

Here in the UK, the Data Standards Authority established standards to streamline data sharing and usage and a CDDO roadmap mission is focused on better data to power decision-making. At GDS, reimagining how government services could be offered if they were designed around (quality) data is something which is at the heart of our thinking. Specifically how GOV.UK One Login could enable a faster, more personalised and more proactive experience of government in the near future.

Generative AI needs guardrails and governance

The conversation naturally shifted from data to generative AI. Like any other technology, AI will only ever be as good as the data we use to train it, or the guardrails we apply to govern its use.

Governments around the world are exploring AI to boost productivity, economic growth, and improve policy-making. At GDS, we're experimenting to see whether it can improve the government-to-citizen experience. And last November, the UK proudly initiated a global dialogue at the first-ever AI Safety Summit. The Bletchley Declaration which followed, endorsed by 29 global signatories, aims to use AI to "enhance human wellbeing, peace and prosperity".

In Washington, fellow leaders balanced the ambition to innovate at pace and at scale, with accountability for the wider potential impacts of AI. As a group we recognised that the AI challenge is as much about the people, policies and governance as it is about the underlying technology. Working together, we can harness AI's potential to enhance public services alongside safeguarding societal values. But only if ethical governance, transparency and collaboration remain central to the work.

Alternative funding models can benefit innovation

The majority of the global partners in the room were still subject to fixed budget cycles. We discussed how traditional government funding approaches can hinder agile software development, and the rapid experimentation and adoption of emerging technologies, and what alternatives might exist.

Our US hosts shared their experience with the Technology Modernisation Fund (TMF), which acts as an alternative to Congress' 12 funding committee routes to accelerate funding for high-potential digital, cybersecurity, and data projects.

The TMF mission is to "enable agencies to reimagine and transform the way they use technology to deliver their mission and services to the American public in an effective, efficient and secure manner". It has supported over 57 projects with $900 million in funding to date, including guardrails to protect the investment by providing technical expertise and board oversight to guide the work.

Adopting innovative funding arrangements - such as the TMF - could enable governments to more rapidly embrace technological advancements and drive digital transformation outside of fixed, traditional funding cycles.

Where next?

Our discussions in Washington, and the generous insight shared by those who attended, underscored the importance of many of the things we are already focusing on: building trust; unlocking the transformative power of data and pursuing responsible innovation.

It was wonderful to come together and reflect on all of the work already happening across the globe to make interacting with government simpler and faster for the millions (billions?) of users who collectively rely on us. But there is so much more to do, and we are up for the challenge.

We will all go further, faster if we keep sharing and learning from one another and we look forward to continuing the conversations and the collaborations sparked with our international colleagues.

Over the last 12 months at the Driver and Vehicle Standards Agency (DVSA), we've published 12 GOV.UK Forms, and we've received more than 14,000 form submissions. We've also developed a clear backlog of forms that could be created using GOV.UK Forms.

We want to share how we've gone about this so you can learn from what we've done - and hopefully so we can learn from your comments and feedback.

Making sure GOV.UK Forms is considered for new forms

At DVSA, we already had a centralised policy and process for getting approval to create new forms.

We've been working to update these so that GOV.UK Forms is considered as an option for creating new forms. As GOV.UK Forms' features increase, more forms will become suitable, and we expect fewer new PDF forms will be published.

We hope to launch this updated policy and process soon. It will mean that when a new form is flagged as being suitable for GOV.UK Forms, the Content Design team will be able to step in and help to design and publish it.

Identifying existing forms to remake with GOV.UK Forms

To bring some structure to identifying existing forms that could be remade with GOV.UK Forms, we started with a content audit.

We audited all of the forms we have published on GOV.UK. It included:

• forms published on dedicated forms pages, such as appeal an MOT test result
• forms embedded in detailed guidance for specialists, such as applying for permission to set up drink-drive rehabilitation courses

Our audit included the:

• page title
• page URL
• form title
• form format (for example, PDF)
• number of submissions per year
• service group the form belongs to

We also recorded whether the form requires:

• supporting documents
• complex branching (more than just skipping one question based on an answer)
• a payment

In total, we identified 136 forms as part of the audit.

Identifying things that would be better as forms

We also audited all of the places on GOV.UK where we ask users to send structured information to a DVSA email address.

We found about 60 of these. In most cases, we ask users to email no more than 5 pieces of data to us as part of a process, such as replacing lost documents.

Example of a page on GOV.UK where we ask users to send us structured data.

When we started talking to the teams that deal with the incoming emails, we found that most miss some or all of the required information. The teams then need to go back to the user and ask for the missing information.

It slows down the outcome for the user, and it's frustrating and inefficient for the teams dealing with the emails.

Again, we recorded whether the email process required supporting documents or payments further down the line.

Starting out simple

We decided to start with some of the most simple things we'd identified - such as processes which mainly involved collecting information by an email address.

Because this early version of GOV.UK Forms didn't have these features at the time, we ruled out anything that needed supporting documents, complex branching or taking payments*.

Designing and publishing forms

At DVSA, we've made the decision to centralise the creation, publication and management of GOV.UK Forms in our Content Design team.

GOV.UK Forms has been designed to help any teams in government to create accessible online forms, regardless of their technical skills. However, at the DVSA we decided we'd achieve the best results by having user-centred design experts working on the forms.

We have the skills to:

• write for user interfaces
• design good questions
• advocate for users by challenging requests that do not support their needs
• make sure the start point on GOV.UK for accessing the form is designed at the same time as the form itself - bringing more consistency across the user journey
• make sure other content on GOV.UK is reviewed and updated as necessary as forms go live

We will not go into detail on how you actually create a form within the platform. But if you want to see a demo of that, do go and read this blog post and watch the embedded video about how easy it is to use GOV.UK forms.

Measuring success

We're trying to follow the guidance in the Service Manual about measuring success as much as possible.

The GOV.UK Forms platform shows us the number of form submissions and forms started but not completed over the last 7 days.

If you keep a record of these, you can build up a picture of your form's completion rate over time.

Example of the analytics currently available within GOV.UK Forms.

We know this is just a minimum viable product (MVP) of metrics and that more detailed performance data will be available later this year. For example, including total form submission numbers and a breakdown of interactions with each question page. These would help to identify where we might need to improve the wording for any questions.

Measuring user satisfaction with GOV.UK feedback pages

We've worked with the content team at the Government Digital Service to create GOV.UK feedback pages (sometimes called 'completed transaction' or 'done' pages) for our forms.

This allows us to collect feedback from users on:

• how satisfied they were with the form
• any improvements they would like to see

We're using this to work out user satisfaction ratings for each form.

Example of a GOV.UK feedback page collecting user satisfaction and feedback.

We're getting feedback that shows that users are finding the forms easy to use.

"It was very straightforward, easy to use."

"Could not be better."

"This is the first time I have used this service and it is much easier than having to email through a request."

We're also getting useful feedback from users on what they want to see next.

"By accepting payment online when filling out the forms."

"Take payment online for replacements."

"Having an online payment option so the lot could be sorted immediately."

Feedback from the teams processing submissions

We've received positive feedback from the teams that we've worked with to design and publish the forms. They're now busy processing the submissions.

We think this is our favourite so far:

"[The form submission email] looks awesome, really easy to action now and gives us all the details we need, huge improvement!!!"

Working towards services that solve whole problems for users

We're not just creating a GOV.UK Form and then moving on as if the job is done. It's not.

Using GOV.UK Forms can help us make some current processes more efficient in the short term. But in the longer term, we need more user-centric policies and services that reduce the need for forms.

As an example, one of our GOV.UK Forms allows MOT centres to apply for a replacement approval certificate. This proves they're approved to carry out MOTs - they have to display it on a public notice board.

When the MOT centre applies for a replacement, a member of staff at DVSA will generate a PDF certificate and then email it to the MOT centre so they can print it out.

So you can see that we could:

• build that feature into our digital service for MOT centres, so they can just print a replacement as and when they need one
• rethink whether or not it's necessary for policy and regulation to mandate that MOT centres have to display a paper certificate

But by using GOV.UK Forms, we can make things more efficient right now. And it makes it easier to make iterative improvements to forms, benefitting thousands of users.

Equally as important, it helps us to:

• refocus on the underlying user needs behind lots of these smaller tasks, record them and build up a bank of them
• collect data and feedback from users that can be fed back into services and policy development

* Providing documents and complex branching are amongst the GOV.UK Forms teams top forthcoming features with work planned for the second half of 2024. GOV.UK Forms now includes the new 'taking payments' feature, which was released in March 2024.

GOV.UK Notify has finished our migration from GOV.UK Platform as a Service (PaaS). In our previous blog post we talked about how we migrated our database. In this post, former Notify team member David McDonald explains how we migrated production traffic to our new apps running in Amazon Web Services (AWS) Elastic Container Service (ECS).

Building the new ECS infrastructure

On PaaS, we had 3 environments: preview, staging and production. We had about 25 apps running per environment, mostly deployed using the Cloud Foundry Python buildpack.

We built an equivalent set of these 3 environments, deploying our apps as Docker containers in ECS. We also built new deployment pipelines and monitoring infrastructure. Those 2 sentences do not do justice to the hard work of our team of 7 who worked on this, alongside other infrastructure we migrated, over a period of 18 months.

The 3 new ECS environments were separate from PaaS and were built to share a minimal amount of infrastructure with their PaaS equivalent. For example, the AWS SQS queues that we put our Celery tasks on were kept separate, so that the same environment that created a task would also process it.

The PaaS environment and equivalent ECS environment did share the same state though. For example, they shared a Postgres database per environment. This meant if you visited the same URL on the PaaS environment and the ECS environment, you would see the same result and the same data.

We had built these 3 new environments but by default they received no traffic.

Our plan for migrating traffic

Near the start of our migration project, we identified all the ways that traffic comes into Notify. We identified 7 of these 'entry points'. We planned how to migrate each of them to stop sending their traffic to our PaaS environment and instead send it to our ECS environment.

Our approach was to:

1. avoid any downtime for our users
2. migrate each entry point independently to keep things simple
3. use a percentage approach where possible to reduce any impact of mistakes or problems. For example, to start by sending only 1% of traffic to ECS before slowly increasing the percentage

I'll explain in detail about how we did this migration for our different types of entry points.

HTTP requests entering at an AWS CloudFront distribution

Five of our entry points were AWS CloudFront distributions. These distributions are where HTTP requests enter our infrastructure. Each distribution links to a particular subdomain.

For example, one distribution is responsible for receiving all traffic to www.notifications.service.gov.uk and forwarding it to the appropriate application. Another distribution is responsible for traffic to api.notifications.service.gov.uk.

On PaaS, when a user visited www.notifications.service.gov.uk, the CloudFront distribution received their request and forwarded it to the relevant PaaS origin.

On AWS ECS, the same CloudFront distribution would need to receive the request and forward it to its new origin, an AWS Application Load Balancer. The load balancer then has our ECS apps as its targets.

We could have simply changed the origin of our CloudFront distribution from the PaaS origin to the ECS origin when we wanted to migrate traffic. However, this would have affected 100% of traffic immediately and we had already decided to take a percentage-based approach.

We used CloudFront Lambda@Edge to run a small bit of Python code for every request that would decide whether to send the request to PaaS or to ECS.

The Python code looked a bit like this:

To start with this code will have no effect. All requests will still go to PaaS because the ECS_TRAFFIC_PERCENTAGE is set to 0.

If we change the value of ECS_TRAFFIC_PERCENTAGE this will divert a percentage of requests to our new ECS_ORIGIN by overwriting the PaaS origin already set in the request object.

We also added 2 further improvements to this Lambda@Edge function.

First, we added support for forcing a request to go to either PaaS or ECS based on custom HTTP headers. If we included our ECS HTTP header in a request then the function would always send the request to ECS. We put the equivalent in place for forcing requests to go to PaaS.

This was useful for our manual testing, and even more important for the testing done by our deployment pipelines. The tests run by the deployment pipelines should tell us that a specific environment is working correctly. Using custom headers meant we could ask our tests to target either the PaaS or ECS environment.

Second, we changed our Lambda@Edge function so we could very quickly decrease the percentage of traffic going to ECS, for example if we spotted a problem and wanted to revert back to sending all traffic to PaaS. With our original function, the percentage was hardcoded and to change it we had to deploy a new version of the function and associate it with the CloudFront distribution - this took about 10 minutes.

We moved the percentage value out of our code and into an AWS DynamoDB table. Changing a value in DynamoDB would only take us a few seconds and we wouldn't need to deploy a new version of our function. Having our function call out to DynamoDB did have a performance hit, roughly 200ms, but we added 30 seconds of time-based caching so that the vast majority of requests wouldn't be slowed down.

Email delivery receipts entering at an AWS Lambda function

The sixth entry point is an AWS Lambda function that receives delivery notifications from AWS Simple Email Service (SES). Whenever it receives one, it takes the JSON it receives and puts it in a Celery task on an SQS queue to be picked up by one of our apps.

For the migration we tweaked the Lambda function code so it put the Celery task on the relevant queue in our PaaS environment, or the equivalent queue in our ECS environment, based on a percentage value. The code we added was similar to the code we used for our CloudFront Lambda@Edge functions.

Celery tasks put on queues by celery beat

The seventh entry point is celery beat. Celery beat is a scheduler - it creates Celery tasks at regular intervals. For example, it is responsible for creating a Celery task:

• every evening at 5:30pm to send letters to our print and postage provider
• every hour to generate the latest billing summary for our users
• every minute to generate delivery metrics so we know how Notify is performing

Celery beat itself doesn't process the tasks it creates. Instead it puts them on an SQS queue and one of our other apps will pick up the task from the SQS queue to process.

We only run a single instance of celery beat in PaaS production. Instances of celery beat don't share state, so if we ran 2 instances then we would have 2 tasks created for every item in our celery beat schedule. While this wouldn't be an issue for most of our Celery tasks, we knew at least some are not able to be run twice without having an unwanted impact.

For example, running our letter sending task twice in one evening would mean 2 duplicate letters arriving on your doorstep!

This behaviour also means we couldn't necessarily run celery beat in PaaS and ECS at the same time. Both environments share the same database and state, so duplicate tasks might end up with duplicate results.

To migrate celery beat, we had to decide between 3 options:

• turn off celery beat in PaaS production and then turn on celery beat in ECS production. There would be a short amount of time (probably under a minute) where celery beat was not running in either environment
• turn on celery beat in ECS production and then turn off celery beat in PaaS production. There would be a short amount of time (probably under a minute) where celery beat was running in both environments and any duplicate tasks run during this time could have an unplanned impact
• review all of our Celery tasks to ensure that they could be run twice, or at the same time, without any impact. Then we could run celery beat in both PaaS production and ECS production at the same time knowing this would not cause problems

We chose the first option because it was quick and simple. We timed the swap over for one of the quieter times of day where no critical tasks were scheduled.

How it went

On 9 January, we started sending our first production traffic to ECS. For our lowest traffic and lowest impact subdomain, we configured its CloudFront distribution to send 1% of requests to ECS. This subdomain only serves GET requests, so if a request is served by ECS and errors, the user can just reload the page. The reload will likely be served by PaaS with no further impact to the user. Regardless, we kept a close eye on our logs to spot any errors.

On 6 February, we started sending 0.1% of traffic for api.notifications.service.gov.uk to ECS. This was our last subdomain because it was the highest risk. If requests to our API fail, then we might break other web services run by the public sector -- maybe even leaving their users stuck in a web journey or never receiving an important notification.

We started with just 0.1% of traffic because our API receives thousands of requests per minute so even 0.1% would be enough to give us confidence that things were working without the risk of affecting a large number of users.

Once we had a small percentage of traffic from each of our CloudFront distributions going to ECS, we had reasonable confidence that our apps were working correctly. Over the next few weeks, we gradually increased the percentage of traffic going to ECS, while continuing to closely monitor. This helped confirm that our autoscaling and the capacity of our new environment were sufficient.

With the migration of HTTP traffic going well, we turned our attention to the Lambda function for our email delivery receipts. Between 27 January and 28 February, we slowly increased the percentage of email delivery receipts going to ECS until we reached 100%.

On 27 February we migrated celery beat.

On 8 March we increased the percentage of traffic going to ECS for our final CloudFront distribution to 100%. At this point, all production traffic was being served by ECS.

Migration complete

We'd managed to migrate all of our applications to our new infrastructure without any downtime. We monitored for one further week before calling it 'job done' and celebrating. Migrating our apps was the last part of migrating off the PaaS. We were only left with the final and most pleasing of tasks - deleting lots of code and infrastructure we no longer needed!

The GOV.UK One Login mission is to ‘make it easy for everyone to access government services’.

However, translating exactly what ‘for everyone’ looks like in the context of identity proofing is not an easy task. There can be many barriers for a user to complete an identity journey and gain access to government services. Our work has shown that these not only exist for various user groups, but they are overlapping in nature and not easy to overcome.

Our ambition means it is essential that we design solutions that help users access services online. User-centred design (UCD) is at the core of our work and how we address our inclusion challenges. 

In this blog, we’ll share a strategic piece of UCD work on user segmentation to help GOV.UK One Login plan delivery of our inclusion agenda. And, in particular, how we’re working to address the barriers of a particular user group that indexed highly in our findings, namely young people. 

Building our evidence base around inclusion 

At GOV.UK One Login, we’ve been doing end-user research since April 2021. To date, we’ve carried out over 150 rounds of research with over 2,000 participants, including live end to end testing and observational studies of participants engaging with our journeys. We’ve got a rich pool of qualitative data and insight around people’s lived experience of proving digital identities and using our service. 

Despite all of this work, we knew we had a gap in understanding of the scale of digital inclusion and the impact of those insights across different user groups.  

In 2023, we started a piece of work on ‘user segmentation’ to uncover inclusion gaps in our coverage and evidence ways to address these. We wanted to understand the technology and identity documents our users have access to, as well as their levels of digital confidence. 

The aim of the user segmentation work was to get more in depth insight into different user groups (or segments) that share similar characteristics in terms of their barriers to using GOV.UK One Login.

Working with colleagues across the programme, we defined a set of parameters that would enable us to build a sustainable data set on inclusion, which the programme can augment over time. We surveyed 2,000 users, ensuring the results were representative of UK population demographics. The survey answers were self-reported allowing us insight into people’s direct experience and perception of barriers to proving their identity. You can access the fuller findings here. 

In addition to our inclusion data set, the output of the user segmentation work is helping GOV.UK One Login more easily quantify the value and impact of inclusion related initiatives and to better understand how the One Login product is performing for different user groups. 

Overlapping barriers and opportunities for the digitally native

Our work revealed something surprising about digital exclusion. 

It is well documented that network connectivity is a common, yet, significant issue across digital services. Despite this challenge, mobile device penetration is extensive, with only 3% of all survey respondents not owning a device that can access the internet. 

A small but significant margin (4-9%) of users did not have the evidence for identification which is critical to digital identity verification. A larger proportion of users, and a significant barrier for GOV.UK One Login, struggle with having enough credit history for identification, if they were using the security questions route. This is because a person may not have a mobile phone contract, as well as an active bank account, credit cards, loans, or mortgages used to establish a financial UK digital footprint. 

Our focus in the past year has been to widen the evidence types and government data sources for users we know will find it particularly difficult to prove their identity online. We believe this data will significantly support addressing these barriers.

The most surprising finding from those surveyed, were the barriers for users under the age of 17: 

- only 4% have enough financial or government footprint for identification

- 46% don't have a phone number (they have a mobile but without a phone number)

- 20% don't have an email address

- 42% have difficulty completing online tasks and therefore need help from friends and family

Users in the 13-17 year old group are between 20% and 33% less likely to be able to prove their identity through our service compared to the average potential user.

Our findings are not unique and speak to recent data from OFCOM. As a result, we did a little more digging to understand how this compares with a wider cohort of young people.   

The 18-24 year olds have similar barriers but experience them less acutely:

- only 22% have access to an active bank account, credit cards, loans, or mortgages used to establish a financial UK digital footprint (compared with 43% for the general population)

- only 16% don’t have a phone number (compared with 20% for the general population, meaning that in this aspect 18-24 year olds are better off)

- 13% don’t have an email address (compared with 8% for the general population)

- 43% have difficulty completing online tasks (compared with 25% for the general population)

Young people (aged 24 and under) will be the first generation to have just one government account, it's crucial that we get it right for them. From applying for their first apprenticeship to signing their mortgage deed, they will rely on their GOV.UK One Login account throughout their lives, which will transform the way they access government services for years to come.

Continuing to iterate our service to meet our user needs 

As a result of our findings, GOV.UK One Login is actively exploring how best to respond to these findings in terms of how they might shape our product roadmap in future. Specifically, we’re focused on access to identification, mobile usage, lack of email, and increasing digital confidence. 

We know this is just the start. When it comes to using the user segmentation data across GOV.UK One Login there are a number of pieces of work underway to combine this data with live service data, performance analytics, our own user research insights and that of other government services.

We have also produced a characterisation of a set of user groups identified from our segmentation work, and are using it to inform our day-to-day design practice and user research participant recruitment and to seed inclusion insights at scale, in all our practice.   

The Government Digital Service (GDS) is on a mission to help the government make brilliant public services for everyone. As part of this, GOV.UK One Login lets users sign in and prove their identity so they can access central government services quickly and easily.

More and more, we are seeing everyday services move online, from shopping to banking to food. It can be taken for granted that people automatically understand how to perform these tasks online, even as those tasks grow in complexity.

As government, we need to ensure that as our services move online, we provide support for people who are less confident. Whether it's seeking assistance with an account, resolving an issue, or simply needing guidance, the ability to connect can make all the difference. GOV.UK One Login’s contact centre is marking one year, and has expanded our offer to provide round-the-clock support.

Through the contact centre you can:

• speak to an agent on the phone, Monday to Friday 8 am until 8 pm
• use a webform to share your issue, and an agent will get back to you
• use the web chat 24/7, with the ability to transfer to a live agent for more complex issues from Monday to Friday 8 am until 8 pm
• use the contact centre’s chatbot for 24/7 support

New services in the GOV.UK One Login contact centre

We've recently introduced two new features to the contact centre: a live web chat, where our agents can interact directly with users, and a 24/7 chatbot, an automated service that quickly and efficiently handles initial user queries. These two services work together to provide a seamless user experience.

The web chat has evolved as the programme has grown and the agents are now able to manage multiple chats with users. The chatbot has also increased the number of tasks it can handle at any one time. This enables users to quickly resolve straightforward issues through a chatbot, meaning our agents are able to dedicate their time to more complex issues.

Using the right channels

Introducing web chat and the chatbot were decisions that came out of the contact centre’s Right Channel Strategy, aimed at making the service fit the needs of our users. This strategy is underpinned by four principles: 

• intelligent channel route
• scaling
• self-serve
• continuous improvement  

Intelligent Channel Route

An Intelligent Channel Route ensures customer inquiries are directed to the most appropriate channel and agent based on complexity, urgency and availability. For example, in an instance of suspected fraud, a user will be immediately routed to a live agent. However if someone wanted to raise a Freedom of Information (FOI) request, they would be routed to webform as FOIs need to be written to be responded to. 

Ability to scale up at busy times

This strategy allows the service to scale up at busier times, such as major deadlines. As the strategy is in motion the contact centre will continuously monitor and measure its performance and tweak the service as it grows. 

Enabling self-serve

There has also been a significant amount of work on the self help guidance pages. These pages provide self-help guidance which empowers users to find solutions to their problems independently. Instead of relying solely on customer support or assistance, users can access resources and information to address their needs promptly.

The self-help guidance pages offer a convenient way for users to troubleshoot issues or learn how to use a One Login without needing to contact customer support or navigate through different menus. User research has found that users appreciate the ability to find answers quickly and easily on their own time, without waiting for assistance.

Contact centre feedback loop

Users get the opportunity at the end of the call to complete a short customer satisfaction survey. On average, nearly 50% of users complete this which is above standard industry benchmarks. From these surveys the contact centre averaged an 85% satisfaction rating with ‘Agent knowledge’ and ‘Professionalism’ scored at 90% or above. Users state that we resolve their query around 75% of the time at first point of contact.

It is our aim that this user satisfaction score will grow over time and more resources are added. By incorporating customer feedback and usage data, we're able to continuously refine and enhance our service, ensuring it meets users' needs more effectively over time.

Introduction

Here at the Government Digital Service, we're committed to gender equality throughout our workforce. So, it's crucial that we do our bit to make sure younger generations can take advantage of the fantastic opportunities that exist through careers in technology. 

That’s why GDS recently hosted three students from Next Tech Girls (NTG), a social enterprise set up in 2016 that works with girls currently in education looking to work in STEM (Science, Technology, Engineering, Maths) careers. They especially focus on working with girls from lower income and minority backgrounds. 

Next Tech Girls facilitates work placement and career opportunities for the students across a broad spectrum of organisations. Their mission is “to inspire today's girls in education to be tomorrow's women in tech”. As a woman who has been on that journey myself, I wanted to get involved to bring opportunities I didn’t have in the past to the next generation of female digital and technology specialists.

“Only 26% of the UK tech industry are women” - Next Tech Girls

In the UK, Next Tech Girls statistics show that there is currently a huge gender gap in STEM careers, with women severely underrepresented. Women are much less likely to enter into a STEM career path due to a number of factors.

This can range from the stereotypes of it being a predominantly male career path (I was one of the only girls on my own university computer science degree out of approximately 100, which was intimidating), issues with a lack of role models and visibility of women working in tech, to a lack of advocacy and support in school to pursue these types of careers in the first place.

This can mean girls are not able to make these choices at the critical time they are building the qualifications and knowledge needed to pursue a career in STEM.

Setting up the work placement

It was important for GDS to understand what the students wanted to get out of the work placement, so we met with the girls before our week together started to understand a bit more about them, their interests and background, and to make sure we were going to provide a good experience for them.

We learnt that the students were especially interested in our individual career paths, what qualifications we each had, what helped us get to where we currently were in our career, and what a day in the life of different disciplines looked like, especially what it was like to work in a modern office environment. 

Part of this was going to be about exposing the students to different STEM roles and activities, so it became important to assemble a set of people they could work with across a variety of roles. This led to putting together a team of designers, developers, delivery managers, data scientists, and leadership roles to spend some time throughout the week giving the students a variety of perspectives.

One of the principles of this project was to ensure the majority of the people speaking to the students were female, and we achieved that. 9 out of 10 of the staff involved were female, and could offer that unique perspective on their careers, challenges and approach to work to the students. 

I also wanted to ensure we inspired the students, and to give them confidence that they can pursue any career they wanted to. Luckily at GDS we are not short of inspirational and accomplished women, who agreed to put together a timetable of immersive activities and learning.

“Our session explored different accessibility features on Android and iOS. It was a hands-on session, and we all learned by playing and exploring (me included!). My favourite bit of the session was when we were able to talk about what we'd covered about accessibility and how they might make changes to their previous coursework to make their website designs more inclusive.

I hope that they can take that experience and build on it in whatever career path they choose.”

- Elise Robinson, Senior Designer at GDS

Our agenda was across an action-packed five days, which we divided up into a morning session, then lunch followed by an afternoon session that finished at 4pm. It was important to respect the time of our volunteers, but also give the students a bit of time to learn from people, then go off and complete some self-directed activities and tasks. The challenge here was not making it too much like the school environment, and exposing them in a safe way to a work environment where they had to use their initiative to develop their own skills.

We kicked the week off with an introduction from myself to GDS and our unique history. I then gave a brief overview of my education and career to date, what I enjoyed and what I found a challenge. This format was used by most of our volunteers to give an insight into how different people approached their careers, followed by some hands-on activities and experience. I also set them a task for the week of summarising what they learned during their time with us, to present back on the Friday afternoon.

A sample of the other sessions students attended were:

• spending time with Senior Interaction Designer Monica doing design challenges in Figma: one challenge focused on redesigning some WhatsApp screens
• enjoying time learning how to do HTML, CSS and Javascript coding with our Front End Developer, Becca

End of the placement week

At the end of the week, we all convened to watch a presentation from the placement students. This was the highlight of the week for me, as not only was it an energetic presentation from the girls, but seeing the sheer amount of things they learned about and put into practice over such a short time that week was incredible. They told us they really enjoyed the week and have a good idea of what to do next to start out in their careers.

As a team, it was also useful to hear feedback from the girls at the end-of-week session. It was great to hear that the students had a positive experience at GDS, and to know we now have the blueprint to offer this sort of work experience again.

We learned it was so important to have a strict agenda to stick to, to give confidence that the students would have a good mix of taught and self-directed learning, and to see the spread of disciplines and activities they would be doing. It was also useful to set up a Slack channel for all volunteers to keep in touch with, and in case of emergencies. 

“I was really inspired by the energy, enthusiasm and creativity of all the girls. It was great to see how quickly they took to the activity of journey mapping, they approached it in a fresh way, bringing their own personality and were able to articulate their detailed design thinking.

It was a rewarding experience to see how the girls work and how their ideas could challenge and bring positive change to the tech industry. I think it’s important that we continue to support young girls and open them up to different opportunities in technology”

- Sally Tolsen, Senior Interaction Designer

We learned it was important to be candid and talk about not just the surface elements of a career, but the deeper aspects such as challenges, pay and uniquely female experiences such as “what happens if I get my period at work?”. We had to remember that they weren’t familiar with many of the daily aspects of work we take for granted as experienced STEM professionals.

One thing that was consistent between all volunteers was that nothing prepared us for how much we would enjoy this week. It was a thoroughly positive, fulfilling and uplifting experience to be part of a group of women supporting the next generation of girls leaving school to start their own careers.