A fix to a separate, more minor display issue uncovered a subsequent bug where historic games could lose access to their detailed data tabs. This was resolved in time for the following weekend after it was first reported. 

This season, we've identified and resolved a series of bugs in the complex logic & code which controls which version of a scorecard to show at any point in time. Regrettably, every scenario we've fixed to date has uncovered / affected another scenario, and we haven't always been able to resolve these issues within the affected matchdays.  

The most recent example of these issues (around the start of June) has been the most widespread, causing a large proportion of visits to PCS Pro scored match centres to not load the page contents. We released a fix for this 2 weeks ago, and are pleased to say that we've not had reports of any new widespread issues since then 

This succession of issues has led us to prioritise a more thorough review & technical redesign of how we handle scorecards between the 3 applications, and we will be working on this for next season in the hope of preventing similar issues from affecting future seasons. 

Early on the morning of 2nd June, the new tools we introduced this season notified us of an issue preventing login across all ECB websites. PCS was unaffected.  

Due to the early warning, and additional support agreed with our suppliers, we were able to resolve the issue by around 9:45am. During this time only 13 users had reported issues. 

No, this is not the case. While explaining the architecture of the entire Play-Cricket estate could take up a small book, we'd like to reassure our users that we always ensure enough server power is online to keep the application well within it's safe operating zone (below 70% steady-state server utilisation). There are some areas where the technologies used haven't allow us to scale up our infrastructure without causing more problems, but these are the areas we have made significant improvements and optimisation to over the off-season, and we can confidently state that server power has not been a constraint on Play-Cricket's performance this season. 

Our support team have been working all weekend so far this season and will continue to do so. 

No incidents have affected users' ability to score matches in real-time this season. 

We've not seen any issues with scorers logging into our scoring apps in the 2024 season. 

We are still working on further improvements in this area to ensure we have headroom for much more traffic, but the improvements made to date mean that we've achieved a 3x optimisation in performance and are now able to support Saturday evening traffic without exceeding 30% of the database's capacity. 

The issue that prevented PCS ball-by-ball views from being available last season has been resolved. This was a result of infrastructure in the PCS back-end being above capacity with no option to increase it's scale. Improvements to caching and content delivery mean that for the past few matchdays PCS has never gone beyond 25% of it's capacity. 

While circumstances may result in particular games being out of sync, we have seen no global issues of this nature. Where individual games are affected, the cause has most commonly been connectivity between the scoring apps and Play-Cricket itself, or data in submitted scorecards failing validation when it reaches the server. 

Last season we didn't always have technical and customer support resources available on Saturdays and Sundays when they were needed most. This was a particular challenge with our scoring partners based in the southern hemisphere where responses/resolution sometimes didn't come until almost a full day after the incident. 

Actions taken: 

ECB CS team online every weekend 

PCS support team onboarded 

On-call support scheduled with all suppliers 

Over the first 

Root Cause: A version update to a code library caused the PCS Pro server to become congested, due to change in the library's internal mechanics. 

Actions Taken: 

Significant rewrite of the application to bring code in line with the updated behaviour of the library. 

Testing processes refined around framework/library updates to ensure any future issues are caught by testing before affecting live users. 

Current Status: COMPLETED  

Work was completed last season within a week of the root cause being identified.. No issues with the PCS Pro < > PC integration have been reported since. 

On several Saturdays last season, when scorers booted up the app in the morning getting ready for their match, they were prevented from accessing the match due to login failing. Users were faced with up to 30s wait times before being told that they were unable to login to the app. 

Cause: Timeouts between PCS server and PC. Root cause under investigation. 

Actions Taken: 

Investigated known instances trying to correlate to known issues or incidents 

Added automated monitoring to login methods every 30s on matchdays to ensure login is always available and any slowdowns are captured and evidenced for investigation 

Extended session duration from 24hrs to 14 days, so that app users can still access and score their matches even if login is temporarily unavailable 

Current Status: MITIGATED 

On several Saturday evenings towards the end of last season, users were unable to load scorecards and results in Play-Cricket. The website was very slow to load and in many cases would fail to load entirely. 

Root Cause: Imbalanced load across our databases. 

Play-Cricket usually runs one primary Postgres database instance, and 2 additional read replicas. The server power of the databases is more than adequate, however we identified that last season, traffic wasn't being spread evenly across the 3 servers. One of the read replicas was bearing almost all of the evening traffic (from people logging in to check results, scorecards and stats) and this server was suffering from severe slowdowns. Inadequate monitoring had been setup on the read replicas due to a miscommunication around their usage between 2 of our suppliers. 

Actions Taken: 

Reviewed appropriate server resources 

Creation of new load balancers to prevent connection overload of the read replicas 

Creation of a new database proxy to evenly spread any load across all 3 database instances 

Improved monitoring and alerting across all resources in the stack 

Current Status: IN PROGRESS 

The database and monitoring work has all been completed, however the database proxy requires tuning under real world conditions to get it working optimally. This will be conducted over the next few weeks / weekends until peak performance is achieved. Changes can be made & reverted within minutes, so no major disturbance is expected as this work is carried out. 

Throughout last season, users viewing matches scored by PCS were unable to access the ball-by-ball tab of their scorecards. They were instead presented with a message saying that ball-by-ball views had been temporarily disabled due to high traffic. 

Root Cause: Ageing PCS backend services unable to support peak load. 

During the first few weekends of the season we saw severe negative impacts on PCS scorecards due to the volume of traffic loading scorecards. All scorecard viewing would attempt to retrieve the full history of all deliveries bowled in the game, and at peak times (with 300k users online at the same time) PCS was unable to support the load. 

A manual switch was built in to mitigate this and allow temporary disabling of the ball-by-ball data in the event of an incident, however due to key technical team members being unavailable in the second half of the season due to poor health, the PCS team made the hard decision to disable this feature entirely on Saturdays to avoid putting the rest of the system at risk. 

Actions Taken: 

Overhaul of the scorecard views to only retrieve data that's needed on screen at any given time 

Caching mechanism and modern CDN introduced to vastly increase the PCS back-end's capacity 

Active usage of the fallback mechanism to ensure continuous availability without needing blanket switch-offs on Saturdays. Improved resource model from the PCS team to ensure key people are available when needed. 

Current Status: COMPLETED.  

Ball by ball views will be enabled as standard on all matchdays, only being disabled for very short periods in the event of an incident, and re-enabled immediately once the issue has subsided. 

On 2 occasions last season we had a large number of reports of Play-Cricket's scorecard summaries falling out of sync with the PCS / PCS Pro scorecards. These were rapidly resolved by our supplier at the time. 

Actions Taken: 

Refinement of sync protocols to ensure data consistency 

Implementation of additional checks and balances within the scoring system 

Current Status: Sync issues have been resolved, with ongoing vigilance to maintain data integrity 

Desc 

Actions taken: 

This communication is the first step in addressing this. We hope that this information is appreciated and useful to you, and would welcome your feedback in the comments on how we could make it better. 

A service bulletin will be shared periodically through the season to update you on how things are going and if an incident does occur we will share a dedicated message to share the reality of what happened and what's being done to address it. 

Current Status: This will be an ongoing theme over this, and the coming, seasons.