Sentry
Enterprise Account
The Ministry of Justice have an enterprise account with Sentry.io which is centrally managed and paid for via the Operations Engineering Team within Platforms and Architecture.
The purpose of this is to enable teams to have an easily accessible performance monitoring, alerting and error tracking tool for digital products.
Quota
The Ministry of Justices’ Sentry enterprise account reserves the following usage quotas (per month):
- 30Million Transactions
- 4Million Errors
- 1Gb of Attachments
The quota renews on the 14th of each month and is a shared quota across all the projects in Sentry.
Teams are responsible for actively managing their Sentry usage. We have provided Quota Management guidance around how to manage your teams Sentry usage, to ensure teams get the most value from Sentry.
Our Sentry plan also includes performance monitoring.
Error Spike Protection
The Operations Engineering team have enabled the Spike Protection feature on an organisational level.
Sentry uses an algorithm to keep track of normal error consumption per project. When there is abnormal amount of errors within a project, Sentry will dynamically rate limit the project accordingly.
The purpose of enabling Spike Protection is to prevent an abnormally high volume of errors unnecessarily consuming a large portion of the error quota that may have been caused by an intermittent network issue. This also helps in ensure protection of the error quota so all teams can make use of it.
User Access
All members of the MoJ GitHub Organization can create and sign in to their Sentry.io account using this SSO link.
Anyone in the MoJ Organisation that joins Sentry could potentially view the information sent to it. While this account is a presumed trusted group, it is still advisable to remove any Personal Identifiable Information (PII) from your application transactions and errors that are sent to Sentry. See Sentry’s Advanced Data Scrubbing documentation to redact sensitive information before it is saved in Sentry.
Quota Management
If a quota is exceeded, all further transactions or errors are dropped by Sentry until the next monthly renewal. To prevent exceeding the quota’s, teams should actively manage their usage of Sentry to ensure they are getting the most value for their consumption. This section details some best practices to help.
Error Rate Limiting
Rate limiting is the best way to manage the error consumption of your project. The feature allows you to define how many errors can be sent to Sentry over a period of time. Essentially, enabling a safety net to ensure a predictable worse case scenario regarding excessive error consumption.
Rate limiting should be enabled on every project. Rate limiting should be applied to each DSN key. The Operations Engineering team monitor which projects have set up rate limiting enabled.
See the Sentry Rate limiting documentation to set up rate limiting for each project.
Also see the Setting Useful Rate Limits documentation to help create a suitable rate limit for your application.
Removing/Ignoring Errors
There will some errors generated by your application that you may be unwilling or unable to resolve. This can be due to a technical constraint or an agreement within the team to ignore the error.
For these type of errors, there’s typically little value in persistently recording the error in Sentry everytime it occurs.
Sentry allows you to Delete and Discard errors. Doing so will ensure these errors are not recorded by Sentry, allowing your team to have better visibility of new error within your application.
We recommend deleting and discarding these type of persistent errors. They can quickly consume large portions of the quota for high traffic applications.
You can find a list of deleted and discarded errors in the “Discarded Issues” tab in [Project] > Settings > Inbound Filters. From here, you can un-discarded any of these issues to receive future events.
Transaction Sampling
Transactions represent interactions with your application, such as users going to a /account endpoint. Recording
transactions are useful for knowing how your application behaves day to day and can create useful insights into how your
application is used.
Typically, there’s little value in recording all transactions of a user for analytical purposes. Transaction sampling is used to record a percentage of user activity. This helps reduce the cost of transaction monitoring whilst still being able to reap the benefits.
We recommend a transaction sampling rate of between 0.01 - 0.05 for most applications. Teams should review the value they get from Sentry transactions and adjust their sampling rates accordingly.
For example, if teams don’t make use of the feature you can disable it completely.
Alternatively, teams may want to temporarily increase sample rates of a new feature.
Transaction sampling can be implemented within the application code. See the Example Project Configurations for a recommended starting point in your SDK.
Ignoring Transactions on Administrative Endpoints
Most applications will contain administrative endpoints, which are intended to be used by other applications to check monitoring metrics such as health status.
There is little value in recording these transactions in Sentry, and they can very quickly consume large portion of the
transaction quota. For example, a /ping endpoint being called every 2 seconds by 3 application will generate
~4Million transactions a month consuming 10% of the quota.
Here are some examples of administrative
endpoints /ping,/ping.json,/health,/health.json,/status/**,/robots.txt
Ignoring specific endpoints (or drastically reducing their sampling rates) can be implemented within the application code. See the Example Project Configurations for a recommended starting point in your SDK.
Example Project Configurations
Please contact the Operation-Engineering team on Slack #ask-operations-engineering if you have any questions or need assistance in setting up transaction filtering / rate limiting within your Sentry project or your application.