The following is the incident report for the Periscope Data by Sisense incident that occurred on June 10th and June 11th, 2019. This issue resulted in limited functionality of slower loading times within the app, longer query run times and caching delays. We understand the impact this had on our customers and sincerely apologize. We have taken a number of steps to prevent this issue from occurring in the future as detailed further below.

ISSUE SUMMARY

Our investigation into the issue found that the system degradation occurred as part of our efforts to upgrade our database infrastructure, specifically the adoption of the most recent version of Aurora Postgres. Over the weekend, Periscope Data performed system maintenance as a part of our routine scalability enhancements, including the Postgres upgrade. Although we configured this update based on AWS’s specifications, we still experienced some hotspots that required adjustments.

REMEDIATION

We are confident that we have identified the necessary corrections and are equipped to handle similar upgrades in the future. The Periscope Data team is committed to creating the most reliable data platform possible. As a response to this issue, we’ve made key improvements to our infrastructure:

• Over-provision the databases in order to account for excess capacity
  
• Introduce additional redundancies to ensure high availability of the database
  
• Implement new integration testing to complement the addition of the increased redundancy and capacity of the databases
  
• Invest in anomaly detection at the component-level to increase visibility for aberrant behaviors

In an effort to better support our global customer base, we are pleased to announce we will be expanding our support hours later this year to begin at 7 a.m. EST. We look forward to expanding our availability even further in 2020. More details to follow.

For more information, please reach out to our Solutions Team at [email protected] or via live chat.