Periscope Data Loading Slowly and Queries Not Running
Incident Report for Sisense for Cloud Data Teams
Postmortem

The following is the incident report for the Periscope Data by Sisense incident that occurred on June 10th and June 11th, 2019. This issue resulted in limited functionality of slower loading times within the app, longer query run times and caching delays. We understand the impact this had on our customers and sincerely apologize. We have taken a number of steps to prevent this issue from occurring in the future as detailed further below.

ISSUE SUMMARY

Our investigation into the issue found that the system degradation occurred as part of our efforts to upgrade our database infrastructure, specifically the adoption of the most recent version of Aurora Postgres. Over the weekend, Periscope Data performed system maintenance as a part of our routine scalability enhancements, including the Postgres upgrade. Although we configured this update based on AWS’s specifications, we still experienced some hotspots that required adjustments.

REMEDIATION

We are confident that we have identified the necessary corrections and are equipped to handle similar upgrades in the future. The Periscope Data team is committed to creating the most reliable data platform possible. As a response to this issue, we’ve made key improvements to our infrastructure:

  • Over-provision the databases in order to account for excess capacity
  • Introduce additional redundancies to ensure high availability of the database
  • Implement new integration testing to complement the addition of the increased redundancy and capacity of the databases
  • Invest in anomaly detection at the component-level to increase visibility for aberrant behaviors

In an effort to better support our global customer base, we are pleased to announce we will be expanding our support hours later this year to begin at 7 a.m. EST. We look forward to expanding our availability even further in 2020. More details to follow.

For more information, please reach out to our Solutions Team at support@periscopedata.com or via live chat.

Posted Jun 11, 2019 - 19:28 PDT

Resolved
All services are operational. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 11, 2019 - 08:17 PDT
Monitoring
Engineers are continuing to monitor the fix.
Posted Jun 11, 2019 - 07:47 PDT
Identified
Engineers have identified a potential fix and have applied the change. Periscope Data and queries are running normally for most users.
Posted Jun 11, 2019 - 06:56 PDT
Update
Periscope Data is loading slower than normal and queries are not completing for some users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 11, 2019 - 06:20 PDT
Investigating
Periscope Data is loading slower than normal for some users and queries are not completing for some users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 11, 2019 - 05:41 PDT
This incident affected: Periscope.