Periscope Data Functionality Limited
Incident Report for Periscope
Postmortem

The following is the incident report for the Periscope Data by Sisense incident that occurred on June 10th and June 11th, 2019. This issue resulted in limited functionality of slower loading times within the app, longer query run times and caching delays. We understand the impact this had on our customers and sincerely apologize. We have taken a number of steps to prevent this issue from occurring in the future as detailed further below.

ISSUE SUMMARY

Our investigation into the issue found that the system degradation occurred as part of our efforts to upgrade our database infrastructure, specifically the adoption of the most recent version of Aurora Postgres. Over the weekend, Periscope Data performed system maintenance as a part of our routine scalability enhancements, including the Postgres upgrade. Although we configured this update based on AWS’s specifications, we still experienced some hotspots that required adjustments.

REMEDIATION

We are confident that we have identified the necessary corrections and are equipped to handle similar upgrades in the future. The Periscope Data team is committed to creating the most reliable data platform possible. As a response to this issue, we’ve made key improvements to our infrastructure:

  • Over-provision the databases in order to account for excess capacity
  • Introduce additional redundancies to ensure high availability of the database
  • Implement new integration testing to complement the addition of the increased redundancy and capacity of the databases
  • Invest in anomaly detection at the component-level to increase visibility for aberrant behaviors

In an effort to better support our global customer base, we are pleased to announce we will be expanding our support hours later this year to begin at 7 a.m. EST. We look forward to expanding our availability even further in 2020. More details to follow.

For more information, please reach out to our Solutions Team at support@periscopedata.com or via live chat.

Posted Jun 11, 2019 - 19:24 PDT

Resolved
All services are operational. The Periscope Data Support Team can be reached at support@periscopedata.com.
Posted Jun 10, 2019 - 18:59 PDT
Monitoring
All services are now operational. Engineers are currently monitoring to ensure continued functionality. The Periscope Data Support Team can be reached at support@periscopedata.com.
Posted Jun 10, 2019 - 18:26 PDT
Identified
Queries, caching and embedded dashboards are running for most users. Engineers have rolled the Periscope Data app back to a previous version and are continuing to investigate issues related to background services and email reports.
Posted Jun 10, 2019 - 17:15 PDT
Update
Queries and embedded dashboards are running for most users and tables are caching for some users. Engineers are continuing to monitor working services and investigating issues related to background services and dashboard reports.
Posted Jun 10, 2019 - 16:07 PDT
Update
Tables are caching and queries are running for some users. Engineers are continuing to monitor working services and investigating issues related to major app functionality.
Posted Jun 10, 2019 - 15:25 PDT
Update
Engineers are progressing on the investigation into underlying app functionality. Impacted functionality may include charts, cache, filter queries, CSVs and embeds.
Posted Jun 10, 2019 - 14:16 PDT
Update
Engineers are continuing to investigate issues related to major app functionality.
Posted Jun 10, 2019 - 13:30 PDT
Update
Engineers are continuing to investigate issues related to major app functionality. The Periscope Data Support Team can continue to be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 12:47 PDT
Update
Engineers are reporting continued improvements to internal metrics related to major app functionality and are continuing to investigate root causes. The Periscope Data Support Team can continue to be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 12:00 PDT
Update
Engineers are making progress on the investigation into issues regarding major app functionality and have seen an improvement in our internal metrics.
Posted Jun 10, 2019 - 11:32 PDT
Update
Engineers are making progress on the investigation into underlying app functionality. Impacted functionality includes charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can continue to be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 10:54 PDT
Update
Engineers are continuing to investigate issues related to major app functionality.
Posted Jun 10, 2019 - 10:17 PDT
Update
Engineers are continuing to investigate issues related to major app functionality.
Posted Jun 10, 2019 - 09:42 PDT
Update
Engineers are continuing to investigate issues related to major app functionality, including charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 09:07 PDT
Update
Engineers are continuing to investigate issues related to major app functionality. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 08:31 PDT
Update
Engineers are continuing to investigate issues related to major app functionality. Impacted functionality includes charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 08:04 PDT
Update
Major functionality including, charts, caching, filter queries, CSVs and embeds are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 07:28 PDT
Update
Charts, caching, filter queries, CSVs and embeds are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 06:57 PDT
Investigating
Charts are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at support@periscopedata.com or via live chat.
Posted Jun 10, 2019 - 06:40 PDT
This incident affected: Periscope.