Periscope Data Functionality Limited

Incident Report for Sisense Service Page

Postmortem

The following is the incident report for the Periscope Data by Sisense incident that occurred on June 10th and June 11th, 2019. This issue resulted in limited functionality of slower loading times within the app, longer query run times and caching delays. We understand the impact this had on our customers and sincerely apologize. We have taken a number of steps to prevent this issue from occurring in the future as detailed further below.

ISSUE SUMMARY

Our investigation into the issue found that the system degradation occurred as part of our efforts to upgrade our database infrastructure, specifically the adoption of the most recent version of Aurora Postgres. Over the weekend, Periscope Data performed system maintenance as a part of our routine scalability enhancements, including the Postgres upgrade. Although we configured this update based on AWS’s specifications, we still experienced some hotspots that required adjustments.

REMEDIATION

We are confident that we have identified the necessary corrections and are equipped to handle similar upgrades in the future. The Periscope Data team is committed to creating the most reliable data platform possible. As a response to this issue, we’ve made key improvements to our infrastructure:

Over-provision the databases in order to account for excess capacity
Introduce additional redundancies to ensure high availability of the database
Implement new integration testing to complement the addition of the increased redundancy and capacity of the databases
Invest in anomaly detection at the component-level to increase visibility for aberrant behaviors

In an effort to better support our global customer base, we are pleased to announce we will be expanding our support hours later this year to begin at 7 a.m. EST. We look forward to expanding our availability even further in 2020. More details to follow.

For more information, please reach out to our Solutions Team at [email protected] or via live chat.

Posted Jun 11, 2019 - 19:24 PDT

Resolved

All services are operational. The Periscope Data Support Team can be reached at [email protected].

Posted Jun 10, 2019 - 18:59 PDT

Monitoring

All services are now operational. Engineers are currently monitoring to ensure continued functionality. The Periscope Data Support Team can be reached at [email protected].

Posted Jun 10, 2019 - 18:26 PDT

Identified

Queries, caching and embedded dashboards are running for most users. Engineers have rolled the Periscope Data app back to a previous version and are continuing to investigate issues related to background services and email reports.

Posted Jun 10, 2019 - 17:15 PDT

Update

Queries and embedded dashboards are running for most users and tables are caching for some users. Engineers are continuing to monitor working services and investigating issues related to background services and dashboard reports.

Posted Jun 10, 2019 - 16:07 PDT

Update

Tables are caching and queries are running for some users. Engineers are continuing to monitor working services and investigating issues related to major app functionality.

Posted Jun 10, 2019 - 15:25 PDT

Update

Engineers are progressing on the investigation into underlying app functionality. Impacted functionality may include charts, cache, filter queries, CSVs and embeds.

Posted Jun 10, 2019 - 14:16 PDT

Update

Engineers are continuing to investigate issues related to major app functionality.

Posted Jun 10, 2019 - 13:30 PDT

Update

Engineers are continuing to investigate issues related to major app functionality. The Periscope Data Support Team can continue to be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 12:47 PDT

Update

Engineers are reporting continued improvements to internal metrics related to major app functionality and are continuing to investigate root causes. The Periscope Data Support Team can continue to be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 12:00 PDT

Update

Engineers are making progress on the investigation into issues regarding major app functionality and have seen an improvement in our internal metrics.

Posted Jun 10, 2019 - 11:32 PDT

Update

Engineers are making progress on the investigation into underlying app functionality. Impacted functionality includes charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can continue to be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 10:54 PDT

Update

Engineers are continuing to investigate issues related to major app functionality.

Posted Jun 10, 2019 - 10:17 PDT

Update

Engineers are continuing to investigate issues related to major app functionality.

Posted Jun 10, 2019 - 09:42 PDT

Update

Engineers are continuing to investigate issues related to major app functionality, including charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 09:07 PDT

Update

Engineers are continuing to investigate issues related to major app functionality. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 08:31 PDT

Update

Engineers are continuing to investigate issues related to major app functionality. Impacted functionality includes charts, cache, filter queries, CSVs and embeds. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 08:04 PDT

Update

Major functionality including, charts, caching, filter queries, CSVs and embeds are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 07:28 PDT

Update

Charts, caching, filter queries, CSVs and embeds are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 06:57 PDT

Investigating

Charts are not loading for most users. Engineers are actively investigating the issue. We will update as soon as we have more information. The Periscope Data Support Team can be reached at [email protected] or via live chat.

Posted Jun 10, 2019 - 06:40 PDT

This incident affected: Sisense for Cloud Data Teams.