Usage Data not working
Incident Report for Sisense for Cloud Data Teams
Postmortem

The following is the incident report for the Sisense incident that occurred on May 18, 2020 from 8:44 AM PST to 5:30 PM PST. This issue specifically affected customers on the community cache clusters. These customers were not able to access their usage data for the period of the outage. We understand the impact this had on our customers and sincerely apologize. We have taken a number of steps to prevent this issue from occurring in the future as detailed further below.

ISSUE SUMMARY

Our investigation into the issue found that the system degradation occurred due to the Usage Data Redshift clusters being incorrectly deleted as part of a larger efficiency project. These clusters should have been flagged for non-deletion but earlier modification to the code led to this flag being removed. As part of our efforts to rectify the system, the clusters were restored based off of an earlier snapshot. This restore allowed customers to be able to access and query their usage data again.

REMEDIATION

We are confident that we have identified the necessary corrections and are equipped to handle similar upgrades in the future. The team is committed to creating the most reliable data platform possible. As a response to this issue, we've made key improvements to our infrastructure:

  • Set up processes and checks to prevent Usage Data clusters from being deleted.
  • Set up additional critical alerts to detect missing Usage Data clusters

For more information, please reach out to our Solutions Team at suppordtt@sisense.com or via live chat.

Posted May 20, 2020 - 16:46 PDT

Resolved
Usage Data is now operational. The Sisense Cloud Support Team can be reached at supportdt@sisense.com.
Posted May 18, 2020 - 17:33 PDT
Monitoring
Usage Data can now be queried. Engineers are currently monitoring the fix. The Sisense Cloud Support Team can be reached at supportdt@sisense.com.
Posted May 18, 2020 - 17:14 PDT
Update
Engineers have identified a fix and are rolling it out across the system. The Sisense Cloud Support Team can be reached at supportdt@sisense.com or via live chat.
Posted May 18, 2020 - 16:42 PDT
Identified
Engineers have identified the root cause and are working on implementing a fix. The Sisense Cloud Support Team can be reached at supportdt@sisense.com or via live chat.
Posted May 18, 2020 - 16:23 PDT
Investigating
Usage Data in the Sisense application may not be working for some users. Engineers are actively investigating the issue. The Sisense Cloud Support Team can be reached at supportdt@sisense.com or via live chat.
Posted May 18, 2020 - 16:03 PDT
This incident affected: Periscope.