tag:status.qsr.cloud,2005:/historyQSR Enterprise Status - Incident History2024-03-28T11:47:18-04:00QSR Enterprisetag:status.qsr.cloud,2005:Incident/202727842024-03-15T19:51:45-04:002024-03-15T19:51:45-04:00ToastConnect V1 Outage<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>19:51</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>17:24</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>17:24</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:39</var> EDT</small><br><strong>Investigating</strong> - Our engineers are actively investigating an issue impacting our ToastConnect V1 customers, specifically preventing the flow of orders to ConnectSmart Kitchen. Our teams are diligently working to identify and resolve the root cause of this disruption.<br /><br />We appreciate your patience and understanding as we work to restore normal operations.</p>tag:status.qsr.cloud,2005:Incident/199672692024-02-20T12:17:35-05:002024-02-20T12:17:35-05:00Delays in Availability of Data<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>12:17</var> EST</small><br><strong>Resolved</strong> - This issue was resolved during today's maintenance window.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>10:20</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>10:00</var> EST</small><br><strong>Investigating</strong> - We are currently experiencing delays in the availability of ConnectSmart Host data from our Enterprise Services affecting the components listed below. <br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/198698412024-01-30T17:36:23-05:002024-01-30T17:36:23-05:00ToastConnect<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>17:36</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>15:57</var> EST</small><br><strong>Monitoring</strong> - Maintenance has concluded, and we are now monitoring the results.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>15:11</var> EST</small><br><strong>Identified</strong> - During our regularly scheduled maintenance window, there was an update made to our Toast POS integration's handling of modifiers and child items. This update has caused unforeseen issues for some customers. In response, we will perform an emergency rollback of the release to offer relief at 3:30 PM ET. We will update the status page again once the maintenance is complete.</p>tag:status.qsr.cloud,2005:Incident/197322482024-01-23T09:40:35-05:002024-01-23T09:40:35-05:00Waitlist Confirmation SMS Notifications<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>09:40</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>10:35</var> EST</small><br><strong>Identified</strong> - We have identified the root cause that is preventing Waitlist Confirmation SMS notifications from being sent for visits added from an in-store ConnectSmart Host Client. Development has begun on a fix with a targeted release scheduled for Tuesday 1/23/2024.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>21:19</var> EST</small><br><strong>Investigating</strong> - Waitlist Confirmation SMS notifications are failing for Walk-In or Call Ahead visits added from an in-store ConnectSmart Host Client. We are currently investigating to determine the root cause.<br /><br />Please note: The issue is NOT AFFECTING Table Ready notifications</p>tag:status.qsr.cloud,2005:Incident/186919942024-01-10T17:37:02-05:002024-01-10T17:37:02-05:00Google Waitlist SMS Notifications aren't Sending<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:37</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:10</var> EDT</small><br><strong>Investigating</strong> - For guests joining the waitlist through Google, SMS confirmation messages are not being sent. We are currently investigating the cause and will implement a fix as soon as possible.</p>tag:status.qsr.cloud,2005:Incident/193152412023-12-04T16:00:28-05:002023-12-04T16:00:28-05:00Increased Error Rates<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>16:00</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>11:00</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:39</var> EST</small><br><strong>Investigating</strong> - We are currently experiencing increased error rates with our Enterprise Services affecting the components listed below. These services may be intermittently unavailable or experience decreased performance.<br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/191977102023-11-30T14:47:03-05:002023-11-30T14:47:03-05:00Minor Issue - Company-Level Reports Insights Portal<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>14:47</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>16:19</var> EST</small><br><strong>Investigating</strong> - QSR has identified an issue affecting a small number of customers within the Insights Portal. Users will encounter an error stating "Error getting data" when company-level reports are run or dashboards are accessed with ALL sites selected in the dropdown.<br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/190802322023-11-10T09:54:35-05:002023-11-10T09:54:35-05:00Toast Connect Emergency Maintenance<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>09:54</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>06:19</var> EST</small><br><strong>Monitoring</strong> - The emergency maintenance has been completed. QSR is currently monitoring Toast Connect.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:05</var> EST</small><br><strong>Identified</strong> - Toastconnect users may be experiencing intermittent issues with online ordering functionality. This may result in a failure of items to properly display on the kitchen stations. To address this issue, QSR will be conducting emergency maintenance on 11/10/2023 between 6 AM and 6:30 AM ET.<br />During this time order processing may be temporarily unavailable. The estimated downtime should be no more than 5 minutes. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve the issue.</p>tag:status.qsr.cloud,2005:Incident/190223742023-11-05T01:20:53-04:002023-11-05T01:20:53-04:00Kitchen Status Webhook Delays<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>01:20</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>20:17</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>19:21</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in Kitchen Status Webhook notifications <br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/190161912023-11-03T22:37:10-04:002023-11-03T22:37:10-04:00Kitchen Status Webhook Delays<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>22:37</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>21:50</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in Kitchen Status Webhook notifications <br /><br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/186789552023-10-03T15:06:24-04:002023-10-03T15:06:24-04:00Increased Error Rates<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>15:06</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>12:00</var> EDT</small><br><strong>Identified</strong> - We have identified and implemented a fix for some components. We will continue to monitor these services.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>11:20</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing increased error rates with our Enterprise Services affecting the components listed below. These services may be intermittently unavailable or experience decreased performance.<br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/178248822023-07-11T14:20:47-04:002023-07-11T14:20:47-04:00Delays in Availability of Data<p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>14:20</var> EDT</small><br><strong>Resolved</strong> - AWS has implemented a fix for the interruption of service. QSR has tested and verified that all services have fully recovered.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>13:44</var> EDT</small><br><strong>Monitoring</strong> - AWS discovered the root cause and has implemented a fix. QSR Is monitoring.</p><p><small>Jul <var data-var='date'>11</var>, <var data-var='time'>10:41</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in the availability of data from our Enterprise Services affecting the components listed below. <br /><br />AWS engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/175642772023-06-13T19:45:59-04:002023-06-13T19:45:59-04:00Increased Error Rates<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:45</var> EDT</small><br><strong>Resolved</strong> - AWS has reported that all services have returned to normal operation.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:16</var> EDT</small><br><strong>Identified</strong> - We have been notified by AWS that they are experiencing issues that could affect QSR customers. Issues could range from increased latency and errors to a site being unable to connect to QSR Enterprise Services. They have identified the root cause and are working on a resolution.<br />Please visit their status page for additional information. https://health.aws.amazon.com/health/status</p>tag:status.qsr.cloud,2005:Incident/165510412023-03-17T21:34:50-04:002023-03-17T21:34:50-04:00Increased Error Rates<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>21:34</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>20:42</var> EDT</small><br><strong>Identified</strong> - We are currently experiencing increased error rates with our Enterprise Services affecting the components listed below. These services may be intermittently unavailable or experience decreased performance.<br /><br />Our engineers have identified the root cause and a resolution is being put in place as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/133092702022-11-17T20:03:06-05:002022-11-17T20:03:06-05:00Kitchen Enterprise Reporting<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>20:03</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>18:41</var> EST</small><br><strong>Investigating</strong> - We are currently experiencing increased failure rates with our Kitchen Enterprise reporting. Our Engineers are aware of and are investigating the incident.</p>tag:status.qsr.cloud,2005:Incident/106813782022-07-24T22:20:29-04:002022-07-24T22:20:29-04:00Kitchen Status Webhook Delays<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>22:20</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>22:19</var> EDT</small><br><strong>Update</strong> - This issue has been resolved.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>13:48</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in data published to the Kitchen Status Webhook</p>tag:status.qsr.cloud,2005:Incident/106774932022-07-24T07:25:40-04:002022-07-24T07:25:40-04:00Kitchen Status Webhook Delays<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>07:25</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>18:35</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in data published to the Kitchen Status Webhook</p>tag:status.qsr.cloud,2005:Incident/106615402022-07-21T14:22:32-04:002022-07-21T14:22:32-04:00Speed of Service API not Returning Data<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>14:22</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>11:33</var> EDT</small><br><strong>Update</strong> - The issue has been identified and our development team is currently working to deploy the solution</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>11:32</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>09:59</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>09:58</var> EDT</small><br><strong>Investigating</strong> - The ConnectSmart Speed of Service API is currently not returning data. Our engineers are currently investigating</p>tag:status.qsr.cloud,2005:Incident/106057952022-07-16T16:04:00-04:002022-07-16T16:04:00-04:00Kitchen Status Webhook Delays<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>16:04</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>16:03</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>15:33</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in data published to the Kitchen Status Webhook exceeding 180 seconds.</p>tag:status.qsr.cloud,2005:Incident/105948522022-07-15T16:43:51-04:002022-07-15T16:43:51-04:00Kitchen Status Webhook Delays<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>16:43</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in data published to the Kitchen Status Webhook exceeding 180 seconds.</p>tag:status.qsr.cloud,2005:Incident/105563922022-07-14T08:24:28-04:002022-07-14T08:24:28-04:00Latency with displaying Reservation Visits in ConnectSmart Portal or when using Host IOS in standalone<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>08:24</var> EDT</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>16:59</var> EDT</small><br><strong>Investigating</strong> - Restaurants accepting Reservations through ConnectSmart Host Server may see latency with reservations displaying in the ConnectSmart Portal that may appear as Reservations disappearing and reappearing.<br /><br />Restaurants accepting Reservations through ConnectSmart Host IOS (standalone mode) may see this behavior in both the ConnectSmart Portal and the Host IOS app.</p>tag:status.qsr.cloud,2005:Incident/99449812022-05-08T18:19:09-04:002022-05-08T18:19:09-04:00Kitchen Status Webhook - Delays in publishing data<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>18:19</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 8</var>, <var data-var='time'>15:29</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing delays in the publishing of the Kitchen Status Webhook.<br /><br />Our engineers are working to identify the root cause and a resolution as quickly as possible.</p>tag:status.qsr.cloud,2005:Incident/99041912022-05-03T11:25:28-04:002022-05-03T11:25:28-04:00QSR Support Email is currently down<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>11:25</var> EDT</small><br><strong>Resolved</strong> - Email services have been restored. Messages sent during the downtime are now being delivered successfully.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>10:36</var> EDT</small><br><strong>Investigating</strong> - QSR is experiencing issues with our Email server. Messages sent to our Support email are currently not being received. Our Support team is still available by phone.</p>tag:status.qsr.cloud,2005:Incident/92392622022-02-07T13:16:27-05:002022-02-07T13:16:27-05:00Kitchen Status Webhook Delays<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>13:16</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>12:55</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:38</var> EST</small><br><strong>Investigating</strong> - Kitchen Status Webhook is currently experiencing delays. Our team is actively investigating.</p>tag:status.qsr.cloud,2005:Incident/90550012022-01-11T11:01:19-05:002022-01-11T11:01:19-05:00Portal Reports Temporarily Unavailable<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>11:01</var> EST</small><br><strong>Resolved</strong> - ConnectSmart Portal Reports have been restored.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>09:53</var> EST</small><br><strong>Investigating</strong> - We are experiencing an unexpected delay completing this morning's scheduled maintenance. ConnectSmart Portal Reports are temporarily unavailable. Our engineers are working to restore this service as quickly as possible.</p>