DFW05 datacenter outage
Incident Report for NCR Aloha
Resolved
All Aloha applications impacted by the DFW05 data center outage are now restored, including Aloha Configuration Center (CFC), NCR Back Office (NBO), Aloha Insight, Command Center (CMC), Loyalty, Stored Value, Mobile Pay, Pulse, Customer Voice, Aloha Restore, Aloha Restaurant Guard and Aloha Update. Database replication for Aloha Insight and NBO is ongoing. If your database has not yet been replicated, please contact your account executive or NCR Support.
Posted Jun 09, 2023 - 13:24 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty, Stored Value, Mobile Pay, Pulse, Customer Voice and Aloha Restore are now online in our new cloud environment. Please read the additional application recovery updates, below.

• Aloha Restaurant Guard is now fully restored.
• Aloha Update (alohaupdates.com; for NCR employee use) is also now online and available. If you have a business-critical product or version not yet available for download, please contact your NCR account executive.

The security and integrity of our systems are top priorities for NCR. Thank you for your patience and continued partnership as we worked to re-establish secure access to the impacted Aloha applications.
Posted May 22, 2023 - 15:41 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty, Stored Value, Mobile Pay and Pulse are now online in our new cloud environment. Please read the additional application recovery updates, below.

• Customer Voice, Aloha Restore and the Aloha Update Suite Installer (package.alohaupdates.com) are now online and available. As a reminder:
o If your company uses the Suite Installer for Aloha application deployment, all existing deployment links will need to be recreated. This is a result of rebuilding Suite Installer in our new environment.
o Aloha Restore historical data is available upon request using the contact info listed below.

• Aloha Restaurant Guard and a version of Aloha Update (alohaupdates.com; for NCR employee use) is on track to go live on Friday, May 19. We will email you confirmation once these applications are successfully restored.
o If you have a business-critical product or version not yet available for download, please contact your NCR account executive.

The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible. We will continue to send updates through the remainder of our restoration process. To read more about our recovery plan updates, view our cybersecurity updates page on ncr.com.
Posted May 18, 2023 - 13:15 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty, Stored Value, Mobile Pay and Pulse are now online in our new cloud environment. Please read additional application recovery updates, below.

• We are targeting tomorrow, Wednesday, May 17, to go live with Aloha Restore, Customer Voice and the Aloha Update Suite Installer (package.alohaupdates.com).
o If your company uses the Suite Installer for Aloha application deployment, all existing deployment links will need to be recreated. This is a result of rebuilding Suite Installer in our new environment.
o Aloha Restore historical data is available upon request using the contact info listed below.
o We will email you confirmation once the applications are live.
• We are also targeting end-of-day Friday, May 19 to restore Restaurant Guard and a version of Aloha Update (alohaupdates.com) for NCR employees to use.
o If you have a business-critical product or version not yet available for download, please contact your NCR account executive.
The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible. We will continue to send updates through the remainder of our restoration process.
Posted May 17, 2023 - 13:31 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty, Stored Value, Mobile Pay and Pulse are now online in our new cloud environment. Please read the additional application recovery updates, below.

Pulse Real-Time API is now enabled and the connection is restored. Data connections should now be flowing to you and your partners (e.g., OpenTable or Up ‘n go).
Posted May 12, 2023 - 18:16 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty, Stored Value and Mobile Pay are now online in our new cloud environment. Please read the additional application recovery updates, below.

• The Pulse Real-Time mobile app is now live, but you may experience slowness in its performance. This is to be expected as we restore this application.
o Please note that historical data may not be immediately visible. This may affect your ability to use tiles or metrics that rely on historical data. We are working on restoring the data, dating back 400 days before the start of the outage, and will notify you when it is available.
o The applets for NCR Back Office, which enable mobile inventory and scheduling, are also now available in the Pulse app.
• We are targeting Thursday, May 11, to enable Pulse Real-Time API.
o This will restore data to you and your partners (e.g., OpenTable or Up ‘n go).
o We are working on restoring the data, dating back 400 days before the start of the outage. We will notify you when the data is available.
• Real-time data is now flowing to the Transaction Document Management (TDM) service on the NCR Commerce Platform.
o Please note: There may be gaps in historical data dating from April 11. We are working to restore historical data as soon as possible. We anticipate data restoration to be complete in the coming weeks.
• We are targeting next week to bring Aloha Update, Customer Voice and Restaurant Guard back online.
Posted May 08, 2023 - 12:53 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office (NBO), Insight, Command Center (CMC), Loyalty and Stored Value are now online in our new cloud environment. Please read the additional application recovery updates, below.

• Mobile Pay, our QR-code and text-based payment solution, is online and available.
• Last week, we emailed to let you know that Aloha Loyalty and Stored Value are now online. Here are some follow-up details:
• Transactions performed with Aloha Loyalty and Stored Value during the outage period are now synced and should be visible in your systems.
• Third-party integrations for Aloha Loyalty and Stored Value are now restored.
• Loyalty/Stored Value alert and report emails continue to send but are still experiencing delivery delays.

The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible. We will continue to send updates through the remainder of our restoration process. To read more about our recovery plan updates, view our cybersecurity updates page on ncr.com.

Regards,

The NCR Hospitality team
Posted May 03, 2023 - 17:33 UTC
Update
Our team continues our 24/7 efforts to execute our recovery plan to re-establish secure access to impacted Aloha applications. Aloha Configuration Center (CFC), NCR Back Office and Command Center (CMC) are now online in our new cloud environment. Please read the additional application recovery updates, below.
• Aloha Insight data continues to poll, and the application is also now live. You may now log into Insight. Here are the details:
• Missing polling data for the outage period will automatically repoll. You do not need to request a repoll of your data.
• Your data may still be polling, resulting in incomplete store reports until the polling process completes.
• Insight alert and report emails will resume sending now. However, email delivery may be initially delayed by up to a few days. You may still pull reports from the portal at this time.
• Aloha Loyalty and Stored Value are also live and available. Here’s what you need to know:
• Transactions performed during the outage period are now syncing. These transactions will not be visible in the systems until the sync is complete.
• Loyalty/Stored Value alert and report emails will resume sending now. However, email delivery may be initially delayed by up to a few days. You may still pull reports from the portal at this time.
The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible. We will continue to send updates through the remainder of our restoration process
Posted Apr 28, 2023 - 13:41 UTC
Update
Our team continues our 24/7 efforts to execute on our recovery plan to re-establish secure access to impacted Aloha applications. Please know we have heard your feedback and developed this plan based on the priority items we believe will help you get back to business as usual as soon as possible.

Here is an overview of our recovery plan progress:

The Command Center (CMC) application is fully restored, and online ordering has resumed for the subset of CMC customers impacted by the outage.
NCR Back Office is now live in our new cloud environment and ready for normal in-store procedures, such as data imports, adding employees, etc. Here are some details:
Each site’s sales and clock-in/out time information from the outage period is stored on the back-of-house controller. You may now, at any time, import this data to begin syncing in the live environment. For larger customers, please note data imports should be staggered to avoid latency.
You may also begin adding employee records, inventory postings, shift edits, and deposits dating back to April 10.
Customers with data replication feeds in place: Please see the email sent to your technical contacts this past Wednesday. We will send out information on when replication feeds will be enabled in the near future.
The NCR Back Office applet in Pulse will not come online until the Pulse mobile app is restored.
We are making progress configuring the Insight and Pulse cloud environments. We emailed replication configuration instructions to technical customer contacts with replication feeds for Insight this past Wednesday.
Aloha Configuration Center (CFC) plans are also progressing. Please ensure you are documenting all offline changes made since April 10 to re-enter once the application is back online. Please see the email sent earlier today from NCR Hospitality for more information and important actions required to ensure a successful restoration.
We are also executing on our recovery plans for all other impacted applications.

The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible and will continue to share updates as the plan progresses.
Posted Apr 24, 2023 - 17:57 UTC
Update
Our team continues our 24/7 efforts to execute on our recovery plan to re-establish secure access to impacted Aloha applications. Please know we have heard your feedback and developed this plan based on the priority items we believe will help you get back to business as usual as soon as possible. We previously shared our goal to bring all impacted applications back online this week, and while that remains our target, we hope to restore your services sooner.

Here is an overview of our recovery plan progress:

- We aim to fully restore the Command Center (CMC) application in the next 24 hours. Once this is complete, online ordering will resume immediately for the subset of CMC customers impacted by the outage. There are no actions required for online orders to begin flowing once this capability is restored.

- We are also standing up Insight and NCR Back Office cloud environments. Once these applications are available, Insight, NBO and Pulse will require time to upload and process your restaurant data. Impacted customers can expect an email with detailed replication configuration instructions in the coming days.

- Aloha Configuration Center (CFC) plans are also progressing. Your technical contacts can expect more information on how to replicate audit logs in the coming days.

- We are also executing on our recovery plans for all other impacted applications.

The security and integrity of our systems are top priorities for NCR. We are working around the clock to re-establish secure access to the impacted Aloha applications as quickly as possible. We will have further updates on the timeline for rebuilding this new environment and we continue to target this week to bring these applications back online.

Regards,
The NCR Hospitality Team
Posted Apr 18, 2023 - 20:35 UTC
Update
At NCR Corporation, we are committed to not only supporting our customers’ day-to-day business operations, but also to being there for our customers when they need us most. To that end, we would like to update you on the recent incident we experienced that is impacting some of our commerce customers and share details on our response, temporary workarounds and our path to recovery.

What Happened

On April 13, NCR determined that a single data center outage that is impacting some functionality for a subset of its commerce customers was caused by a cyber ransomware incident. Upon such determination, NCR immediately started contacting customers, enacted its cybersecurity protocol and engaged outside experts to contain the incident and begin the recovery process. The investigation into the incident includes NCR experts, external forensic cybersecurity experts and federal law enforcement.

We believe this incident is limited to specific functionality in Aloha cloud-based services and Counterpoint. At this time, our ongoing investigation also indicates that no customer systems or networks are involved. None of our ATM, digital banking, payments or other retail products are processed at this data center.

Supporting Our Customers

While in-restaurant purchases and transactions continue to process to serve your customers, we understand the role the currently impacted applications play in supporting your business processes. We are working around the clock to restore them as quickly as we can. In parallel, we have local workarounds in place while restoration is underway and have established dedicated teams to support our impacted customers’ operations.

If you have additional questions or require further support, we encourage you to contact NCR Support or your Account Representative. We are committed to finding solutions and working alongside you as we move forward together.

Our Path to Recovery

We are committed to re-establishing secure access to the impacted Aloha and Counterpoint applications as quickly as possible. We are restoring impacted applications in a new secure environment. We will have further updates on the timeline for rebuilding this new environment, and we are targeting this week to bring these applications back online. We will also be contacting customers with a few key steps to access our new environment.
Posted Apr 17, 2023 - 19:39 UTC
Update
As a valued customer of NCR Corporation, we are reaching out with additional information about a single data center outage that is impacting a limited number of ancillary Aloha applications for a subset of our hospitality customers. On April 13, we confirmed that the outage was the result of a ransomware incident. Immediately upon discovering this development we began contacting customers, engaged third-party cybersecurity experts and launched an investigation. Law enforcement has also been notified.

Please rest assured that we have a clear path to recovery and we are executing against it. We are working around the clock to restore full service for our customers. In addition, we are providing our customers with dedicated assistance and workarounds to support their operations as we work toward full restoration. Restaurants impacted are still able to serve their customers. Only specific functionality is impaired. There is no impact to payment applications or on-premises systems.

The security and integrity of our systems is a top priority for NCR. We will continue to keep you updated with pertinent information and will let you know as soon as impacted services have been fully restored. In the meantime, please contact NCR Support or your Account Representative if you have any questions or need additional support.

We regret any concern this incident may cause. We appreciate your patience and support as we work toward full restoration.
Posted Apr 15, 2023 - 02:38 UTC
Update
We are continuing to investigate this issue.
Posted Apr 14, 2023 - 01:26 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 18:58 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 17:27 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 16:36 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 15:33 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 14:24 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 13:20 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 12:21 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 11:19 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 10:18 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 09:18 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 08:14 UTC
Update
We are aware of the outage and we are investigating it further
Posted Apr 13, 2023 - 07:07 UTC
Update
We are continuing to investigate the issue
Posted Apr 12, 2023 - 22:43 UTC
Update
We are continuing to investigate the issue
Posted Apr 12, 2023 - 22:10 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 21:31 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 21:01 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 20:32 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 20:02 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 19:32 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 18:56 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 16:58 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 14:33 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 13:34 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 12:54 UTC
Update
We are continuing to investigate this issue.
Posted Apr 12, 2023 - 11:04 UTC
Investigating
We are currently investigating this issue.
Posted Apr 12, 2023 - 10:57 UTC
This incident affected: North America (Aloha Insight-US, Aloha Keyless Licensing-US, Aloha Loyalty-US, Aloha Stored Value-US, Aloha Update-US, Command Center-US, Command Center WebOrder-US, Configuration Center-US, Configuration Center API-US, Customer Voice-US, Mobile Pay API-US, NCR Back Office (NBO)-US, Pulse Realtime-US, Pulse Realtime API-US, Radiant POS Management-US, Restaurant Guard Audit-US), Asia Pacific (Online Ordering Classic-APAC, Online Ordering API-APAC), and Europe (Online Ordering Classic-EU, Online Ordering API-EU).