Engine Yard

February 2017

Degraded performance for EBS in us-east-1
At this point we are considering this issue resolved and normal functionality has been restored. If you have any questions or want to discuss, please file a ticket and we are happy to assist
Feb 23, 20:17-21:34 UTC
StatusCheck Alerts
This incident has been resolved.
Feb 16, 04:30 - Feb 17, 02:50 UTC

January 2017

[Scheduled] Dashboard Maintenance
Scheduled maintenance has been completed.
Jan 8, 05:00-05:25 UTC

December 2016

No incidents reported for this month.

November 2016

[Scheduled] November 19th, 2016 Platform Maintenance
The scheduled maintenance has been completed.
Nov 21, 18:42 UTC

October 2016

[Scheduled] Engine Yard Cloud Maintenance
The scheduled maintenance has been completed.
Oct 31, 08:00-11:00 UTC
Cloud.engineyard.com Reporting Down
Upstream providers have pronounced this issue resolved as of 22:17 UTC. As we have seen no further reports of trouble since that time we are considering this issue resolved.
Oct 21, 16:37 - Oct 22, 01:11 UTC
Cloud Dashboard currently not working
6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.
Oct 21, 11:40-14:02 UTC
Deployment Failures
Deployments are working once more.
Oct 4, 17:35-17:55 UTC
Environment Dashboard and other features not working
This incident has been resolved.
Oct 4, 03:30-04:32 UTC

September 2016

No incidents reported for this month.

August 2016

Network slowness
We believe the issue is mostly resolved for our customers at this point. If you are encountering any lingering issues please let us know.
Aug 29, 23:15 - Aug 30, 04:46 UTC
Environments with incorrect app master label
The issues related to application instance roles should now be resolved. If you do experience any issues at this point feel free to submit a support ticket.
Aug 24, 22:57 - Aug 25, 00:46 UTC
ImageMagick vulnerability CVE-2016-3714
We have released =media-gfx/imagemagick-6.7.8.7-r2 and =media-gfx/imagemagick-6.9.0.3-r2 for our Stable V4 stack to resolve both cve vulnerabilities.
May 4, 15:16 - Aug 18, 16:39 UTC
Dashboard not responding
This incident has been resolved.
Aug 15, 17:04-21:47 UTC
Unexpected Platform Maintenance due to underlying hardware
At 6:20 AM PDT we had unexpected maintenance due to underlying hardware from our infrastructure provider. The maintenance was resolved at 6:35 AM PDT
Aug 15, 14:58 UTC
Deploy hooks issue
A fix has now been issued for this issue, so deploy hooks now execute successfully on deploys. Should you see any further issues please submit a ticket at https://support.cloud.engineyard.com.
Aug 12, 07:29-11:08 UTC

July 2016

AWS Services Increased EC2 Error Rates and ELB Provisioning Times
AWS has reported that they have Resolved the increased EC2 API Error Rates, and ELB Provisioning times in N. Virginia Region.
Jul 30, 01:59-02:22 UTC
http://cloud.engineyard.com is down and being investigated
We have resolved the issue, and the platform is working normally again.
Jul 20, 02:28-02:38 UTC
Problems with Booting Instances and Clicking Apply
The issue has been fixed. Clicking Apply and booting new instances are working again.
Jul 11, 03:54-04:12 UTC

June 2016

Platform task failures
The platform issue has been solved. You can now boot and terminate instances successfully. If you see any other issues, please submit a ticket.
Jun 22, 13:11 - Jun 23, 04:39 UTC
Provisioning Issues in US East 1
This has been resolved
Jun 9, 20:36-21:14 UTC
AWS Issues on Sydney Region
AWS issues in the Sydney Region have been reported as resolved per AWS http://status.aws.amazon.com/ . If you have any outstanding issues please open a support ticket
Jun 5, 05:59-19:37 UTC

May 2016

Can't Edit Environment Settings If Using EBS Encryption
Editing environment settings now works if using EBS encryption. If you see any issue, please open a support ticket.
May 6, 02:01 - May 12, 01:37 UTC

April 2016

Customers unable to submit support tickets
The issue affecting ticket submission has now been identified and resolved and tickets can again be submitted via https://support.cloud.engineyard.com.
Apr 29, 10:32-14:57 UTC
Trial Account Process
Trial signups should be working properly once more
Apr 22, 19:59-20:54 UTC
US East 1 (Virginia) having network issues
This incident has been resolved.
Apr 21, 21:35-22:23 UTC
Platform task failures
Permanent solution has been put in place. All provisioning operations work normally. Issue is resolved. For any further issues or assistance please submit a ticket at https://support.cloud.engineyard.com.
Apr 13, 11:15-14:19 UTC

March 2016

AWS US-East-1 connectivity issues
As of 2:02 AM PST AWS have marked this issue as resolved and are reporting the service to be running as normal.
Mar 10, 09:02-10:16 UTC

February 2016

No incidents reported for this month.

January 2016

Upcoming Maintenance Friday January 29th, 2016
We have completed the scheduled maintenance of our account and billing systems and all impacted services are fully restored.
Jan 26, 00:45 - Jan 29, 17:58 UTC

December 2015

Zendesk outage
Zendesk has reported that all services have been restored at 12:45pm PST.
Dec 23, 14:43 - Dec 24, 03:27 UTC
Dashboard Unavailable for Scheduled Maintenance
This incident has been resolved.
Dec 22, 03:57-05:12 UTC
Zendesk outage
Zendesk have resolved the incident and as such our support site should be fully functional again.
Dec 17, 12:49-15:21 UTC

November 2015

Inability to boot t2 type instances on stack v4-2.0.97
Ability to boot t2, c4, and m4 type instances has been restored. The incident is resolved.
Oct 31, 16:31 - Nov 3, 10:58 UTC

October 2015

Uptime monitoring not reporting to the dashboard
We have fixed this on our side after identifying the change in provider payload
Oct 19, 11:08 - Oct 29, 20:35 UTC
Dashboard Currently Not Working
Our dashboard is working properly and is currently stable.
Oct 14, 19:17-23:14 UTC
Composer functionality
We have resolved this issue via updating curl. Please update to the newest stack version to resolve this issue.
Jul 17, 18:56 - Oct 14, 19:15 UTC

September 2015

No incidents reported for this month.

August 2015

Chef Recipes Error When Upgrading
The issue with ntp failing to install during an upgrade has been solved. You should not see any errors when you click Upgrade.
Aug 11, 08:31 - Aug 12, 03:42 UTC
Deployments and Chef runs not working due to AWS S3 errors
AWS has reported that the error has been fixed and AWS S3 service is now operating normally. Customers may now deploy, or click Apply on the environment dashboard.
Aug 10, 08:13-11:00 UTC
Detached Instances
Previously Detached instances are once again detached
Aug 6, 00:23-01:25 UTC

July 2015

Database Snapshots When Booting New Instances
The database snapshot selection issue has now been resolved, meaning database instances can be booted from snapshots without issue. If you require any further assistance please submit a ticket at https://support.cloud.engineyard.com.
Jul 30, 06:36 - Jul 31, 13:59 UTC
Elastic Load Balancers failing to provision
Issues regarding provisioing of ELBs for environments have been resolved. In case of needing any futher assistance please submit a ticket to Support.
Jul 13, 10:47 - Jul 14, 11:36 UTC

June 2015

Applies and Deploys failing
We have deployed a fix for this. Deployment and Applies should be working as expected. If you are still having issues please feel free to open a support ticket for further investigation.
Jun 22, 16:14-16:27 UTC
CloudFlare DNS issues
CloudFlare have implemented a fix for the DNS issues: https://www.cloudflarestatus.com. If you see further issues or require assistance please contact support via https://support.cloud.engineyard.com.
Jun 12, 09:47-10:10 UTC
Timeouts on ui.engineyard.com
The timeout issue has now been identified and resolved and ui.engineyard.com is now accessible again. Should you see any further issues please submit a support ticket at https://support.cloud.engineyard.com.
Jun 6, 09:00-09:56 UTC
DB backups in AWS EU (Frankfurt) Region
The problem has been resolved with our latest stack upgrades.
May 12, 14:46 - Jun 4, 17:17 UTC
Ticketing system
Our ticketing provider has resolved the issue and service has returned to normal.
Jun 3, 15:22-15:44 UTC

May 2015

No incidents reported for this month.

April 2015

Intermittent Chef errors when clicking Apply
The issue with chef errors when adding or removing instances has been resolved.
Apr 30, 13:29-16:36 UTC
Trial customers unable to boot into a Region
This has been resolved and new accounts are able to boot without issue.
Apr 22, 14:22-16:18 UTC

March 2015

OpenSSL vulnerabilities reported
We have provided OpenSSL patches through our standard stack upgrade path. For notes on this process, please visit: https://support.cloud.engineyard.com/entries/89894297-Engine-Yard-Release-Notes-March-2015#20150320
Mar 18, 18:32 - Mar 30, 20:40 UTC
GitHub connectivity issues
GitHub are reporting that the connectivity issues have been resolved, so no further deploy issues should be seen at Engine Yard, although the DDoS attack continues. Please continue to monitor https://status.github.com for updates to the situation and submit a ticket at https://support.cloud.engineyard.com if you have any further questions or issues.
Mar 27, 09:22-13:03 UTC
Upgrade failures on non-Ruby environments
A Hotfix has been released for non-Ruby environments. Please feel free to update your environments at this time.
Mar 18, 12:12-19:27 UTC
Emails not getting sent
The issue has been fixed. You should now be receiving emails from the platform.
Mar 16, 05:25-06:50 UTC
Loss of network connectivity to Engine Yard on Terremark
We are currently awaiting official analysis of the cause for this outage. Affected customers will have their related support tickets updated once this has been issued by Terremark.
Mar 13, 05:08 - Mar 14, 08:23 UTC
Xen Security Advisory for Engine Yard Cloud
This morning, AWS has confirmed the completion of the reboots for all affected instances. With the finalization of of this reboot process, we will be marking this event as resolved. It was our pleasure to support you during this time and welcome any support tickets with questions or comments you may have!
Feb 27, 18:20 - Mar 10, 14:09 UTC
Scheduled EY Cloud Dashboard maintenance - 9th March 2015
Our scheduled dashboard maintenance was carried out successfully, and has now concluded.
Mar 6, 12:23 - Mar 10, 05:57 UTC
Termination, Provisioning Errors
We have identified the issue with those processes and have resolved the issue.
Mar 6, 18:59-20:37 UTC

February 2015

Applies, Deploys, and Upgrades behaving slowly
We have found and resolved the underlying problem where applies, deploys and upgrades weren't working as expected.
Feb 24, 17:33-17:45 UTC
Applies and Deploys failing
This has been resolved by our engineers. If you experience any further issues or need assistance, please open a support ticket.
Feb 11, 20:19 - Feb 12, 14:12 UTC
Ghost Vulnerability CVE-2015-0235
Our final patch for the Ghost Vulnerability CVE-2015-0235 has been released and announced for our stable-v2 (2009a) stack. Please see our Security Known Issue article for updated information and instructions on how to apply the patch: https://support.cloud.engineyard.com/entries/105852596-GHOST-CVE-2015-0235-vulnerability
Jan 27, 22:35 - Feb 10, 21:50 UTC
Ticketing system
The issue with the ticketing system is now resolved.
Feb 2, 12:54-14:45 UTC

January 2015

Ticketing system
Our ticketing system is now back online. You can file a support ticket here: https://support.cloud.engineyard.com/tickets/new
Jan 30, 18:58-19:46 UTC
False site down alerts
We are marking this as resolved, but the issues may arise again. If you receive any additional alerts from our platform, don't hesitate to open a ticket for our support team to investigate.
Jan 22, 17:05-20:19 UTC

December 2014

Stack Upgrade causing issue
The Hotfix has been released and available for the previously identified versions.
Dec 12, 19:18-20:04 UTC
Networking issues in AWS Tokyo region (AP-NORTHEAST-1)
AWS have officially marked the connectivity issue in the Tokyo region as resolved, so no further problems should be seen. Should you require further support please submit a ticket at https://support.cloud.engineyard.com.
Dec 11, 13:19-14:04 UTC
[Scheduled] Minor systems maintenance
The scheduled maintenance has been completed.
Dec 10, 18:00-18:15 UTC
Platform functionality issues
The platform issues have been identified and resolved and deploys and Chef runs should now complete without issue. If you required further assistance please submit a ticket.
Dec 10, 16:25-16:53 UTC
Networking issues in AWS Ireland region (EU-WEST-1)
AWS have officially marked the connectivity issue in the Ireland region as resolved, so no further problems should be seen. Should you require further support please submit a ticket at https://support.cloud.engineyard.com.
Dec 10, 11:09-11:39 UTC
DNS Resolution for some sites
The DNSimple issues have been resolved.
Dec 1, 20:01 - Dec 2, 09:34 UTC

November 2014

Deploy and Chef Failures
The issue has been fixed. You can now deploy and click Apply on your environments.
Nov 27, 04:18-04:36 UTC
Downtime for customers using ELBs
AWS has identified the ELB issues as being linked to Cloudfront DNS errors reported here: http://status.aws.amazon.com/. We are marking this as Resolved as alerts have ceased environments are stable. Please open a support ticket if you are seeing further issues or if you have additional questions.
Nov 27, 01:50-02:54 UTC
Deploy and Chef Failures
The issues with deploys and chef runs have been fixed. You can now deploy and click Apply.
Nov 26, 04:56-06:01 UTC
Engine Yard Dashboard Loading Slowly
The dashboard (cloud.engineyard.com) is working once more.
Nov 22, 01:07-01:35 UTC

October 2014

Networking issues in AWS Sydney region (ap-southeast-2)
AWS have officially marked the connectivity issue in the Sydney region as resolved, so no further problems should be seen. Should you require further support please submit a ticket at https://support.cloud.engineyard.com.
Oct 23, 04:42-08:21 UTC
Booting New Instances
We have seen this resolved. If you continue to see these issues please open a ticket.
Oct 16, 23:12 - Oct 17, 01:05 UTC
Nextgen internal server error
The 'New UI' issues have now been resolved and the dashboard is now functioning correctly again.
Oct 6, 10:02-11:33 UTC
AWS Maintenance Event
The AWS mandatory maintenance window is complete and a statement has been released: http://aws.amazon.com/blogs/aws/ec2-maintenance-update-2/
Sep 25, 04:51 - Oct 1, 17:28 UTC

September 2014

Applies failing on Engine Yard Cloud
The issue with Engine Yard Cloud which affected running "Apply" and deploys has been resolved. If you see any further issues, please open a Support ticket.
Sep 28, 01:21-03:14 UTC

August 2014

Redirect loop in ticketing system
We have identified the problem and a resolution is in place. Everything should be working as expected.
Aug 26, 22:26-22:38 UTC
Maintenance continues on Engine Yard Cloud Dash
Maintenance is now complete, and our cloud dashboard is available once more.
Aug 21, 06:39-07:56 UTC
[Scheduled] Engine Yard Cloud Scheduled Maintenance
The scheduled maintenance has been completed.
Aug 21, 05:00-06:00 UTC
Dashboard not responding as exptected
This issue has been resolved and our dashboards should be functioning as normal
Aug 14, 21:18-21:22 UTC
Aug 6th, 2014 - Cloud Dashboard outage
The issue responsible for the inaccessibility of the Cloud dashboard has now been identified and resolved, and as such the dashboard is now back online. If you continue to experience any issues, please contact support by submitting a ticket at support.cloud.engineyard.com.
Aug 6, 11:16-12:23 UTC
AWS Networking issues in US-East-1
AWS reports the issue has been resolved and the service is operating normally.
Aug 2, 17:39-18:23 UTC

July 2014

AWS EU Region Networking issues
AWS is reporting that service has returned to normal. We are working on impacted customer environments currently and will provide updates via tickets. If you need assistance, please open a support ticket!
Jul 24, 16:28-16:45 UTC
Email Alerts
We've identified and resolved the issue. Environments with email alerts configured are now receiving dashboard alerts via email.
Jul 7, 05:58-08:05 UTC

June 2014

Engine Yard email queuing
We resolved an issue where some customer facing emails were queuing since June 18th. You may notice an increase in emails from Engine Yard today, as the queue is cleared. If you have any questions or concerns, please open a support ticket and we will be happy to assist!
Jun 20, 13:01 UTC
Security Status
We confirmed a condition where an internal application was exposed to the internet. This application contains customer AWS login name, without any credentials or keys, contact email address, estimated monthly spend with Engine Yard and url listings for Github repositories. As no credentials were exposed, private repositories could not be accessed. This issue has been resolved by requiring internal authentication before execution. All internal applications are now be required to follow this protocol. No sensitive customer information was disclosed as a result of this condition. As no sensitive customer information was involved, this condition was not detected in our processes to ensure that sensitive data are protected correctly. If you have any concerns or questions about any possible effect from this condition on your systems, please submit a ticket through our support site and we will respond to you promptly. The safety and security of your systems and data are paramount to all of us at Engine Yard and I appreciate the trust that you continue to place with us. Jeff Reich Chief Security Officer
Jun 17, 23:59 UTC
Engine Yard Ticketing System
All ticketing systems are operational and been cleared by our vendor.
Jun 13, 16:19-17:37 UTC
Login errors
We received reports of errors when logging into login.engineyard.com. These have been identified and fixed by our engineers. If you experience any further issues, please open a support ticket: https://support.cloud.engineyard.com/tickets/new
Jun 10, 17:48 UTC

May 2014

May 29, 2014 - Cloud Dashboard outage
Our EY cloud dashboard was unavailable for about 15 minutes around 11:10PM 29th May PDT (6:10AM 30th May UTC). This issue was due to a database master becoming unresponsive - we're reviewing the incident to see how we can improve recovery times in future.
May 30, 07:39 UTC
AWS - Single AZ issues in Region US-West-1 (US West (Northern California) Region)
AWS continues to restore power to affected instances. All alerting sites have been restored. If you experience any further issues, please open a support ticket: https://support.cloud.engineyard.com/tickets/new
May 27, 01:22-02:47 UTC

April 2014

AWS Instance Connectivity issue in US-WEST-2
AWS reports that the issue has been resolved and the service is operating normally. If you have any further issues or need additional assistance, please open a ticket so we can help!
Apr 30, 19:04-21:00 UTC
Dashboard Tasks Not Finishing
This incident has been resolved.
Apr 25, 00:50-02:43 UTC
April 7, 2014 - OpenSSL security vulnerability
Engine Yard stacks on Gentoo are not currently running a version of OpenSSL greater than 1.0.0. which removes it from the scope of this vulnerability. The current issue facing our customers with this vulnerability presents itself with the use of AWS ELB’s and our Ubuntu offerings, which do run a version of OpenSSL that is vulnerable in CVE-2014-0160. AWS has updated their ELB offerings to address the vulnerability: https://aws.amazon.com/security/security-bulletins/aws-services-updated-to-address-openssl-vulnerability/ Contrary to the statement in the link, all ELBs in use by Engine Yard, and in the Engine Yard pool of resources, have been updated in all regions, including US East. At this time, if you have been using an ELB in your environment, it is recommended that you rotate your SSL certificates. If running on our Ubuntu stack, open a support ticket and Engine Yard Support can verify that your environment is secure.
Apr 8, 03:45-21:10 UTC

March 2014

Failures in ap-southeast-2 (Sydney) AWS region
Issues affecting the Sydney AWS region (ap-southeast-2) are now resolved and environment configurations and application deployments are no longer affected. If you have any further issues please submit a ticket to Engine Yard support at https://support.cloud.engineyard.com.
Mar 26, 11:47 - Mar 28, 08:30 UTC
Increased API error rates in the US-EAST-1 region
AWS has resolved the issue with the API errors.
Mar 21, 06:30-08:08 UTC
Installation failures for npm
The cert was rolled back by npm, which resolves the SSL errors faced earlier. If you have any further issues, please open a support ticket.
Feb 28, 12:57 - Mar 2, 20:22 UTC

February 2014

Namecheap and DNS is reporting issues
Update from Namecheap: We have successfully mitigated most of the DDoS attack and services are returning to normal. Services may be somewhat degraded while everything returns to normal over the next 60 minutes. Thank you for your understanding and patience during this difficult time.
Feb 20, 17:59-21:49 UTC
Issues with packagist.org and getcomposer.org affecting PHP deployments
Issues affecting packagist.org and getcomposer.org and have now been resolved and they should no longer impact PHP application deployments through our platform. If you experience any further issues, please open a ticket at https://support.cloud.engineyard.com.
Feb 11, 11:13-13:16 UTC
February 7th 2014 - Cloud Dashboard timeouts
The Cloud dashboard timeout issue has now been identified and resolved. If you see any further issues please submit a support ticket at https://support.cloud.engineyard.com.
Feb 7, 12:39-14:00 UTC
AWS S3 Errors
The S3 issue was resolved at 7:23 PM PST. If you still see any issue with your deployments, please open a ticket.
Feb 7, 00:58-04:07 UTC
Issues provisioning in AWS Region S America
This issue has been resolved internally by AWS.
Feb 5, 17:27-23:24 UTC
February 4th Ticket System Issues
February 4th, 2014 10:19 AM PST (6:19 PM UTC) - The ticketing system is working correctly once more.
Feb 4, 17:54-18:19 UTC
Freenode IRC issues
The freenode IRC issue has now been mainly resolved, and freenode recommend updating configurations to use the round-robin hostname chat.freenode.net. For any specific or ongoing issues please see @freenodestaff.
Feb 3, 09:20 - Feb 4, 08:43 UTC

January 2014

Dashboard issues
Issues relating to deployments and instance provisioning have now been resolved. If you still see issues please submit a ticket at https://support.cloud.engineyard.com/
Jan 30, 09:40-11:13 UTC
AU sites intermittently alerting down
Issue identified and resolved as per AWS: We experienced an issue with an Internet provider outside of our network earlier today, which may have impacted connectivity between some customers networks and the AP-SOUTHEAST-2 Region. Connectivity to instances and services within the region was not impacted by the event. Our team confirmed that this issue has been resolved.
Jan 28, 15:09-15:48 UTC
EY Cloud Dash sign in results in "You retried too quickly"
Engineers have resolved the sign-in issue. If you notice any further issues or need assistance, please open a support ticket: https://support.cloud.engineyard.com/tickets/new
Jan 22, 13:33-14:51 UTC
01/14/2014 Failure to Deploy
January 14th, 2014 12:36 PM PST(8:36 PM UTC) - We have resolved the issue and deployments are working once more.
Jan 14, 20:12-20:37 UTC
Jan 13, 2014 - Worldnic DNS Issues
Worldnic's DNS servers are responding again and we see that the sites are back up. If you still see any issues, please open a ticket.
Jan 13, 08:02-08:08 UTC
January 9th, 2014 Add-ons Unavailable
January 9th, 2014 2:50 PM PST (10:50 PM UTC) -Add-on services are available.
Jan 9, 22:33-22:51 UTC
01/02/2014 Billing Pages
January 2nd, 2014 11:105am PST (07:15pm UTC)- The billing pages are accessible once more.
Jan 2, 18:52-19:26 UTC

December 2013

12/19/2013 Dashboard and API currently unavailable
December 19, 2013 07:47pm PDT (03:47am UTC) - All Services have been restored and functioning at this point.
Dec 20, 01:15-03:46 UTC
Issue adding and terminating instances
The queue is now clear and all requests are being served as expected. If you have any further issues, please open a support ticket. https://support.cloud.engineyard.com/entries/33626396-December-19-2013-Issues-adding-and-terminating-instances
Dec 19, 14:10-15:30 UTC

November 2013

No incidents reported for this month.