Everything you need to know about issues with EC2 T2 instance not accessible and alternative solutions

by INVOKE Team


Posted on Oct 03, 2018 at 12:00 AM


Amazon EC2 schedulers

AWS EC2 instances are one of the widely used resources available in AWS ecosystem. Amazon Elastic Compute Cloud (Amazon EC2 ) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers.

Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use.

There are 5 types of AWS EC2 instances available, which serves different needs:

  1. General Purpose (T2, T3, etc..)
  2. Compute Optimized
  3. Memory Optimized
  4. Accelerated Computing
  5. Storage Optimized

Pricing for these instances differs based on 4 dimensions:

  1. Type of rental ( ex: On-Demand, Reserved Instance)
  2. Model of the instance (ex: t2.micro, m5.large etc.,)
  3. Type of operating system (ex: Windows, Linux etc.,)
  4. Region (ex: US East (N. Virginia), EU (Frankfurt) etc.,)

Out of all types of EC2 instances, behaviour of T2 type instances are little bit tricky in terms of system functionality. When users launch T2 (standard) type of servers, AWS imposes certain restriction on available CPU. Because T2 types are supported in free tier and widely used by new users, understanding how this CPU credit allocations works and impacts your EC2 instance performance will make AWS users life easy in deploying correct type of applications on these instances during testing.

T2 instances are designed to provide a baseline level of CPU performance with the ability to burst to a higher level when required by your workload. T2 instances are well suited for a wide range of general-purpose applications like microservices, low-latency interactive applications, small and medium databases, virtual desktops, development, build, and stage environments, code repositories, and product prototypes.

Baseline level of CPU performance: The percentage of single core performance of the underlying physical CPU allocated to the instance. For example, a t2.small instance has access to 20% of a single core of an Intel Xeon processor running at 2.5 GHz (up to 3.3 GHz in Turbo mode).

Burstable performance: This means allocating more CPU than baseline performance. AWS uses “CPU Credits” to calculate how much more capacity it can assign to this EC2 instance.

What is CPU Credit? It is equal to one vCPU running at 100% utilization for one minute. Other combinations of number of vCPUs, utilization, and time can also equate to one CPU credit. For example, one CPU credit is equal to one vCPU running at 50% utilization for two minutes, or two vCPUs running at 25% utilization for two minutes.

AWS allocates 2 types of CPU Credits:

    • Launch Credits:

    Credits allocated when you start an instance. T2 Standard instances get 30 launch credits per vCPU at launch or start.
    • Earned Credits:

    The credits earned When a T2 Standard EC2 instance is in a running state.

How to earn CPU Credit? Each T2 instance continuously earns (at a millisecond-level resolution) a set rate of credits per hour, depending on the instance size. For example, a t2.nano earns three credits per hour, while a t2.small earns 12 credits per hour. If a T2 instance uses fewer CPU resources than it is required for baseline performance (such as when it is idle), the unspent credits are accrued in the CPU credit balance.

Credit Accrual Limit: While earned credits never expire on a running instance, there is a limit to the number of earned credits an instance can accrue. The limit is determined by the credit balance limit. Once the limit is reached, any new credits that are earned are discarded. Note that “Launch Credits” do not count towards CPU credit balance limit.

CPU credits on a running instance do not expire, however, the balance does not persist between EC2 instance stops and start.

Another complex math to understand with accrued CPU credits and expiry is, constantly running your EC2 instance above baseline usage will have 2 folded effects 1) burning out CPU credits 2) no accrual because of above limit consumption. When AWS observes that EC2 instance CPU is above allowed limit, it will limit allocation to baseline performance.

What is the problem? When you rent t2.micro instance, 10% baseline of a one core capacity will be allocated. Users who deploy continuous CPU consumption applications will be throttled and experience very bad performance (sometimes the EC2 instance could become un-accessible too), if application needs more than baseline performance.

For example, the application running on t2.micro requires 12% of CPU and have no CPU credits, AWS will limit CPU capacity to 10%, which slows down application performance as well as if anyone trying to access the box using other mechanisms (like SSH) while CPU is busy, users will experience availability issues.

EC2 Problem - 100% CPU Usage, No SSH connectivity thread on AWS support forum is great example of this issue. Due to an issue with instance, cpu usage was 100% always, which resulted in having no resources to process SSH connection request.

Teams who launch T2 type of EC2 instances need to pay attention to application resource usage requirements and choose appropriate types to have better hosting experience.

Sometimes it would be other configuration issues like open security groups could also cause drain in cpu resource which will keep the instance busy and put it into completely unavailable state. Here is another question posted on AWS forum whose EC2 instance is unavailable state and suggestion by support to restrict the ports.

What is the solution? There are different solutions based on requirements:

  1. If application usage is constantly below baseline performance, then T2 is enough, make sure to secure the ports and assign appropriate security groups. This will avoid some bad guys using instance for their purpose and drain the resource.
  2. Opt for T2 Unlimited option, these EC2 instance can sustain high CPU performance for any period of time whenever required. The hourly T2 instance price automatically covers all interim spikes in usage if the average CPU utilization of the instance is at or below the baseline over a rolling 24-hour period or the instance lifetime, whichever is shorter. If the instance runs at higher CPU utilization for a prolonged period, it can do so for a flat additional rate per vCPU-hour.
  3. Use T3 instances: T3 instances are the next generation burstable general-purpose instance type that provide a baseline level of CPU performance with the ability to burst CPU usage at any time for as long as required.
  4. Buy consistent capacity instances like M5 instances or other non t2 type instances

Have any questions on cloud hosting? Talk to us, we could help you.

Using schedulers to save cloud hosting costs? you should read the blog AWS EC2 Schedulers - not good enough tool to reduce AWS bill

READY TO SAVE ON CLOUD COSTS?  Get Started for free