1 (866) 866-2320 Resources Events Blog

Inside Amazon’s AMI Black Box: Not What You Would Have Expected

Blog

Inside Amazon’s AMI Black Box: Not What You Would Have Expected


 

by Sasha Gilenson

Inside Amazon's AMI Black Box: Not What You Would Have ExpectedThink you know your cloud instances? Think again! Learn what surprising discoveries we found in the Amazon Instance library by looking inside.

The standard method of operation in the cloud is to spin off new servers based on virtual machine images. This can mean launching a server with an image that is "close" to what you want, like from the Amazon selection. Amazon offers a list of Amazon Machine Instances (AMI) which are used to create new server instances within the Amazon Elastic Compute Cloud (EC2). 

Amazon makes it easy to setup new instances, offering a variety of existing images to choose from - Public Community, built by Amazon or by other vendors. You can find here Windows based images, various flavors of Linux, LAMPStack, JBoss, Microsoft stack including IIS and MS SQL server etc. 

Sound simple? Wait! It's actually anything but that. Information on the content of the AMIs is limited to a text description, typically listing key components installed and their versions. So using available images can be quite frustrating, especially without using tools to help manage them.

One of the big challenges for leveraging these available instances is the lack of visibility into the AMIs. Images are opaque, and it's hard to tell what's in an image. Even using cloud management tools like RightScale provides or Amazon Console, it's still not easy to explore and to figure out what's installed and how it's configured. 

As a result many Amazon users might simply prefer to create their own images that they can control, spending time and resources on setup of the AMIs and their subsequent management.

What Did We Do?

Inside Amazon's AMI Black Box: Not What You Would Have Expected

We took several different Amazon Machine Instances (from AWS, Public Community, and from other vendors), and spun off EC2 instances based on these AMIs. Then we scanned each of these instances using Evolven change and configuration software , gathering all configuration information about each instance, and to see what useful information we could get. This was kind of a reverse engineering of the AMI, providing visibility into the configuration details. Here is a list of AMIs we scanned, for demonstrating the need for visibility:

Type of AMIs 32/64-bit Description
Amazon AMIs 32 Microsoft Windows 2008 R1 SP2 Datacenter edition
Amazon AMIs 64 Microsoft Windows 2008 R2 SP1 Datacenter edition
Amazon AMIs 64 Microsoft Windows Server 2008 R2 SP1 Datacenter edition, Microsoft SQLServer 2008 Express, Internet Information Services 7, ASP.NET 3.5.
Amazon AMIs 64 Microsoft Windows Server 2008 R2 SP1 Datacenter,  Microsoft SQL Server 2008 R2 Web Edition.
Amazon AMIs 32 Red Hat Enterprise Linux 6.2
Amazon AMIs 64 Red Hat Enterprise Linux 6.2
Vendor AMIs 64 LAMPStack powered by BitNami 5.3.12-0
Vendor AMIs 64 TomcatStack powered by BitNami 7.0.26-0
Vendor AMIs 64 JBoss powered by BitNami 6.0.0-0
Public Community
AMIs
32 bitnami-phpbb-3.0.10-1-linux-redhat-6.2-i386-ebs
Public Community
AMIs
32 bitnami-phpbb-3.0.10-0-linux-redhat-6.2-i386
Public Community
AMIs
32 bitnami-mantis-1.2.8-0-linux-redhat-6.2-i386
Public Community
AMIs
32 bitnami-mantis-1.2.9-0-linux-redhat-6.2-i386
Public Community
AMIs
64 411009282317/RightImage_Windows_2008_x64_v5.5.2
Public Community
AMIs
64 411009282317/RightImage Windows_2008_x64_v5.4.3
Public Community
AMIs
64 411009282317/RightImage_Windows_2008R2_x64_v5.7.1.3

What Did We Find?

Using this approach, we discovered that different vendors configure the operating systems in different ways. Each AMI configuration could be significantly different, and the images differ between vendors and versions.

Here we show some sample results of comparisons between different versions of essentially the same images: 

Bitnami PHP 3.0.10-0 versus 3.0.10-1

  • Compared components included Linux OS, Apache and My SQL
  • Red Hat Kernel was updated from 2.6.32-220.4.2.el6.i686 to 2.6.32-220.7.1.el6.i686
  • A significant amount of differences was found in the versions of installed Red Hat packages and utilities such as nss, nspr, ntsysv, curl, gnutls and others
  • In Red Hat crontab execution frequency was changed from 30 to 11 minutes
  • Few new chains were defined in Red Hat kernel firewall in 3.0.10-1 version of AMI 

Inside Amazon's AMI Black Box: Not What You Would Have Expected
Bitnami Mantis 1.2.8-0 versus 1.2.9-0

  • Compared components included Linux OS, Apache and My SQL
  • A significant amount of differences was found in the versions of installed Red Hat packages and utilities such as nss, nspr, passwd, autofs, cvs and others
  • In Red Hat crontab execution frequency was changed from 30 to 4 minutes 

RightImage Windows 2008 5.4.3 versus 5.5.2

  • Compared components included Windows and Terminal Server
  • A remote management service (psexec, which is part of Windows sysinternals tools) was deployed in 5.5.2 

RightImage Windows 2008 5.5.2 versus 5.7.1.3

  • Compared components including the Windows and Terminal Server. While the service pack level of Windows was promoted from SP1 to R2 between the images there were a few interesting changes in configuration not related to the service pack level
  • For example, Windows Firewall is enabled in 5.7.1.3 only
  • A number of Windows services are changed from auto start mode to manual, e.g. Secondary Logon, Background Intelligent Transfer Service etc.
  • Initial and maximum size of a paging file was significantly increased

Here are examples of differences detected when comparing AMIs with the same components but built by different vendors: 

Windows 2008 R1 from AWS versus RightImage

  • Active Desktop is enabled in Terminal Server of AWS and disabled in RightImage
  • Time Zone is set for UTC in AWS and GMT in RightImage
  • Windows Firewall is enabled in AWS and disabled in RightImage 

Inside Amazon's AMI Black Box: Not What You Would Have Expected
Windows 2008 R2 from AWS versus RightImage

  • Time Zone is set for UTC in AWS and GMT in RightImage
  • There are various European language packs installed only in RightImage
  • There is a service Block Level Backup Engine Service deployed in RightImage only
  • The starting modes of few services (IKE and AuthIP IPsec Keying Modules and Background Intelligent Transfer Service) differ between the images 

Why it's Important

The actual configuration information provides you clear visibility into these instances, allowing you to use AMIs confidently, eliminating any issues related to the choice of vendors or changes over time of the AMIs you pick.

You can be confident that:

  • The correct versions/revisions of applications that you need are present on the image
  • The AMI does not contain unneeded (or harmful) software, that could compromise your image security
  • You are aware of the small things that sometimes prevent application deployment. A very good example is a strict mode of Selinux that may prevent creating files and may take ages to locate.

Your Turn
How are you using instances provided in Amazon's catalog?

About the Author
Sasha Gilenson
Sasha Gilenson enjoyed a long and successful career at Mercury Interactive (acquired by HP), having led the company's QA organization, participating in establishing Mercury's Software as a Service (SaaS), as well as leading a Business Unit in Europe and Asia.

Sasha played a key role in the development of Mercury's worldwide Business Technology Optimization (BTO) strategy and drove field operations of the Wireless Business Unit, all while taking on the duties as the Mercury's top "guru" in quality processes and IT practices domain. In this capacity, Sasha has advised numerous Fortune 500 companies on technology and process optimization, and in turn, acquired a comprehensive and rare knowledge of the market and industry practices.

Sasha holds an M.Sc. in Computer Science from Latvian University and MBA from London Business School.