Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Friday, 6 October 2017

AWS Introduction (Compute & Networking)

Today we learn basics of AWS. What AWS means????
AWS (Amazon Web service) is an amazon cpublic cloud. Amazon is the biggest cloud provider company in world. Openstack is an open source and use for public and private clouds.
AWS has many components not possible to discuss all here.

The Differences that Distinguish AWS

AWS is readily distinguished from other vendors in the traditional IT computing landscape because it is:
Flexible. AWS enables organizations to use the programming models, operating systems, databases, and architectures with which they are already familiar. In addition, this flexibility helps organizations mix and match architectures in order to serve their diverse business needs.
Cost-effective. With AWS, organizations pay only for what they use, without up-front or long-term commitments.
Scalable and elastic. Organizations can quickly add and subtract AWS resources to their applications in order to meet customer demand and manage costs.
Secure. In order to provide end-to-end security and end-to-end privacy, AWS builds services in accordance with security best practices, provides the appropriate security features in those services, and documents how to use those features.
Experienced. When using AWS, organizations can leverage Amazon’s more than fifteen years of experience delivering large-scale, global infrastructure in a reliable, secure fashion.

Compute & Networking

Amazon Elastic Compute Cloud (Amazon EC2)
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers and system administrators.
Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use. Amazon EC2 provides developers and system administrators the tools to build failure resilient applications and isolate themselves from common failure scenarios.

Auto Scaling
Auto Scaling allows you to scale your Amazon EC2 capacity up or down automatically according to conditions you define. With Auto Scaling, you can ensure that the number of Amazon EC2 instances you’re using increases seamlessly during demand spikes to maintain performance, and decreases automatically during demand lulls to minimize costs. Auto Scaling is particularly well suited for applications that experience hourly, daily, or weekly variability in usage. Auto Scaling is enabled by Amazon CloudWatch and available at no additional charge beyond Amazon CloudWatch fees.

Elastic Load Balancing
Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve even greater fault tolerance in your applications, seamlessly providing the amount of load balancing capacity needed in response to incoming application traffic. Elastic Load Balancing detects unhealthy instances and automatically reroutes traffic to healthy instances until the unhealthy instances have been restored. Customers can enable Elastic Load Balancing within a single Availability Zone or across multiple zones for even more consistent application performance.

Amazon WorkSpaces
Amazon WorkSpaces is a fully managed desktop computing service in the cloud. Amazon WorkSpaces allows customers to easily provision cloud-based desktops that allow end-users to access the documents, applications and resources they need with the device of their choice, including laptops, iPad, Kindle Fire, or Android tablets. With a few clicks in the AWS Management Console, customers can provision a high-quality desktop experience for any number of users at a cost that is highly competitive with traditional desktops and half the cost of most virtual desktop infrastructure (VDI) solutions.

Amazon Virtual Private Cloud (Amazon VPC)
Amazon Virtual Private Cloud lets you provision a logically isolated section of the Amazon Web Services (AWS) Cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways. You can easily customize the network configuration for your Amazon VPC. For example, you can create a public-facing subnet for your webservers that has access to the Internet, and place your backend systems such as databases or application servers in a private-facing subnet with no Internet access. You can leverage multiple layers of security (including security groups and network access control lists) to help control access to Amazon EC2 instances in each subnet.

Amazon Route 53
Amazon Route 53 is a highly available and scalable Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost-effective way to route end users to Internet applications by translating human readable names, such as www.example.com, into the numeric IP addresses, such as 192.0.2.1, that computers use to connect to each other. Route 53 effectively connects user requests to infrastructure running in AWS, such as an EC2 instance, an elastic load balancer, or an Amazon S3 bucket. Route 53 can also be used to route users to infrastructure outside of AWS. Amazon Route 53 is designed to be fast, easy to use, and cost effective. It answers DNS queries with low latency by using a global network of DNS servers. Queries for your domain are automatically routed to the nearest DNS server, and thus are answered with the best possible performance. With Route 53, you can create and manage your public DNS records with the AWS Management Console or with an easy-to-use API. It’s also integrated with other Amazon Web Services. For instance, by using the AWS Identity and Access Management (IAM) service with Route 53, you can control who in your organization can make changes to your DNS records. Like other Amazon Web Services, there are no long-term contracts or minimum usage requirements for using Route 53—you pay only for managing domains through the service and the number of queries that the service answers.

AWS Direct Connect
AWS Direct Connect makes it easy to establish a dedicated network connection from your premises to AWS. Using AWS Direct Connect, you can establish private connectivity between AWS and your data center, office, or co-location environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections. AWS Direct Connect lets you establish a dedicated network connection between your network and one of the AWS Direct Connect locations. Using industry standard 802.1Q virtual LANS (VLANs), this dedicated connection can be partitioned into multiple logical connections. This allows you to use the same connection to access public resources such as objects stored in Amazon S3 using public IP address space, and private resources such as Amazon EC2 instances running within an Amazon VPC using private IP space, while maintaining network separation between the public and private environments. Logical connections can be reconfigured at any time to meet your changing needs.

So in next blog we will discuss about Storage & Content Delivery Network , subscribe to this blog for latest update.

Reference: Take help from amazon official documentation

Tuesday, 23 May 2017

AWS VS OpenStack?????

Let us compare the popularity of two top cloud-computing platforms:
1) the infamous Amazon Web Services, which companies typically leverage for the speed and convenience of Amazon's global, hosted, cloud-computing infrastructure, and
2) the increasingly versatile OpenStack, which allows organizations to roll their own cloud-computing services on standard hardware.

OpenStack has similarly grown in popularity since its launch in 2010, having had a nice jump in the Spring of 2013. Some of the more notable companies contributing to OpenStack include: AT&T, MD, Canonical, Cisco, Citrix, Comcast, Cray, Dell, Dreamhost, EMC, Ericsson, Fujitsu, GoDaddy, Google, HP, Hitachi, Huawei, IBM, Intel, Juniper Networks, Mirantis, Oracle, Red Hat, SUSE Linux, VMware, and Yahoo!.
While OpenStack has a lot of diverse contributors, AWS is the fifth largest web hosting provider globally.
Worldwide Market Share by Number of Clients in 2015:
  1. GoDaddy - 4.26%
  2. BlueHost - 2.56%
  3. Host Gator - 2.15%
  4. OVH.com - 1.91%
  5. Amazon Web Services - 1.81%
  6. Rackspace - 1.59%
  7. 1&1 - 1.54%
  8. Hetzner - 1.29%
  9. SoftLayer - 1.19%
  10. DreamHost - 1.01%

source: http://hostadvice.com/marketshare/ (2015)

As an open-source cloud-computing protocol, OpenStack obviously can't compete on these terms with a multi-billion dollar cloud-computing and software-as-a-service company. There are a number selling points to consider, however:
  • WalMart uses OpenStack to coordinate 100,000+ cores, this provided 100% uptime during Black Friday last year
  • Developers gave over 300 talks at the OpenStack Summit in Tokyo this October
  • Debian, Canonical, Red Hat, and SUSE Linux all support OpenStack and are active contributors
  • OpenStack has enabled companies like Disney, Bloomberg, and Wells Fargo to manage their own clouds at a fraction of the cost of proprietary solutions like AWS
  • OpenStack is the only solution that supports mixed hypervisor and bare metal server environments

I think these points lend themselves to the conclusion that adoption and further development in OpenStack are likely to keep pace.

Here is a subset of AWS services:

In the Compute realm we have...
  • Amazon Elastic Compute Cloud (EC2) scalable virtual private servers using Xen
  • Amazon Elastic MapReduce (EMR) Hadoop-based big data analytics
In Networking we have...
  • Amazon Route 53 scalable DNS
  • Amazon Virtual Private Cloud (VPC) isolated EC2 instances with the ability to extend corporate networks VPN
  • Amazon Elastic Load Balancing (ELB)
In Content Delivery we have...
  • Amazon CloudFront CDN
In Storage we have...
  • Amazon Simple Storage Service (S3)
  • Amazon Glacier low-cost, long-term storage for data archival
  • Amazon Elastic File System (EFS) to accompany EC2
In the Database realm we have...
  • Amazon DynamoDB low-latency NoSQL SSD-backed databases
  • Amazon Relational Database Service (RDS) with MySQL, Oracle, SQL Server, and PostgreSQL support
  • Amazon SimpleDB distributed database with EC2 and S3 interoperability, written in Erlang
In the Deployment realm we have...
  • AWS Elastic Beanstalk for quick deployment and cloud app management
  • AWS OpsWorks EC2 configuration services via Chef, which we discussed previously
In Management we have...
  • Amazon Identity and Access Management (IAM) to authenticate into the various services
  • AWS Directory Service for tying into an on-premises Microsoft Active Directory or for setting up a new stand-alone AWS directory
  • Amazon CloudWatch for application and resource monitoring
  • Amazon CloudHSM Hardware Security Module for data security and for meeting regulatory compliance requirements
  • AWS Key Management Service (KMS) for creating and managing encryption keys
In the Application Services realm we have...
  • Amazon DevPay (beta) for billing and account management
  • Amazon Elastic Transcoder (ETS) for mobile video transcription from S3
  • Amazon Simple Email Service (SES) for sending bulk and transactional email
  • Amazon Simple Notification Service (SNS) multi-protocol application "push" notifications
  • Amazon Cognito secure application-user data management and synchronization tool
In Analytics we have...

Here are the main components of the modular OpenStack architecture:

Compute (Nova)
  • An Infrastructure as a Service (IaaS) system
  • Management and automatation of pools of computer resources
  • Bare metal and high-performance computing (HPC) configurations
  • KVM, VMware, and Xen hypervisor virtualization
  • Hyper-V and LXC containerization
  • Python-based with various external libraries: Eventlet for concurrent programming, Kombu for AMQP communication, SQLAlchemy for database access, etc.
  • Designed to scale horizontally on standard hardware with no proprietary hardware or software requirements
  • Interoperable with legacy systems
Image Service (Glance)
  • OpenStack Image Service for discovery, registration, and delivery of services for disk and server images
  • Template-building from stored images
  • Storage and cataloging of unlimited backups
  • REST interface for querying disk image information
  • Streaming of images to servers
  • VMware integration, with vMotion Dynamic Resource Scheduling (DRS) and live migration of running virtual machines
  • All OpenStack OS images built on virtual machines
  • Maintenance of image metadata
  • Creation, deletion, sharing, and duplification of images
Object Storage (Swift)
  • Scalable redundant storage system
  • Automatic replication of content from failed disks to other active nodes
  • Suitable for inexpensive commodity hard drives and servers
Dashboard (Horizon)
  • GUI for access, provision, and automation of cloud-based resources for administrators and users
  • Third-party billing, monitoring, management tool integration
  • Customizable (brandable) dashboard
  • EC2 compatibility
Identity Service (Keystone)
  • Unified authentication system across the cloud OS
  • Integration with existing backend directory services such as LDAP
  • Various authentication methods: username/password, token-based systems, and AWS-style logins
  • Queryable, single registry of all deployed services, with programmatic determination of access for users and third-party tools
Networking (Neutron)
  • Manual and automatic management of networks and IP addresses
  • Distict networking models for different applications and user groups
  • Flat networks (VLAN's) for separating servers and traffic.
  • Static IP addresses, DHCP
  • Floating IP addresses for dynamic rerouting to resources on the network
  • Software-defined networking (SDN), OpenFlow, for multi-tenancy and scalability.
  • Management of intrusion detection systems (IDS), load balancing, firewalls, VPN's, etc.
Block Storage (Cinder)
  • Persistent block-level storage for databases and expandable file systems
  • Block storage integration into OpenStack Compute and Dashboard for allocation of storage
  • Various storage platforms supported: Ceph, CloudByte, Coraid, EMC (ScaleIO, VMAX and VNX), GlusterFS, Hitachi Data Systems, IBM Storage (Storwize family, SAN Volume Controller, XIV Storage System, and GPFS), Linux LIO, NetApp, Nexenta, Scality, SolidFire, HP (StoreVirtual and 3PAR StoreServ families) and Pure Storag
  • Snapshot management for backing up data stored on block storage volumes
  • Restoring of snapshots, use of snapshots as templates for new block storage volumes
Orchestration (Heat)
  • Orchestratation of multiple composite cloud applications using templates
  • OpenStack-native REST API
  • CloudFormation-compatible Query API
Telemetry (Ceilometer)
  • Billing system Single Point Of Contact
  • Traceable, auditable delivery of counters for billing
  • Counters extensible to new projects
  • Independent data collection
Database (Trove)
  • Database-as-a-service (DaaS) provisioning relational database engine
  • DaaS non-relational database engine
Elastic Map Reduce (Sahara)
  • Hadoop cluster provisioning
  • Setting of parameters based on: Hadoop version, cluster topology, node hardware details, etc.
  • Cluster deployment in minutes
  • Scaling of already-provisioned clusters by adding and removing worker nodes on demand
Bare Metal Provisioning (Ironic)
  • Provisioning of bare metal machines (as opposed to virtual machines)
  • Bare-metal hypervisor API
  • Plugins for interacting with bare-metal hypervisors
  • PXE and IPMI simultaneous provisioning, turning machines on and off as needed
  • Extensible with vendor-specific plugins for additional functionality
Multiple Tenant Cloud Messaging (Zaqar)
  • Multi-tenant cloud messaging service for Web developers
  • Some components inspired by Amazon's SQS, with additional semantics for event broadcasting
  • Fully RESTful API for sending messages between various components of their SaaS and mobile applications
  • Surfacing of events to end users and guest agents that run in the "over-cloud" layer
Shared File System Service (Manila)
  • Vendor-agnostic share management API
  • Create, delete, give/deny access to a share
  • Support for commercial storage appliances from: EMC, NetApp, HP, IBM, Oracle, Quobyte, and Hitachi Data Systems
  • Support for Red Hat's GlusterFS filesystem
DNSaaS (Designate)
  • DNS as a Service
Security API (Barbican)
  • REST API for secure storage, provisioning and management of secrets
  • Built for use in all environments, including large ephemeral clouds
AWS Compatibility

  • Interoperability with Amazon EC2 and Amazon S3
  • Minimal effort to port AWS client applications to OpenStack

Multi tenancy: AWS vs OpenStack

The first conceptual difference between AWS and OpenStack is about multi-tenancy. OpenStack offers a multi-layer tenant mechanism with domain and projects. A domain is a collection of users, groups, and projects, in a way parallel to AWS’s account. LDAP groups are attached to domains. OpenStack’s Project is a container of virtual resources such as virtual machines, networks and volumes. Using projects, users can establish several isolated and independently controlled groups of resources that serve different objectives. In the Kilo release, Keystone introduced the hierarchical multi-tenancy concept, using sub-projects.
Accommodating more than one million customers, Amazon Web Services is a multi-tenant cloud by nature; however, at the account level, a single user receives a single tenant experience. Having said that, AWS offers Virtual Private Cloud, VPC, which is somewhat parallel to OpenStack’s project. Amazon’s VPC lets the user provision a logically isolated section of the Amazon Web Services (AWS) cloud where the user can launch AWS resources in a virtual network that s/he defines.  AWS’s VPC is limited to one router and one IP block; though not compulsory, this is a common practice for OpenStack projects. It is worth noting that all EC2’s virtual networking capabilities are only available using VPC.
On the other hand, unlike OpenStack, Amazon’s VPC offers extremely valuable tools that simplify the establishment of secured connectivity between VPCs and between VPC and on-premise resources. The classical use case for enterprises is running the web servers or the entire customer facing application on Amazon’s public cloud, while keeping the rest of the servers on-premise. Through its API, Amazon allows the user to establish a VPN connection and even control the customer’s and AWS’s gateways. This is extremely valuable for enterprises that chose the hybrid cloud path, especially given the fact that Amazon has integrated its VPN gateway with the market leading VPN CPEs (customer premise equipment). OpenStack’s Neutron project does offers VPN as a Service capabilities (VPNaaS); however, it is experimental and lacks the end-to-end integration that Amazon provides.

Networking: Neutron vs AWS VPC

From the network perspective, while OpenStack provides control over the L2 elements of the virtual network, AWS exposes only subnets. OpenStack Neutron’s API allows granular control of elements such as ports (the connection point for attaching a virtual server to a virtual network) and the ability to allocate VLAN IDs that correspond to VLANs present in the physical network. This is especially useful for provider networks, which are mapped to existing physical networks in the data center. Those differences are again attributed to the different concepts of AWS and OpenStack. In the private cloud, the user manages the physical networking by himself. Thus, it is crucial for virtual networking to be fully integrated with the physical data center networking. However, the public cloud is a managed service that takes all the hassle of physical network management away from the user, therefore  providing control on L2 is irrelevant to the user.
As for Layer 3 networking, conceptually, Amazon’s AWS and OpenStack’s Neutron provides comparable capabilities. Both cloud services allow creation of network subnets. OpenStack allows use of several subnets on the same virtual network, although it is not a common practice. AWS allows users to define Elastic IP addresses, which are public IP addresses reachable from the Internet. OpenStack offers a similar mechanism, the floating IP, which is part of the virtual router’s API.
Both clouds provide routing services; in AWS each VPC includes an implicit virtual router and the API allows the user to set the routing table (which contains a set of rules, called routes, that are used to determine where network traffic is directed). OpenStack’s Neutron API also allows management of the routing table;  however, it also allows management of the router entities themselves and does not limiting the number of routers per project. Moreover, a single router can be connected to more than one project.
From a network security perspective, AWS and OpenStack offer similar mechanisms. Security groups are used to inspect the traffic at the instance level. Networking ACLs and virtual firewalls are used by AWS and OpenStack respectively to inspect traffic going between subnets. There are minor nuances unique to each API; however, the general concept is very similar.

Here we compare some main components of AWS and OpenStack

Compute

 Why you need it?

To run an application you need a server with CPU, memory and storage, with or without pre-installed operating systems and applications.
OpenStack
AWS
Definition
Compute is virtual machines/servers
Instance
Instance/VM
Sizes
How much memory and CPU and temporary (ephemeral) storage is assigned to the instances/VM.
Flavors: Variety of sizes: micro, small, medium, large etc.
Variety of sizes: micro, small, medium, large etc.
Operating systems offered
What operating systems does the cloud offer to end-users
Whatever operating systems the cloud administrators host on the OpenStack cloud. (Red Hat certifies Microsoft Windows, RHEL and SUSE)
AMIs provided by the AWS marketplace.
Templates/images
A base configuration of a virtual machine, from which other virtual machines can be created.
Catalogs of virtual machine images can be created from which users can select a virtual machine.
Glance 
OpenStack administrators upload images and create catalogs for users.
Users can upload their own images.
(AMI) Amazon Machine Image
AWS provides an online marketplace of pre-defined images.
Users can upload their own images.

Networking

 Why you need it?
To network virtual servers to each other. You also need to control who can access the server. You want to protect/firewall the server especially if it is exposed to the Internet.
OpenStack
AWS
Definition
Networking provides connectivity for users to virtual machines. Connects virtual machines to one another and to external networks (the Internet).  
Neutron
Networking
A private IP address internal only and non-routable to the Internet
Every virtual instance is automatically assigned a private IP address, typically using DHCP.
AWS allocates a private IP address for the instance using DHCP.
Public IP address
A floating IP is a public IP address, that you can dynamically add to a running virtual instance.
AWS public IP address is mapped to the primary private IP address.
Networking service
You can create networks and networking functions, eg. L3 forwarding, NAT, edge firewalls, and IPsec VPN.
Virtual routers or switches can be added if you use AWS VPC, a virtual public cloud.
Load Balance VM traffic
OpenStack LBaaS (Load Balancing as a Service) balances traffic from one network to application services.
ELB (Elastic Load Balancing) automatically distributes incoming application traffic across Amazon EC2 instances.
DNS.
Manage the DNS entries for your virtual servers and web applications.
The OpenStack DNS project (Designate) is in “incubation” and is not part of core OpenStack (as of the April 2015 Kilo release).
Route 53 –  AWS’s DNS service.
SRIOV
A method of device virtualization that provides higher I/O performance and lower CPU utilization compared to traditional implementations.
Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging or they may be extended to an upstream physical switch (IEEE 802.1br).
AWS support enhanced networking capabilities using SR-IOV, provides higher packet per second (PPS) performance, lower inter-instance latencies, and very low network jitter.

Monitoring

Why you need it?

You get insight into usage patterns and utilization of the physical and virtual resources. You may want to account for individual usage and optionally bill users for their usage.
OpenStack
AWS
Definition
Monitoring provides metering and usage of the cloud.  
Ceilometer
Cloudwatch
System-wide metering and usage.
Option to bill users for their usage
To collect measurements of the utilization of the physical and virtual resources comprising deployed clouds.
Persist data for subsequent retrieval and analysis, and trigger actions when defined criteria are met.
Monitoring service for AWS cloud resources and the applications  on AWS.
Collect and track metrics, collect and monitor log files, and set alarms.

Security

Why you need it?

You need the  option of public key cryptography for SSH and password decryption. You want to firewall virtual machines to only allow certain traffic in (ingress) or out (egress).


OpenStack
AWS
Definition
Control access to your virtual machines.  
Keypairs, security groups.
Keypairs, security groups.
Key pairs
To login to your VM or instance, you must create a key pair.
Linux: used to SSH.
Windows: used to decrypt the Administrator password.
When you launch a virtual machine, you can inject a key pair, which provides SSH access to your instance.
To log in to your instance, specify the name of the key pair when you launch the instance, and provide the private key when you connect to the instance.
Assign and control access to VM instances.
A security group is a named collection of network access rules that limit the traffic that access an instance.
When you launch an instance, you can assign one or more security groups to it.
Supported
Supported