| A compendium of Windows Azure, Service Bus, EAI & EDI,Access Control, Connect, SQL Azure Database, and other cloud-computing articles. |  |
Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:
- Windows Azure Blob, Drive, Table, Queue and Hadoop Services
- SQL Azure Database, Federations and Reporting
- Marketplace DataMarket, Cloud Numerics, Big Data and OData
- Windows Azure Service Bus, Access Control, Caching, Active Directory, and Workflow
- Windows Azure Virtual Machines, Virtual Networks, Web Sites, Connect, RDP and CDN
- Live Windows Azure Apps, APIs, Tools and Test Harnesses
- Visual Studio LightSwitch and Entity Framework v4+
- Windows Azure Infrastructure, Media Services and DevOps
- Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
- Cloud Security and Governance
- Cloud Computing Events
- Other Cloud Computing Platforms and Services
Azure Blob, Drive, Table, Queue and Hadoop Services
Arun Murthy (@acmurthy) posted Apache Hadoop YARN – Background and an Overview to the Hortonworks Blog on 8/7/2012:
Celebrating the significant milestone that was Apache Hadoop YARN being promoted to a full-fledged sub-project of Apache Hadoop in the ASF we present the first blog in a multi-part series on Apache Hadoop YARN – a general-purpose, distributed, application management framework that supersedes the classic Apache Hadoop MapReduce framework for processing data in Hadoop clusters.
MapReduce – The Paradigm
Essentially, the MapReduce model consists of a first, embarrassingly parallel, map phase where input data is split into discreet chunks to be processed. It is followed by the second and final reduce phase where the output of the map phase is aggregated to produce the desired result. The simple, and fairly restricted, nature of the programming model lends itself to very efficient and extremely large-scale implementations across thousands of cheap, commodity nodes.
Apache Hadoop MapReduce is the most popular open-source implementation of the MapReduce model.
In particular, when MapReduce is paired with a distributed file-system such as Apache Hadoop HDFS, which can provide very high aggregate I/O bandwidth across a large cluster, the economics of the system are extremely compelling – a key factor in the popularity of Hadoop.
One of the keys to this is the lack of data motion i.e. move compute to data and do not move data to the compute node via the network. Specifically, the MapReduce tasks can be scheduled on the same physical nodes on which data is resident in HDFS, which exposes the underlying storage layout across the cluster. This significantly reduces the network I/O patterns and keeps most of the I/O on the local disk or within the same rack – a core advantage.
Apache Hadoop MapReduce, circa 2011 – A Recap
Apache Hadoop MapReduce is an open-source, Apache Software Foundation project, which is an implementation of the MapReduce programming paradigm described above. Now, as someone who has spent over six years working full-time on Apache Hadoop, I normally like to point out that the Apache Hadoop MapReduce project itself can be broken down into the following major facets:
- The end-user MapReduce API for programming the desired MapReduce application.
- The MapReduce framework, which is the runtime implementation of various phases such as the map phase, the sort/shuffle/merge aggregation and the reduce phase.
- The MapReduce system, which is the backend infrastructure required to run the user’s MapReduce application, manage cluster resources, schedule thousands of concurrent jobs etc.
This separation of concerns has significant benefits, particularly for the end-users – they can completely focus on the application via the API and allow the combination of the MapReduce Framework and the MapReduce System to deal with the ugly details such as resource management, fault-tolerance, scheduling etc.
The current Apache Hadoop MapReduce System is composed of the JobTracker, which is the master, and the per-node slaves called TaskTrackers.
The JobTracker is responsible for resource management (managing the worker nodes i.e. TaskTrackers), tracking resource consumption/availability and also job life-cycle management (scheduling individual tasks of the job, tracking progress, providing fault-tolerance for tasks etc).
The TaskTracker has simple responsibilities – launch/teardown tasks on orders from the JobTracker and provide task-status information to the JobTracker periodically.
For a while, we have understood that the Apache Hadoop MapReduce framework needed an overhaul. In particular, with regards to the JobTracker, we needed to address several aspects regarding scalability, cluster utilization, ability for customers to control upgrades to the stack i.e. customer agility and equally importantly, supporting workloads other than MapReduce itself.
We’ve done running repairs over time, including recent support for JobTracker availability and resiliency to HDFS issues (both of which are available in Hortonworks Data Platform v1 i.e. HDP1) but lately they’ve come at an ever-increasing maintenance cost and yet, did not address core issues such as support for non-MapReduce and customer agility.
Why support non-MapReduce workloads?
MapReduce is great for many applications, but not everything; other programming models better serve requirements such as graph processing (Google Pregel / Apache Giraph) and iterative modeling (MPI). When all the data in the enterprise is already available in Hadoop HDFS having multiple paths for processing is critical.
Furthermore, since MapReduce is essentially batch-oriented, support for real-time and near real-time processing such as stream processing and CEPFresil are emerging requirements from our customer base.
Providing these within Hadoop enables organizations to see an increased return on the Hadoop investments by lowering operational costs for administrators, reducing the need to move data between Hadoop HDFS and other storage systems etc.
Why improve scalability?
Moore’s Law… Essentially, at the same price-point, the processing power available in data-centers continues to increase rapidly. As an example, consider the following definitions of commodity servers:
- 2009 – 8 cores, 16GB of RAM, 4x1TB disk
- 2012 – 16+ cores, 48-96GB of RAM, 12x2TB or 12x3TB of disk.
Generally, at the same price-point, servers are twice as capable today as they were 2-3 years ago – on every single dimension. Apache Hadoop MapReduce is known to scale to production deployments of ~5000 nodes of hardware of 2009 vintage. Thus, ongoing scalability needs are ever present given the above hardware trends.
What are the common scenarios for low cluster utilization?
In the current system, JobTracker views the cluster as composed of nodes (managed by individual TaskTrackers) with distinct map slots and reduce slots, which are not fungible. Utilization issues occur because maps slots might be ‘full’ while reduce slots are empty (and vice-versa). Fixing this was necessary to ensure the entire system could be used to its maximum capacity for high utilization.
What is the notion of customer agility?
In real-world deployments, Hadoop is very commonly deployed as a shared, multi-tenant system. As a result, changes to the Hadoop software stack affect a large cross-section if not the entire enterprise. Against that backdrop, customers are very keen on controlling upgrades to the software stack as it has a direct impact on their applications. Thus, allowing multiple, if limited, versions of the MapReduce framework is critical for Hadoop.
Enter Apache Hadoop YARN
The fundamental idea of YARN is to split up the two major responsibilities of the JobTracker i.e. resource management and job scheduling/monitoring, into separate daemons: a global ResourceManager and per-application ApplicationMaster (AM).
The ResourceManager and per-node slave, the NodeManager (NM), form the new, and generic, system for managing applications in a distributed manner.
The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The per-application ApplicationMaster is, in effect, a framework specific entity and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the component tasks.
The ResourceManager has a pluggable Scheduler, which is responsible for allocating resources to the various running applications subject to familiar constraints of capacities, queues etc. The Scheduler is a pure scheduler in the sense that it performs no monitoring or tracking of status for the application, offering no guarantees on restarting failed tasks either due to application failure or hardware failures. The Scheduler performs its scheduling function based on the resource requirements of the applications; it does so based on the abstract notion of a Resource Container which incorporates resource elements such as memory, cpu, disk, network etc.
The NodeManager is the per-machine slave, which is responsible for launching the applications’ containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager.
The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress. From the system perspective, the ApplicationMaster itself runs as a normal container.
Here is an architectural view of YARN:
One of the crucial implementation details for MapReduce within the new YARN system that I’d like to point out is that we have reused the existing MapReduce framework without any major surgery. This was very important to ensure compatibility for existing MapReduce applications and users. More on this later.
The next post will dive further into the intricacies of the architecture and its benefits such as significantly better scaling, support for multiple data processing frameworks (MapReduce, MPI etc.) and cluster utilization.
Matt Winkler wrote the following on 5/29/2012 in a [HadoopOnAzureCTP] Re: Daytona support Yahoo! group thread:
In the hadoop 2.0 timeframe, there are interesting capabilities that YARN brings to the table. We are evaluating those now but don’t have any concrete plans. As YARN evolves we will see other parts of the stack (like pig and hive) evolve to take advantage of the new model.
We are working closely to understand how this evolve[s] and what opportunities exist.
It’s a good bet that YARN will be part of the Apache Hadoop on Windows Azure picture in the future because Hortonworks is a Microsoft partner, but not the near future.
<Return to section navigation list>
SQL Azure Database, Federations and Reporting
Himanshu Singh (@himanshuks) posted Data Series: Control Database Access Using Windows Azure SQL Database Firewall Rules to the Windows Azure blog on 8/14/2012:
Editor’s Note: Today’s post comes from Kumar Vivek [pictured at right], Technical Writer in our Customer Experience team. This post provides an overview of the newly-introduced database-level firewall rules in Windows Azure SQL Database.
Windows Azure SQL Database firewall prevents access to your SQL Database server to help protect your data. You could specify firewall rules to control access to your SQL Database server by specifying ranges of acceptable IP addresses. However, these firewall rules were defined at the server level, and enabled clients to access your entire SQL Database server, that is, all the databases within the same logical server. What if you wanted to control access to particular databases (containing secure information) within your SQL Database server; you could not do so earlier.
Well, now you can! Introducing database-level firewall rules in Windows Azure SQL Database! In addition to the server-level firewall rules, you can now define firewall rules for each database in your SQL Database server to restrict access to selective clients. To do so, you must create a database-level firewall rule for the required database with an IP address range that is beyond the IP address range specified in the server-level firewall rule, and ensure that the IP address of the client falls in the range specified in the database-level firewall rule.
This is how the connection attempt from a client passes through the firewall rules in Windows Azure SQL Database:
- If the IP address of the request is within one of the ranges specified in the server-level firewall rules, the connection is granted to your SQL Database server.
- If the IP address of the request is not within one of the ranges specified in the server-level firewall rule, the database-level firewall rules are checked. If the IP address of the request is within one of the ranges specified in the database-level firewall rules, the connection is granted only to the database that has a matching database-level rule.
- If the IP address of the request is not within the ranges specified in any of the server-level or database-level firewall rules, the connection request fails.
For detailed information, see the full article Windows Azure SQL Database Firewall.
Managing Database-Level Firewall Rules
Unlike server-level firewall rules, the database-level firewall rules are created per database and are stored in the individual databases (including master). The sys.database_firewall_rules view in each database displays the current database-level firewall rules. Further, you can use the sp_set_database_firewall_rule and sp_delete_database_firewall_rule stored procedures in each database to create and delete the database-level firewall rules for the database.
For detailed information about managing database-level firewall rules, see the complete article How to: Configure the Database-Level Firewall Settings.
Cyrielle Simeone (@cyriellesimeone, pictured below) posted Thomas Mechelke’s Using a Windows Azure SQL Database with Autohosted apps for SharePoint on 8/13/2012:
This article is brought to you by Thomas Mechelke, Program Manager for SharePoint Developer Experience team. Thomas has been monitoring our new apps for Office and SharePoint forums and providing help on various topics. In today’s post, Thomas will walk you through how to use a Windows Azure SQL Database with autohosted apps for SharePoint, as it is one of the most active thread on the forum. Thanks for reading !
Getting started
In a previous post, Jay described the experience of creating a new autohosted app for SharePoint. That will be our starting point.
If you haven’t already, create a new app for SharePoint 2013 project and accept all the defaults. Change the app name if you like. I called mine "Autohosted App with DB". Accepting the defaults creates a solution with two projects: the SharePoint project with a default icon and app manifest, and a web project with some basic boilerplate code.
Configuring the SQL Server project
Autohosted apps for SharePoint support the design and deployment of a data tier application (DACPAC for short) to Windows Azure SQL Database. There are several ways to create a DACPAC file. The premier tools for creating a DACPAC are the SQL Server Data Tools, which are part of Visual Studio 2012.
Let’s add a SQL Server Database Project to our autohosted app:
- Right-click the solution node in Solution Explorer, and then choose Add New Project.
- Under the SQL Server node, find the SQL Server Database Project.
- Name the project (I called it AutohostedAppDB), and then choose OK.
A few steps are necessary to set up the relationship between the SQL Server project and the app for SharePoint, and to make sure the database we design will run both on the local machine for debugging and in Windows Azure SQL Database.
First, we need to set the target platform for the SQL Server Database project. To do that, right-click the database project node, and then select SQL Azure as the target platform.
Next, we need to ensure that the database project will update the local instance of the database every time we debug our app. To do that, right-click the solution, and then choose Set Startup Projects. Then, choose Start as the action for your database project.
Now, build the app (right-click Solution Node and then choose Build). This generates a DACPAC file in the database output folder. In my case, the file is at /bin/Debug/projectname.dacpac.
Now we can link the DACPAC file with the app for SharePoint project by setting the SQL Package property.
Setting the SQL Package property ensures that whenever the SharePoint app is packaged for deployment to a SharePoint site, the DACPAC file is included and deployed to Windows Azure SQL Database, which is connected to the SharePoint farm.
This was the hard part. Now we can move into building the actual database and data access code.
Building the database
SQL Server Data Tools adds a new view to Visual Studio called SQL Server Object Explorer. If this view doesn’t show up in your Visual Studio layout (usually as a tab next to Solution Explorer), you can activate it from the View menu. The view shows the local database generated from your SQL Server project under the node for (localdb)\YourProjectName.
This view is very helpful during debugging because it provides a simple way to get at the properties of various database objects and provides access to the data in tables.
Adding a table
For the purposes of this walkthrough, we’ll keep it simple and just add one table:
- Right-click the database project, and then add a table named Messages.
- Add a column of type nvarchar(50) to hold messages.
- Select the Id column, and then change the Is Identity property to be true.
After this is done, the table should look like this:
Great. Now we have a database and a table. Let’s add some data.
To do that, we’ll use a feature of data-tier applications called Post Deployment Scripts. These scripts are executed after the schema of the data-tier application has been deployed. They can be used to populate look up tables and sample data. So that’s what we’ll do.
Add a script to the database project. That brings up a dialog box with several script options. Select Post Deployment Script, and then choose Add.
Use the script editor to add the following two lines:
delete from Messages
insert into Messages values (‘Hello World!’)
The delete ensures the table is empty whenever the script is run. For a production app, you’ll want to be careful not to wipe out data that may have been entered by the end user.
Then we add the "Hello World!" message. That’s it.
Configuring the web app for data access
After all this work, when we run the app we still see the same behavior as when we first created the project. Let’s change that. The app for SharePoint knows about the database and will deploy it when required. The web app, however, does not yet know the database exists.
To change that we need to add a line to the web.config file to hold the connection string. For that we are using a property in the <appSettings> section named SqlAzureConnectionString.
To add the property, create a key value pair in the <appSettings> section of the web.config file in your web app:
<add key="SqlAzureConnectionString" value="Data Source=(localdb)\YourDBProjectName;Initial Catalog=AutohostedAppDB;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=False" />
The SqlAzureConnectionString property is special in that its value is set by SharePoint during app installation. So, as long as your web app always gets its connections string from this property, it will work whether it’s installed on a local machine or in Office 365.
You may wonder why the connection string for the app is not stored in the <connectionStrings> section. We implemented it that way in the preview because we already know the implementation will change for the final release, to support geo-distributed disaster recovery (GeoDR) for app databases. In GeoDR, there will always be two synchronized copies of the database in different geographies. This requires the management of two connection strings, one for the active database and one for the backup. Managing those two strings is non-trivial and we don’t want to require every app to implement the correct logic to deal with failovers. So, in the final design, SharePoint will provide an API to retrieve the current connection string and hide most of the complexity of GeoDR from the app.
I’ll structure the sample code for the web app in such a way that it should be very easy to switch to the new approach when the GeoDR API is ready.
Writing the data access code
At last, the app is ready to work with the database. Let’s write some data access code.
First let’s write a few helper functions that set up the pattern to prepare for GeoDR in the future.
GetActiveSqlConnection()
GetActiveSqlConnection is the method to use anywhere in the app where you need a SqlConnection to the app database. When the GeoDR API becomes available, it will wrap it. For now, it will just get the current connection string from web.config and create a SqlConnection object:
// Create SqlConnection.
protected SqlConnection GetActiveSqlConnection()
{
return new SqlConnection(GetCurrentConnectionString());
}
GetCurrentConnectionString()
GetCurrentConnectionString retrieves the connection string from web.config and returns it as a string.
// Retrieve authoritative connection string.
protected string GetCurrentConnectionString()
{
return WebConfigurationManager.AppSettings["SqlAzureConnectionString"];
}
As with all statements about the future, things are subject to change—but this approach can help to protect you from making false assumptions about the reliability of the connection string in web.config.
With that, we are squarely in the realm of standard ADO.NET data access programming.
Add this code to the Page_Load() event to retrieve and display data from the app database:
// Display the current connection string (don’t do this in production).
Response.Write("<h2>Database Server</h2>");
Response.Write("<p>" + GetDBServer() + "</p>");
// Display the query results.
Response.Write("<h2>SQL Data</h2>");
using (SqlConnection conn = GetActiveSqlConnection())
{
using (SqlCommand cmd = conn.CreateCommand())
{
conn.Open();
cmd.CommandText = "select * from Messages";
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
Response.Write("<p>" + reader["Message"].ToString() + "</p>");
}
}
}
}
We are done. This should run. Let’s hit F5 to see what happens.
It should look something like this. Note that the Database Server name should match your connection string in web.config.
Now for the real test. Right-click the SharePoint project and choose Deploy. Your results should be similar to the following image.
The Database Server name will vary, but the output from the app should not.
Using Entity Framework
If you prefer working with the Entity Framework, you can generate an entity model from the database and easily create an Entity Framework connection string from the one provided by GetCurrentConnectionString(). Use code like this:
// Get Entity Framework connection string.
protected string GetEntityFrameworkConnectionString()
{
EntityConnectionStringBuilder efBuilder =
new EntityConnectionStringBuilder(GetCurrentConnectionString());
return efBuilder.ConnectionString;
}
We need your feedback
I hope this post helps you get working on the next cool app with SharePoint, ASP.NET, and SQL. We’d love to hear your feedback about where you want us to take the platform and the tools to enable you to build great apps for SharePoint and Office.
Han reported SQL Data Sync Preview 6 is now live! in an 8/13/2012 post to the Sync Team blog:
- Enhance overall performance on initial provision and sync tasks
- Enhance sync performance between on-promise databases and Windows Azure SQL databases
Please download the new Agent from http://www.microsoft.com/en-us/download/details.aspx?id=27693
Also, for now on, we will be calling all subsequent preview releases Preview instead of the usual Service Update.
In another episode of Microsoft’s recent branding frenzy, SQL Azure Data Sync has become simply SQL Data Sync, similar to SQL Azure -> Windows Azure SQL Database, but (strangely) missing the Window Azure prefix.
Cihan Biyikoglu (@cihangirb) described Setting up Azure Data Sync Service with Federations in Windows Azure SQL Database For Reference Data Replication in an 8/12/2012 post:
In a previous post, I talked about the ability to use Data Sync Service with Federation Members. In this post, like to walk you through the details.
The scenario here is to sync a reference table called language_codes_tbl across federation members. The table represents language codes for the blogs_federation in my BlogsRUs_DB.
1. Create a “sync server” and a “sync group” called sync_codes
2. Add the root database as the hub database; blogsrus_db with conflict resolution set to “Hub Wins” and schedule set to every 5 mins.
3. Define the Sync dataset as the dbo.language_code_tbl
4. Add federation member databases into the sync_codes sync group and “deploy” the changes.
With this setup, replication happens bi-directionally. This means, I can update any one of the federation member dbs and the changes will first get replicated to the root db copy of my reference table and then will be replicated to all other federation member dbs automatically by SQL Data Sync. SQL Data Sync provides powerful control over the direction of data flow and conflict resolution to create the desired topology for syncing reference data in federation members.
Handling Federation Repartitioning Operations
This will work as long as you don’t reconfigure these members with an operation like ALTER FEDERATION … SPLIT. With SPLIT we drop the existing member database and create 2 new member databases that contain the redistributed data based on the new split point of the federation. Lets assume we issue the following statement to split the existing range 350000-400000.
alter federation blogs_federation split at (id=355000)
With that, you will notice that the sync group will start reporting an error on the member is impacted by the split operation – see the red error indicator marked below the database icon.
1. Given this database no longer exists, you need to deprovision the database from the sync group. To do this first remove the database with the “remove database” button above the topology area. You finalize the operation by deploying the change using the “deploy” button above. Since the database is dropped, you will need to do a forced removal after the deploy.
2. Next you need to add in the 2 new member names that are created by the SPLIT operation. we do this by running the following script. Once you have the new database names for the members covering the new ranges 350000-355000 and 355000 – 400000, you can follow step #4 above to add the names to the sync group.
Limitations with Azure Data Sync Service:
There are a few limitation to be aware of SQL Data Sync however; First the service has 5 min as its lowest latency for replication. There is no scripting support for set up of the data sync relationships. This means you will need to populate all the db names through the UI by hand. SQL Data sync also does not allow synchronization between more than 30 databases in sync groups in a single sync server at the moment. You can only create a single sync server with DSS today. SQL Data Sync is currently in preview mode and is continuously collecting feedback. Vote for your favorite request or add a new one at SQL Data Sync Feature Voting website!
<Return to section navigation list>
Marketplace DataMarket, Cloud Numerics, Big Data and OData
No significant articles today.
<Return to section navigation list>
Windows Azure Service Bus, Access Control Services, Caching, Active Directory and Workflow
Sheik Uduman Ali (@udooz) described Azure ServiceBus Message Payload Serialization using protobuf in an 8/7/2012 post:
Choices are between ready made coffee maker and make it ourselves available in Windows Azure kitchen. As long as we want cappuccino, Windows Azure .NET libraries are good to go in terms of productivity and maintainability. Sometimes, we may need to prepare blended iced cappuccino. REST API (the actual service interface to Windows Azure services) is the way for that. Here, I am talking about Windows Azure ServiceBus queue and how to use custom serialization on message payload (or body).
My Blended Iced Cappuccino
This is not the common case in the messaging world. A queue may be designated for receiving document type message or command type message. For command type messages, DataContractSerializer is fine. When the message type is document, both sender and receiver can agreed upon specific content-type of the message payload. For this, REST is the best friend.
Solution
To send a message, Windows Azure ServiceBus REST API requires the following:
- URI – http{s}://{serviceNamespace}.servicebus.Windows.net/{queue path}/messages
- Method – POST
- Header – Authorization header with WRAP token as value
- Request Body – could be anything
If everything going well, this web request returns 201.
To receive the message,
- URI – https://{serviceNamespace}.servicebus.Windows.net/{queue path}/messages/head?timeout={seconds}
- Method – POST (peek n lock) or DELETE (destructive)
- Header – Authorization header with WRAP token as value
This would returns message properties (for destructive nothing will be returned) and payload with response code 200.
In this case, there is no restriction on which serialization to be used on message payload. So, we can use protobuf. The main reason is content size. When a sender sends the message with HTTP content-type as application/protobuf, the receive always gets the message with the same content-type. …
Udooz continues with source code samples.
<Return to section navigation list>
Windows Azure Virtual Machines, Virtual Networks, Web Sites, Connect, RDP and CDN
Michael Washam (@MWashamMS) described Publishing and Synchronizing Web Farms using Windows Azure Virtual Machines in an 8/13/2012 post:
Deploying new web applications is pretty painless with Windows Azure Web Sites and “fairly” painless using Windows Azure PaaS style cloud services. However, for existing web apps that are being migrated to the cloud both solutions can require significant rewriting/re-architecture. That is where Windows Azure Infrastructure as a Service comes in. Running Virtual Machines allows you to have the economies of scale of using a cloud based solution and have full access to cloud services such as storage, service bus etc.. while not requiring you to re-architect your application to take advantage of these services.
In this post I will walk through how to use Windows Azure Virtual Machines to create a web farm that you can directly publish to using Visual Studio Web Deploy. In addition to simple publishing I will also show how you can automatically synchronize web content across multiple virtual machines in your service to make web farm content synchronization simple and painless.
Step #1 – Image Preparation
Create a new virtual machine using either Windows Server 2008 R2 or Server 2012. On this machine install the Application Server and Web Server roles and enable ASP.NET).
TIP: Don’t forget to install the .NET Framework 4.0 if you are using Server 2008 R2.
For this solution you will also need the Windows Azure PowerShell Cmdlets on the web server. See this article for configuring your publish settings with the PS cmdlets.
I will use the cmdlets to discover the VM names in my web farm without having to manually keep track of them. This helps if you need the ability to grow and shrink your web farm at will without updating your synchronization scripts.The tool I will use for content sync is Web Deploy 3.0. Download but do not install Web Deploy 3.0.
Web Deploy works by a starting a remote agent that listens for commands from either Visual Studio or the MSDeploy.exe client. By default it will listen on port 80. This default port configuration will not work in a load balanced environment.
To install on an alternate external port such as 8080:
C:\WebDeployInstall>msiexec /I webdeploy_amd64_en-us.msi /passive ADDLOCAL=ALL LISTENURL=http://+:8080/Once installed you will need to configure a firewall rule to allow traffic in on port 8080 for publishing and synchronization.
Now that the image is configured you will sysprep the vm to remove any unique characteristics like machine names etc. Ensure you have Enter System-Out-Of-Box Experience, Generalize and Shutdown all selected.
Once the VM status is shown as shut down in the Windows Azure Management portal highlight the VM and click capture. This will be the customized image you can use to quickly provision new VMs for your web farm using the management portal or powershell.
Ensure you check I have sysprepped this VM and name the image WebAppImg and click the check mark button to capture the image.
Step #2 – Virtual Machine Deployment
Once the image has been created you can use the portal or the Windows Azure PowerShell cmdlets to provision the web farm.
Here is a PowerShell example of using the new image as the basis for a three VM web farm.
A few things to note: I have created a load balanced endpoint for port 80 but for 8080 I’m only selecting a single server.
This server will be the target server for publishing from Visual Studio that will then be used as the source server for publishing to the other nodes in the web farm.$imgname = 'WebAppImg' $cloudsvc = 'MyWebFarm123' $pass = 'your password' $iisvm1 = New-AzureVMConfig -Name 'iis1' -InstanceSize Small -ImageName $imgname | Add-AzureEndpoint -Name web -LocalPort 80 -PublicPort 80 -Protocol tcp -LBSetName web -ProbePath '/' -ProbeProtocol http -ProbePort 80 | Add-AzureEndpoint -Name webdeploy -LocalPort 8080 -PublicPort 8080 -Protocol tcp | Add-AzureProvisioningConfig -Windows -Password $pass $iisvm2 = New-AzureVMConfig -Name 'iis2' -InstanceSize Small -ImageName $imgname | Add-AzureEndpoint -Name web -LocalPort 80 -PublicPort 80 -Protocol tcp -LBSetName web -ProbePath '/' -ProbeProtocol http -ProbePort 80 | Add-AzureProvisioningConfig -Windows -Password $pass $iisvm3 = New-AzureVMConfig -Name 'iis3' -InstanceSize Small -ImageName $imgname | Add-AzureEndpoint -Name web -LocalPort 80 -PublicPort 80 -Protocol tcp -LBSetName web -ProbePath '/' -ProbeProtocol http -ProbePort 80 | Add-AzureProvisioningConfig -Windows -Password $pass New-AzureVM -ServiceName $cloudsvc -VMs $iisvm1,$iisvm2,$iisvm3 -Location 'West US'Once the VMs are provisioned RDP into iis1 by clicking connect in the management portal. This is where you will configure a PowerShell script that will run MSDeploy to synchronize content across the other servers.
Inside of the iis1 virtual machine create a new text file named sync.ps1 in a directory off of your root such as C:\SynchScript and paste the following in (ensuring that you update $serviceName with your cloud service name).
Import-Module 'C:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\Azure\Azure.psd1' $publishingServer = (gc env:computername).toLower() $serviceName = 'REPLACE WITH YOUR CLOUD SERVICE' Get-AzureVM -ServiceName $serviceName | foreach { if ($_.Name.toLower() -ne $publishingServer) { $target = $_.Name + ":8080" $source = $publishingServer + ":8080" $exe = "C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe" [Array]$params = "-verb:sync", "-source:contentPath=C:\Inetpub\wwwroot,computerName=$source", "-dest:contentPath=C:\Inetpub\wwwroot,computerName=$target"; & $exe $params; } }This script enumerates all of the virtual machines in your cloud service and attempts to run a web deploy sync job on them. If you have other servers in your cloud service like database etc.. you could exclude them by filtering on the VM name. Note: Web Deploy supports MANY more operations other than just synchronizing directories. Click here to find more information.
To enable content synchronization you will need to create a new scheduled task by going into Control Panel -> Administrative Tools -> Scheduled Tasks -> Create a new Task.
Accept the defaults for everything except when it gets to the action screen.
Program/Script: powershell.exe
Parameters: -File C:\WebDeployInstall\sync.ps1Open the properties of the new task and you’ll need to modify the schedule to synchronize content fairly often so content isn’t out of sync during a publish.
Ensure you select Run Whether User Is Logged on or Not. You will need to provide an account for the task to run as. I’m choosing the administrator account because I am lazy. However, you could create new duplicate accounts on each of the VMs to use for synchronization.
Step #3 – Publishing with Visual Studio
Finally, to test the configuration create a new MVC app and tweak the code slightly to show the computer name.
Now right click on the project and select publish. In the drop down select new profile.
- In the settings page add your cloud app url and append :8080 to it for the service URL.
- Set the site/app name to Default Web Site
- Set the Destination URL to your cloud app url (without :8080)
Finish the wizard and let Visual Studio publish.
When the web app first launches you may or may not see the new content. It may show the default IIS8 content. As soon as the scheduled task runs the content should sync across all of the servers.
Once it has synchronized press CTRL F5 a few times and you should see the content with the individual machine names to verify the load balancing is working.
In this post you have seen how you can configure a custom OS image that can be used to provision virtual machines for a web farm. You have then seen how you can use Web Deploy along with PowerShell to synchronize content published from Visual Studio across all of the servers in your farm.
<Return to section navigation list>
Live Windows Azure Apps, APIs, Tools and Test Harnesses
BusinessWire announced Quest Software Debuts Foglight for Windows Azure Applications in an 8/13/2012 press release:
Building on its 20-year heritage as an industry leader in application performance monitoring, Quest Software today introduced Foglight® for Windows Azure Applications, an application performance monitoring (APM) solution available via software-as-a-service (SaaS). Available immediately as a beta, the newest addition to the industry-leading Foglight APM portfolio enables IT administrators to monitor performance and understand what end users are experiencing with Windows Azure-based applications.
News Facts:
- Enterprises recognize that end users have high expectations from applications. The success or failure of an application often is determined by the end user’s experience with that application; however, IT administrators sometimes struggle to understand the end user experience, particularly within cloud-based applications.
- Foglight for Windows Azure Applications enables enterprises to leverage cloud-based performance monitoring technology for applications built on the Windows Azure platform, and allows IT administrators to gain critical insight as to how end users interact with these applications.
- Foglight for Windows Azure Applications gives application owners and operators confidence that their users are getting the service expected, and the applications and infrastructure are performing optimally. Specifically, the product provides:
- Insight at a glance into the current and historical availability, as well as the health of the application and its supporting infrastructure
- The ability to drill down into problems to understand both their impact and probable cause
- Insight into the quality of service experienced by the application’s users, combined with an at-a-glance view of response time, showing normal behavior and drawing attention to anomalies
- A geographical view of performance and user location, drawing attention to problems affecting particular geographies rather than all users
- A true understanding of performance issues related to browser types, mobile device and other user agents, identifying compatibility issues and showing how users access the application
- Alarms that are reserved for truly important matters. In addition to appearing in the product, these can be forwarded to email clients so that IT staff are notified when critical issues arise, enabling them to take action when needed
- To access the beta version of Foglight for Windows Azure Applications, please visit http://www.foglight-on-demand.com/content/en/appPerformance/guided-tour.html?Azure
Quest’s Foglight Products Continue to Simplify APM:
- For more than 20 years, Quest Software’s industry-leading Foglight application performance monitoring solution has provided unparalleled monitoring capabilities for enterprises worldwide.
- Quest’s longstanding leadership in the performance monitoring of Windows applications and deep-dive monitoring of multivendor databases is ideally suited for helping large enterprises maximize operational efficiencies while reducing the cost, complexity and administrative overhead of managing application performance.
- Unique user experience management capabilities, real-time session capture, replay and analysis offer a 360-degree view from business, IT and end user perspectives to expedite the resolution of performance problems and improve transaction conversion rates. …
The press release continues with supporting quotes, etc.
Richard Conway (@azurecoder) described Building a virus scanning gateway in Windows Azure with Endpoint Protection in an 8/12/2012 post to the Elastacloud blog:
I remember being on a project some 9 years ago and having to build one of these. To build a realtime gateway is not as easy as you would think. In my project there were accountants uploading invoices of various types and formats that we had to translate into text using an OCR software package. We built a workflow using a TIBCO workflow designer solution (which I wouldn’t hesitate now to replace with WF!)
At a certain point people from outside the organisation had the ability to upload a file and this file had to be intercepted by a gateway before being persisted and operated on the through the workflow. You would think that this was an easy and common solution to implement. However, at the time it wasn’t. We used a Symantec gateway product and its C API which allowed us to use the ICAP protocol and thus do real time scanning.
Begin everything with a web role
For the last 6 months I’ve wanted to talk about Microsoft Endpoint Protection (http://www.microsoft.com/en-us/download/details.aspx?id=29209) which is still in CTP as a I write this. It’s a lesser known plugin which exists for Windows Azure. For anybody that receives uploaded content, this should be a commonplace part of the design. In this piece I want to look at a pattern for rolling your gateway with Endpoint Protection. It’s not ideal because it literally is a virus scanner, enabling real time protection and certain other aspects but uses Diagnostics to show issues that have taken place.
The files which are part of the endpoint protection plugin
So initially we’ll enable the imports:
<Imports> <Import moduleName="Diagnostics" /> <Import moduleName="Antimalware" /> <Import moduleName="RemoteAccess" /> <Import moduleName="RemoteForwarder" /> </Imports>You can see the addition of Antimalware here.
Correspondingly, our service configuration gives us the following new settings:
<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="<my connection string>" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.ServiceLocation" value="North Europe" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.EnableAntimalware" value="true" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.EnableRealtimeProtection" value="true" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.EnableWeeklyScheduledScans" value="false" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.DayForWeeklyScheduledScans" value="7" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.TimeForWeeklyScheduledScans" value="120" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.ExcludedExtensions" value="txt|rtf|jpg" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.ExcludedPaths" value="" /> <Setting name="Microsoft.WindowsAzure.Plugins.Antimalware.ExcludedProcesses" value="" />The settings are using Endpoint Protection for real time protection and scheduled scan. It’s obviously highly configurable like most virus scanners and in the background will update all malware definitions securely from a Microsoft source.
Endpoint protection installed on our webrole
First thing we’ll do is download a free virus test file from http://www.eicar.org/85-0-Download.html. Eicar has ensured that this definition is picked by most of the common virus scanning so Endpoint Protection should recognise this immediately. I’ve tested this with the .zip file but any of them are fine.
The first port of call is setting up diagnostics to proliferate the event log entries. We can do this within our RoleEntryPoint.OnStart method for our web role.
var config = DiagnosticMonitor.GetDefaultInitialConfiguration(); //exclude informational and verbose event log entries config.WindowsEventLog.DataSources.Add("System!*[System[Provider[@Name='Microsoft Antimalware'] and (Level=1 or Level=2 or Level=3 or Level=4)]]"); //write to persisted storage every 1 minute config.WindowsEventLog.ScheduledTransferPeriod = System.TimeSpan.FromMinutes(1.0); DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
Diagnostics info in Azure Management Studio
Okay, so in testing it looks like the whole process of cutting and pasting the file onto the desktop or another location takes about 10 seconds for the Endpoint Protection to pick this up and quarante the file. Given this we’ll set the bar at 20 seconds.
Endpoint protection discovers malware
I created a very simple ASP.NET web forms application with a file upload control. There are two ways to detect whether the file has been flagged as malware:
- Check to see whether the file is still around or has been removed and placed in quarantine
- Check the eventlog entry to see whether this has been flagged as malware.
We’re going to focus on No.2 so I’ve created a simple button click event which will persist the file. Endpoint protection will kick in within the short period so we’ll write the file to disk and then pause for 20 seconds. After our wait we’ll then check the eventlog and in the message string we’ll have a wealth of information about the file which has been quarantined.
bool hasFile = fuEndpointProtection.HasFile; string path = ""; if(hasFile) { path = Path.Combine(Server.MapPath("."), fuEndpointProtection.FileName); fuEndpointProtection.SaveAs(path); } // block here until we check endpoint protection to see whether the file has been delivered okay! Thread.Sleep(20000); var log = new EventLog("System", Environment.MachineName, "Microsoft Antimalware"); foreach(EventLogEntry entry in log.Entries) { if(entry.InstanceId == 1116 && entry.TimeWritten > DateTime.Now.Subtract(new TimeSpan(0, 2, 0))) { if(entry.Message.Contains(value: fuEndpointProtection.FileName.ToLower())) { Label1.Text = "File has been found to be malware and quarantined!"; return; } } } Label1.Text = path;
When I upload a normal file
When I upload the Eicar test file
The eventlog entry should look like this, which contains details on the affected process, the fact that it is a virus and also some indication on where to get some more information by providing a threat URL.%%860 4.0.1521.0 {872DA7D0-383A-4A18-A447-DC4C7E71785F} 2012-08-12T09:31:18.362Z 2147519003 Virus:DOS/EICAR_Test_File 5 Severe 42 Virus http://go.microsoft.com/fwlink/?linkid=37020&name=Virus:DOS/EICAR_Test_File&threatid=2147519003 3 2 3 %%818 D:\Windows\System32\inetsrv\w3wp.exe NT AUTHORITY\NETWORK SERVICE containerfile:_F:\sitesroot\eicar_com.zip;file:_F:\sitesroot\eicar_com.zip->(Zip);file:_F:\sitesroot\eicar_com.zip->eicar.com 1 %%845 1 %%813 0 %%822 0 2 %%809 0x00000000 The operation completed successfully. 0 0 No additional actions required NT AUTHORITY\SYSTEM AV: 1.131.1864.0, AS: 1.131.1864.0, NIS: 0.0.0.0 AM: 1.1.8601.0, NIS: 0.0.0.0Okay, so this is very tamed example but it does prove the concept. In the real world you may even want to have a proper gateway which acts as a proxy and then forwards the file onto a "checked" store if it succeeds. We looked at the two ways you can check to see whether the file has been treated as malware. The first, checking to see whether the file has been deleted from it’s location is too non-deterministic because although "real time" means real time we don’t want to block and wait and timeout on this. The second is better because we will get a report if it’s detected. This being the case, a more hardened version of this example will entail building a class which may treat the file write as a task and asynchronously ping back the user if the file has been treated as malware – something like this could be written as an HttpModule or ISASPI filter pursue the test and either continue with the request or end the request and return an HTTP error code to the user with a description of the problems with the file.
<Return to section navigation list>
Visual Studio LightSwitch and Entity Framework 4.1+
Paul van Bladel (@paulbladel) described Saving data async with cancellation support in an 8/13/2012 post:
Introduction
I can’t get enough from the async calls in LightSwitch. Today I’m discovering a well known feature in WCF Ria Services: cancellation support, which is perfectly supported in LightSwitch as well.
Please read first my previous post on loading data async with cancellation support: http://blog.pragmaswitch.com/?p=394.
This post is quite analogue and focuses on the save pipeline.
The setup
I have a command table and a fictious table called CommandSideEffect. When a command is inserted, I create 1000 records in the CommandSideEffect table :
partial void Commands_Inserting(Command entity) { ApplicationData appData = this.Application.CreateDataWorkspace().ApplicationData; foreach (CommandSideEffect item in appData.CommandSideEffects) { item.Delete(); } appData.SaveChanges(); int totalRecords = 1000; DateTime timeStamp = DateTime.Now; for (int i = 0; i < totalRecords; i++) { CommandSideEffect sideEffectRecord = appData.CommandSideEffects.AddNew(); sideEffectRecord.Effect = i.ToString() + " " + timeStamp; } appData.SaveChanges(); }The user has 2 buttons on the command listdetail screen: Execute command async and Cancel Command.
The client side code
public partial class CommandsListDetail { ISubmitOperationInvocation _saveChangesInvocation; partial void ExecuteCommandAsync_Execute() { Command command = this.Commands.AddNew(); command.Verb = "RunCommand"; _saveChangesInvocation.ExecuteAsync(); } partial void ExecuteCommandAsync_CanExecute(ref bool result) { result = _saveChangesInvocation != null && _saveChangesInvocation.CanExecute; } partial void CommandsListDetail_InitializeDataWorkspace(List<IDataService> saveChangesTo) { _saveChangesInvocation = this.DataWorkspace.ApplicationData.Details.Methods.SaveChanges.CreateInvocation(new object[] { }); _saveChangesInvocation.ExecuteCompleted += new EventHandler<ExecuteCompletedEventArgs>((s1, e1) => { this.Details.Dispatcher.BeginInvoke(() => { switch (_saveChangesInvocation.ExecutionState) { case ExecutionState.Cancelled: this.DataWorkspace.ApplicationData.Details.DiscardChanges(); break; case ExecutionState.Executed: break; case ExecutionState.Executing: break; case ExecutionState.HasError: break; case ExecutionState.NotExecuted: break; default: break; } this.ShowMessageBox(GetStateMessage(_saveChangesInvocation.ExecutionState)); }); }); } private string GetStateMessage(ExecutionState state) { return "Save : " + state.ToString(); } partial void CancelCommandExecution_Execute() { if (_saveChangesInvocation != null) { _saveChangesInvocation.ExecuteAsyncCancel(); } } partial void CancelCommandExecution_CanExecute(ref bool result) { result = _saveChangesInvocation != null && _saveChangesInvocation.CanExecuteAsyncCancel; } }Disclaimer
As for cancellation in the query pipeline, the cancellation is only a client side matter: when a save is cancelled, the command execution will continue server side but the client will no longer wait for the result. In the query pipeline this normally does not lead to collateral damage, but this is not true for the save pipeline. In case you want to undo also the effects of the server side processing as a result of the command triggering you will need to do some more steps: you should implement a “compensating transaction” mechanism, which you will trigger probably also via the command mechanism, but that story is a bit more involved
Paul van Bladel (@paulbladel) described Loading data in LightSwich async with cancellation support. in an 8/13/2012 post:
Introduction
I’m showing in the blog post a technique which is completely useless in the context of my simplified example but which could be useful in other circumstances.
WCF Ria services has out-of-the-box support for client side cancellation. This means that you can decide to cancel client side the load operation which has been started asynchronously.
A UI prototype
Since I’m focusing here rather on a technique than a practical solution, we simply stick to a very rudimentary UI. We have customer search screen with 2 buttons: “Load Customers Async” and “Cancel Load”.
Our MVVM mind wants of course that the buttons are enabled appropriately: when a load operation is busy, the load button should be disabled and the Cancel Load button should be enabled. When no load operation is taking place, the cancel load button should be disabled.
How?
The trick is to cast the loader of the ScreenCollection to an IScreenCollectionPropertyLoader, which derives from IExecutable.
The cool thing is that an IExecutable supports async cancellation.
public interface IExecutable { bool CanExecute { get; } bool CanExecuteAsync { get; } bool CanExecuteAsyncCancel { get; } Exception ExecutionError { get; } ExecutionState ExecutionState { get; } event EventHandler<ExecuteCompletedEventArgs> ExecuteCompleted; void Execute(); void ExecuteAsync(); void ExecuteAsyncCancel(); }Take a look at the implementation:
partial void SearchCustomers_InitializeDataWorkspace(List<IDataService> saveChangesTo) { _loader = this.Details.Properties.Customers.Loader as IScreenCollectionPropertyLoader; _loader.ExecuteCompleted += new EventHandler<ExecuteCompletedEventArgs>((s, e) => { this.Details.Dispatcher.BeginInvoke(() => { string message = "all records successfully loaded"; if (!_loader.IsLoaded) { message = "opeation canceled by user"; } this.ShowMessageBox(message); }); }); } partial void CancelLoad_Execute() { _loader.ExecuteAsyncCancel(); } partial void CancelLoad_CanExecute(ref bool result) { result = _loader != null && _loader.CanExecuteAsyncCancel; } partial void LoadCustomersAsync_Execute() { _loader.ExecuteAsync(); } partial void LoadCustomersAsync_CanExecute(ref bool result) { result = _loader != null && _loader.CanExecuteAsync; }What’s happening?
It is very important to have a clear understanding what is happening here: client side cancellation means that you are no longer interested in the result of the load operation, but this does not mean that the server is informed about this cancellation. So, the server will still process the request as if there was no cancellation, it will return the response, but the client will not use it because it’s no longer waiting for it. You can verify easily with fiddler that, for a cancelled load operation, the response is still going over the wire. In case you want simulate a long running load operation, put a Thread.Sleep(5000) in the PreProcessQuery method.
Return to section navigation list>
Windows Azure Infrastructure, Media Services and DevOps
Lori MacVittie (@lmacvittie) asserted “Maintaining Consistent Performance of Elastic Applications in the Cloud Requires the Right Mix of Services” in an introduction to her Curing the Cloud Performance Arrhythmia post of 7/13/2012 to F5’s DevCenter blog:
Arrhythmias are most often associated with the human heart. The heart beats in a specific, known and measurable rhythm to deliver oxygen to the entire body in a predictable fashion. Arrhythmias occur when the heart beats irregularly. Some arrhythmias are little more than annoying, such as PVCs, but others can be life-threatening, such as ventricular fibrillation. All arrhythmias should be actively managed.
Inconsistent application performance is much like a cardiac arrhythmia. Users may experience a sudden interruption in performance at any time, with no real rhyme or reason. In cloud computing environments, this is more likely, because there are relatively few, if any, means of managing these incidents.
A 2011 global study on cloud conducted on behalf of Alcatel-Lucent showed that while security is still top of mind for IT decision makers considering cloud computing, performance – in particular reliable performance – ranks higher on the list of demands than security or costs.
THE PERFORMANCE PRESCRIPTION
One of the underlying reasons for performance arrhythmias in the cloud is a lack of attention paid to TCP management at the load balancing layer. TCP has not gotten any lighter during our migration to cloud computing and while most enterprise implementations have long since taken advantage of TCP management capabilities in the data center to redress inconsistent performance, these techniques are either not available or simply not enabled in cloud computing environments.
Two capabilities critical to managing performance arrhythmias of web applications are caching and TCP multiplexing. These two technologies, enabled at the load balancing layer, reduce the burden of delivering content on web and application servers by offloading to a service specifically designed to perform these tasks – and do so fast and reliably.
In doing so, the Load balancer is able to process the 10,000th connection with the same vim and verve as the first. This is not true of servers, whose ability to process connections degrades as load increases, which in turn necessarily raises latency in response times that manifests as degrading performance to the end-user. Test
Failure to cache HTTP objects outside the web or application server has a similar negative impact due to the need to repetitively serve up the same static content to every user, chewing up valuable resources that eventually burdens the server and degrades performance.
Caching such objects at the load balancing layer offloads the burden of processing and delivering these objects, enabling servers to more efficiently process those requests that require business logic and data.
FAILURE in the CLOUD
Interestingly, customers are very aware of the disparity between cloud computing and data center environments in terms of services available.
In a recent article on this topic from Shamus McGillicuddy, "Tom Hollingsworth, a senior network engineer with United Systems, an Oklahoma City-based value-added reseller (VAR). "I want to replicate [in the cloud with] as much functionality [customers] have for load balancers, firewalls and things like that."
So why are cloud providers resistant to offering such services?
Shamus offered some insight in the aforementioned article, citing maintenance and scalability as inhibitors to cloud provider offerings in the L4-7 service space. Additionally, the reality is that such offload technologies, while improving and making more consistent performance of applications also have a side effect of making more efficient the use of resources available to the application. This ultimately means a single virtual instance can scale more efficiently, which means the customer needs fewer instances to support the same user base. This translates into fewer instances for the provider, which negatively impacts their ARPU (Annual Revenue Per User) – one of the key metrics used to evaluate the health and growth of providers today.
But the reality is that providers will need to start addressing these concerns if they are to woo enterprise customers and convince them the cloud is where it’s at. Enabling consistent performance is a requirement, and a decade of experience has shown customers that consistent performance in a scalable environment requires more than simple load balancing – it requires the very L4-7 services that today do not exist in provider environments.
Lori MacVittie (@lmacvittie) asserted “Cloud Integrating environments occurs in layers …” in an introduction to her The Cloud Integration Stack article of 8/8/2012 for F5’s DevCentral blog:
We use the term “hybrid cloud” to indicate a joining together of two disparate environments. We often simplify the “cloud” to encompass public IaaS, PaaS, SaaS and private cloud. But even though the adoption of such hybrid architectures may be a foregone conclusion, the devil is, as they say, in the details and how that adoption will be executed is not so easily concluded.
At its core, cloud is about integrating infrastructure. We integrate infrastructure from the application and networking domains to enable elasticity and scalability. We integrate infrastructure from security and delivery realms to ensure a comprehensive, secure delivery chain that promises performance and reliability. We integrate infrastructure to manage these disparate worlds in a unified way, to reduce the burden on operations imposed by necessarily disconnected systems created by integrating environments.
How these integrations are realized can be broken down into a fairly simple stack comprised of the network, resources, elasticity, and control.
The NETWORK INTEGRATION LAYER
At the network layer, the goal is normalize connectivity and provide optimization of network traffic between two disconnected environments. This is generally applicable only to the integration of IaaS environments, where connectivity today is achieved primarily through the use of secured network tunnels. This enables secure communications over which data and applications may be transferred between environments (and why optimization for performance sake may be desired) and over which management can occur. The most basic of network integration enabling a hybrid cloud environment is often referred to as bridging, after the common networking term.
Bridging does not necessarily imply layer 3 normalization, however, and some sort of overlay networking technology will be required to achieve that normalization (and is often cited as a use of emerging technology like SDN).
Look for solutions in this layer to be included in cloud “bridges” or “bridging” offerings.
The RESOURCE INTEGRATION LAYER
At the resource layer, integration occurs at the virtualization layer. Resources such as compute and storage are integrated with data center residing systems in such a way as to be included in provisioning processes. This integration enables visibility into the health and performance of said resources, providing the means to collect actionable performance and status related metrics for everything from capacity planning to redistribution of clients to the provisioning of performance-related services such as acceleration and optimization.
This layer of integration is also heavily invested in the notion of maintaining operational consistency. One way this is achieved is by integrating remote resources into existing delivery network architectures that allow the enforcement of policy to ensure compliance with operational and business requirements.
Another means of achieving operational consistency through resource integration is to integrate remotely deployed infrastructure solutions providing application delivery services. Such resources can be integrated with data center deployed management systems in such a way as to enforce operational consistency through synchronization of policies across all managed environments, cloud or otherwise.
Look for solutions in this layer to be included in cloud “gateway” offerings.
The ELASTICITY INTEGRATION LAYER
Elasticity integration is closely related to resource integration but not wholly dependent upon it. Elasticity is the notion of expanding or contracting capacity of resources (whether storage, network, or compute) to meet demand. That elasticity requires visibility into demand (not as easy as it sounds, by the way) as well as integration with the broader systems that provision and de-provision resources.
Consider a hybrid cloud in which there is no network or resource integration, but rather systems are in place to aggregate demand metrics from both cloud and data center deployed applications. When some defined threshold is met, a trigger occurs that instructs the system to interact with the appropriate control-plane API to provision or de-provision resources. Elasticity requires not only the elasticity of compute capacity, but may also require network or storage capacity be adjusted as well. This is the primary reason why simple “launch a VM” or “stop a VM” responses to changes in demand are wholly inadequate to achieve true elasticity – such simple responses do not take into consideration the ecosystem that is cloud, regardless of its confines to a single public provider or its spread across multiple public/private locations.
True elasticity requires integration of the broader application delivery ecosystem to ensure consistent performance and security across all related applications.
Look for solutions in this layer to be included in cloud “gateway” offerings.
The CONTROL INTEGRATION LAYER
Finally, the control integration layer is particularly useful when attempting to integrate SaaS with private cloud or traditional data center models. This is primarily because integration at other layers is virtually non-existent (this is also true of PaaS environments, which are often highly self-contained and only truly enable integration and control over the application layer).
The control layer is focused on integrating processes, such as access and authentication, for purposes of maintaining control over security and delivery policies. This often involves some system under the organization’s control (i.e. in the data center) brokering specific functions as part of a larger process. Currently the most common control integration solution is the brokering of access to cloud hosted resources such as SaaS. The initial authentication and authorization steps of a broader log-in process occur in the data center, with the enterprise-controlled systems then providing assurance in the form of tokens or assertions (SAML, specifically crafted encrypted tokens, one time passwords, etc…) to the resource that the user is authorized to access the system.
Control integration layers are also used to manage disconnected instances of services across environments for purposes of operational consistency. This control enables the replication and synchronization of policies across environments to ensure security policy enforcement as well as consistent performance.
Look for solutions in this layer to be included in cloud “broker” offerings.
Eventually, the entire integration stack will be leveraged to manage hybrid clouds with confidence, eliminating many of the obstacles still cited by even excited prospective customers as reasons they are not fully invested in cloud computing.
Steve Fox (@redmondhockey) described SharePoint 2013 and Windows Azure Media Services in an 8/4/2012 post (missed when published):
One of the cool new kids on the block for Windows Azure is Media Services. Windows Azure Media Services (WAMS) is currently in Beta, and as I dig in more I personally feel there’s a ton of potential here—especially when it comes to SharePoint. For those that don’t know what WAMS is all about, it’s a slick set of services and APIs within the Azure platform that enable you to do things like upload videos, encode into new file formats/codecs, save to BLOB storage, set permissions for consumption, stream BLOBs with global cache, and so on. Here’s the thing: you can do this all through a set of cloud-based APIs, which when compared to previous media management workflows and processes buys you quite a bit. Using WAMS, you can not only optimize your multimedia processing and management, but you can build some pretty darn compelling apps in the process.
Windows Azure Media Services
So, why should you give a rip, right? Well, what’s interesting is that there’s been a relative constant when it comes to the adoption of SharePoint: in many organizations it’s being used as a multimedia learning platform (or even a corporate communications platform). That is, the document libraries, lists and ability to manage sites and media within those sites give you the ability to build out multimedia solutions that are built on SharePoint. It’s just easy to integrate multimedia with the ability to manage sites, documents, and permissions against those artifacts. However, what’s been tricky has been the ability to process and manage the multimedia that sits behind those solutions. Think creating and projecting a WMV format across 3-4 different other codecs; think having a common streaming point where we can assess permissions using the native SharePoint security authentication; and think about having a set of services and APIs that can now allow you to not only publish to SharePoint, but also project cross-device.
Some interesting scenarios that bring these two technologies together are:
- Organizational Web sites
- Learning solutions built on SharePoint
- Corporate communications
- Digital marketing campaigns in SharePoint FIS
- Multimedia publishing platform
- Social networking/community web sites
- …
In short, there’s lots of potential here.
WAMS & SharePoint
Now for readers of this blog, you know I’ve focused a lot on the Azure and SharePoint kick; albeit I’ve focused a lot on SharePoint 2010. However, with the veil lifting on what SharePoint 2013 has to offer, all of a sudden not only is the integration of SharePoint and Windows Azure native, but it also is very compelling (and know that I’ll be focusing some energy on this moving forward). For example, for those of you who are in the SharePoint community, I’m sure you’re now up to speed on the fact that the future direction for SharePoint app development is the cloud; it’s the guidepost for future app design and deployment and is somewhat of a shift from what we’ve come to understand in SharePoint 2010. That is, within SharePoint 2013, there exist a couple of ways to build and deploy apps within a cloud-hosted app model: the first is the autohosted app; and the second is the provider-hosted app. The autohosted app model is an inclusive cloud model, where you build an app where configuration information lives in SharePoint and the core server-side code lives in Windows Azure (but through deployment they are registered and live as one). The provider-hosted app model is much more flexible. You can deploy code to any web platform, including Windows Azure, and then register that app with SharePoint.
So, the question becomes how we begin to mesh this interesting world of WAMS with the (new and existing) world of SharePoint—thus, the integration of media and SharePoint. To help illustrate, let’s walk through an example. And in this walkthrough, we’ll do two things:
1. Build out an HTML5 cloud-based app that leverages a video streamed from Azure Media Services; and
2. Create a simple SharePoint 2013 “Client App” that surfaces the media within SharePoint.
Yes, there is an assumption here about managing and consuming media from WAMS. What about the upload? What about the transcode? What about the management of media? Well, it’s all here. Using the SDK and How-to’s, you can build yourself a handy-dandy app that moves your apps into your Azure account and then makes them available for use within other solutions or platforms like SharePoint. And it’s within these how-to’s and the SDK that you’ll hopefully begin to see the power of WAMS. But this is a blog, and we must scopeth that which we delivereth. Thus, the assumption with this walkthrough is that you’ve followed the aforementioned tutorials and have created a WMV, uploaded and encoded it as a MP4, and have then saved it to Blob storage so you can stream it from Azure. So, we start from a video that is ready to be streamed from WAMS, e.g. http://myvidsite.cloudapp.net/home/default.html.
Building an HTML-based Media App
You can create a Windows Azure app using the .NET 4.0 framework and all the latest and greatest Azure tools/SDK (v. 1.7). To do this:
- Open Visual Studio 2012.
- Click File, New Project.
- Select Cloud and then select Windows Azure Cloud Service.
- Provide a name for the project and click OK.
- When prompted, select the ASP.NET MVC 4 Web Role, and then edit the name of your role to make it more intuitive.
- Click OK.
- At this point, Visual Studio creates the project structure and default code spit for you.
At this point, I removed a bunch of the default project goo—such as the ASPX pages, App_Data folder, etc. You’ll note that I added a set of supplement libraries/resources for my HTML video player, e.g. a CSS folder with a playerframework.min.css file (and debug file), a JS file added to the Scripts folder called playerframework.min.js (and again a debug file), and then focused on the heart of the Azure app by creating the default.html page—which will load the BLOB from Azure (or WAMS). (You could probably trim even more if you wanted from the default project, but I chose to stop here.)
- To hit the default.html page when you’re debugging, right-click and select Set as Start Page.
- Within the default.html page, add code that resembles the following snippet of code. (What will vary in your code will be the bolded URI that points to your video.)
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server"><title>HTML5 Video Player</title>
<link rel="stylesheet" href="../CSS/playerframework.min.css"/>
<script type="text/javascript" src="../Scripts/playerframework.min.js"></script>
</head>
<body>
<form id="form1" runat="server">
<div>
<video id="myVideo" class="pf-video" width="480" height="320" controls="controls">
type=’video/mp4;
codecs="avc1.42E01E, mp4a.40.2"’ /></video>
<script type="text/javascript">
var myPlayer = new PlayerFramework.Player("myVideo");
</script>
</div>
</form>
</body>
</html>
Now there’s a couple of things of interest here. The first is that we’re using HTML5 and the <video> tag to load and play our Azure-hosted video. (If you’re looking for a good primer on the video tag, check this blog-post out.) The second is that you can see that we’re not only referencing the MP4 file (http://myblobstore.blob.core.windows.net/html/BigBuckBunnyMP4.mp4) but we’ve also got some strange GUID thingy (sr=c&si=new&sig=5CxDGrmtYndAm8bhdwITw7tlABlZFTxtsL%2BfR051Ngg%3D) at the end of the URL: this is the access signature that you create programmatically using the WAMS API—or you can use the Cloud Storage Explorer to do some of this manually (also known as an Access Policy ID). This indicates the level of access you can set for consumers for a specific video asset. For example, if you click on an MP4 asset in your Cloud Explorer view, and then click Security, you’ll have the ability to manually set the shared access permissions options.
If you then click the Shared Access Signatures tab, you’ll be able to leverage a Shared Access Policy that you’ve created for your BLOB, and you can click the Generate Signature button, which creates a full BLOB URL for your video. You can then Copy to Clipboard and then paste in a browser to load the video and stream from Azure. (As per this post, ensure you have the correct MIME type; else, this will cause a failure for your video to load properly.)
You can see that when you click the Shared Access Policies tab you can create new policies with specific access rights and start and expiration times—all of which are enforced against your video BLOB.
While not complete, the below excerpt from the WAMS SDK gives you a sense for how you can do some of the above programmatically:
static String GetAssetSasUrl(IAsset asset, TimeSpan accessPolicyTimeout)
{
IAccessPolicy readPolicy = _context.AccessPolicies.Create("My Test Policy", accessPolicyTimeout, AccessPermissions.Read);
ILocator locator = _context.Locators.CreateSasLocator(asset, readPolicy, DateTime.UtcNow.AddMinutes(-5));
…
var theOutputFile = from f in asset.Files
where f.Name.EndsWith(".mp4")
select f;
IFileInfo theFile = theOutputFile.FirstOrDefault();
string fileName = theFile.Name;
var uriBuilder = new UriBuilder(locator.Path);
uriBuilder.Path += "/" + fileName;
…
return uriBuilder.Uri.AbsoluteUri;
}
- When you’re done adding the HTML and required resources, hit F5 to debug. You should see something similar to the below: your video being streamed from WAMS within your HTML5 video player.
- The next step is to publish your app to Windows Azure. To do this, right-click your Cloud project and select Publish. Then, publish the application to your Windows Azure subscription/account. This will then make the app accessible from the wider Web, hosted on Windows Azure.
- Once you’ve published your HTML5 video player app that consumes the video being streamed from WAMS, you are now ready to move onto the next step: creating a simple SharePoint 2013 app that surfaces the Azure app.
Integrating the Azure App with SharePoint 2013
In the first exercise, you created and deployed an HTML5-based video player that was hosted on Windows Azure and leveraged the core WAMS features to upload, transcode, store, and stream a video (although this WAMS-specific functionality was a pointer to the SDK). Where you should have left off was having a deployed app where the HTML5-based video loaded for you from Windows Azure. In essence, a URL not unlike the following: http://myvidsite.cloudapp.net/home/default.html. Now, you’ll use the new SharePoint 2013 app model to integrate the Azure app to SharePoint. You’ll use the autohosted Client App template to do this. (Note: to get started with Office 2013 development, visit the MSDN Developer page.)
- Open Visual Studio 2012, and click File, New Project.
- Select Office/SharePoint, and then select Apps and App for SharePoint 2013.
- Provide a name and click OK.
- In the next dialog in the wizard, select the SharePoint site you’ll be deploying the app to, and then select Autohosted in the drop-down list.
- Visual Studio will create a project comprising two parts: one part configuration and another part web app code. Delete the “..Web” part of the project.
- Right-click the project, and then select Add and then New Item. Select “Client Web Part (Host Web),” provide a name for the new Client Web Part and click Add. You can see below, I added one called “MyAzureVidPlayer” to the project.
- Now you’ll want to copy the URL to the Azure video player page and configure two parts of the project.
- First, open the Elements.xml file and amend the Content element, as per the bolded XML code within the snippet below.
<?xml version="1.0" encoding="utf-8"?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
<ClientWebPart Name="MyAzureVidPlayer" Title="MyAzureVidPlayer Title" Description="MyAzureVidPlayer Description" DefaultWidth="300" DefaultHeight="200"><Content Type="html" Src="http://myvidsite.cloudapp.net/home/default.html" />
</ClientWebPart>
</Elements>
- Second, right-click the AppManifest.xml file and click View Code. Amend the XML file as per the bolded code so it looks like the below:
<?xml version="1.0" encoding="utf-8" ?>
<App xmlns="http://schemas.microsoft.com/sharepoint/2012/app/manifest"Name="MyFirstAzureVidPlayer"
ProductID="{70e24eec-9734-4b8d-b084-5851ff7be7c4}"Version="1.0.0.0"
SharePointMinVersion="15.0.0.0">
<Properties>
<Title>MyFirstAzureVidPlayer</Title>
<StartPage>http://myvidsite.cloudapp.net/home/default.html</StartPage>
</Properties>
<AppPrincipal>
<AutoDeployedWebApplication/></AppPrincipal>
<AppPrerequisites>
<AppPrerequisite Type="AutoProvisioning" ID="RemoteWebHost" /></AppPrerequisites>
<AppPermissionRequests><AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read" /></AppPermissionRequests></App>
- Now you can hit F6 to build, and then right-click the project and select Publish. Publishing the app will build an autohosted cloud app that you can deploy to your SharePoint site. When the project is done building, copy the auto-opened Windows Explorer path for use in the next step.
- Now, navigate to your SharePoint site. In this case, I’m using SharePoint Online, so I clicked Apps in Testing and then selected new app to deploy, which enables me to upload the .app I just built (you can now click Browse and paste in the folder path from the previous step).
- Once you’ve uploaded the .app, click Deploy and then hit the Trust It button to trust the app. This deploys the Azure app to SharePoint but integrates them natively using the new SharePoint 2013 (autohosted) cloud model.
- To get your app onto a page, click Page, Edit, Insert, and then select App Part and then choose the app part you just deployed. And voila, your WAMS video player app (hosted in Azure) now shows up as an integrated app in SharePoint.
Congratulations! You’ve now build your first WAMS and SharePoint 2013 integrated app using the new cloud app model.
Final Thoughts
In this blog-post, I showed you separately how to create the Azure app and how to create the SharePoint app; although, breaking them apart wasn’t for naught. Hopefully, you now can see that while there may be separate pieces to the cloud puzzle, they are increasingly coming together as a united front—whether it be through concerted templates/functionality or by integrating Azure and SharePoint natively. (Note that by using the Provider hosted template in SharePoint 2013, you’ll be able to pull the above exercise together within one Visual Studio deployment.) And what’s interesting is that while here we talked about Windows Azure and WAMS, it’s about the wider web being your play-ground; it’s about being able to not only create a great Azure media app, but also integrate a great HTML5 or PhP app with SharePoint as well. But, WAMS is hugely compelling; especially from the process and workflow perspective. Again, within this blog I pointed to the SDK as an example of this, which makes the after-integration of WAMS and SharePoint look easy—which is where I focused.
I encourage you to play around with the WAMS SDK. When thinking about SharePoint, there are some interesting integrations you can accomplish with both 2010 and 2013. Expect to see more here.
Dan Turkenkopf (@dturkenk) posted East Coast vs. West Coast PaaS Psychology. And Why it Matters to the Apprenda blog on 7/31/2012 (missed when published)
The first time I heard someone differentiate between East Coast and West Coast Platform as a Service (PaaS), something clicked. Over the past year or so, I’ve had more than one “polite discussion” about what organizations should be looking for in a platform. It usually seems like we are talking past each other rather than to each other.
James Urquhart of EnStratus, Master of Ceremonies at DeployCon, was the first thought leader I’ve heard suggest the theory that distinct cultural viewpoints might be the cause of so much controversy in the PaaS ecosystem. While external manifestations of different approaches were argued in great detail that day, the underlying divide wasn’t.
On the heels of that talk, Rodrigo Flores of Cisco explored a similar divide between “Silicon Valley” PaaS and “Enterprise” PaaS. To Flores, Silicon Valley PaaS consists of a “black-box” application hosting platform. Enterprise PaaS, he theorizes, is more a managed set of composable application stack components that consumers can select to assemble whatever application hosting stack needed.
I definitely appreciate the attempt to identify two different approaches to PaaS, I still suspect that this dichotomy misses the mark slightly.The biggest issue with Flores’ split is that his Enterprise PaaS, while being valuable in many circumstances, doesn’t provide the core values to look for in a platform as a service.
The two biggest requirements – which is the essential essence of a PaaS – are to treat the application as the unit of abstraction, and to hide the underlying infrastructure from the consumer.
Making the application THE first-class citizen acknowledges that the business value delivered by the application is the true goal of IT, and puts the developer in control of ensuring that value is realized.
The second requirement follows from the first. If all consumer interaction with the platform occurs within the scope of the application, then there really is no need to understand what’s beneath the application. The conversation is raised to capabilities and capacity, rather than servers and setup.
This is why you’ll see the same set of basic functionality in just about every platform as a service you look at. Nearly all of them provide developers self-service application management and deployment and the ability to leverage value-adding services offered by the platform or by third parties. None of them require the developer to access the underlying virtual machine or operating system to perform her task. And most offer some level of health monitoring and application resiliency simply by running on the platform.
Once these basic needs are met, you start to see the West/East divide form; the most apparent manifestation being public PaaS versus private PaaS.
The West Coast mindset (to use Urquhart’s term and expand out of just Silicon Valley to Seattle and Portland) predicts a much more rapid transition to a fully public cloud where IT operations are essentially outsourced to cloud providers.
The East Coast mentality recognizes there are some use cases where running in the public cloud makes sense, but is hesitant to give up the control of their IT; whether for regulatory/security concerns (be they real or imagined), or because of large existing investments in infrastructure and staff, or due to large amounts of business critical data.
I’m not going to explore the underlying psychology behind the split beyond suggesting it has something to do with where technology supports the business versus where technology IS the business.
If your goal in life as a PaaS vendor is to become a service provider and take over IT operations for companies, gaining economies of scale is probably the most important thing you can do. And to get economies of scales, you need to draw as many developers as you possibly can. Make the barriers of entry low, and offer support for a wide variety of languages and frameworks. But, because you need to support so many different flavors of technology to gain the mass audience you need, you can only offer fairly limited support for each of them. It might not be truly the lowest common denominator, but it’s not far off. Trying to do much more will likely lead to product fragmentation.
That means that most, if not all, the West Coast PaaS flavors, provide the definitional capabilities of self-service deployment and location transparency, but can’t go much deeper than that. And there can be lots of value in that, especially for start ups and individual developers. But there’s also a limit to that value.
East Coast PaaS, on the other hand, tries to support the entire organization – of which developers are the most important piece. It’s often delivered in a private or hybrid form factor, and relies on the existing organizational capabilities of the IT staff to manage and maintain a robust delivery infrastructure. There’s no need to build up economies of scale just for the PaaS – because they’re already built into the traditional enterprise offerings.
What East Coast PaaS offers is a renewed focus on developers, giving the same self-service application management and location transparency that West Coast PaaS provides. But, almost paradoxically, because East Coast PaaS providers recognize and support the IT operations for enterprises, they can provide deeper value to the developer. Enterprises generally have standardized a few stacks (Java and .NET for the most part), which means the PaaS providers can remain intensely focused on those technologies and leverage their advanced capabilities to enhance the developer experience. For example, by augmenting capabilities in the .NET stack, Apprenda can dynamically transform an application into a SaaS application at deployment time, saving upwards of a year of effort. It’s unlikely you’ll ever see those sorts of capabilities evolve in a polyglot world. …
Dan continues with an analysis of the polyglot developer world.
I’m definitely a member of the West Coast PaaS camp.
Kristian Nese (@KristianNese) described Cloud Services in Windows Azure in an 8/11/2012 post:
As a part of the new offerings in Windows Azure, Hosted Services are now replaced with Cloud Services in the new Windows Azure portal.
A hosted service was previously a service in Azure that could contain Web Roles, Worker Roles and VM Roles.
With the new Virtual Machine (persistent) in Azure, you can also add them to a cloud service so that they can communicate in their private network.
This is important to know if you’re planning to extend your infrastructure and create connectivity between resources on-premise and in Windows Azure. Instead of going through the external IP/DNS name, you can take advantage of this private network.
So let’s repeat the PaaS service model in Windows Azure
A hosted service in Windows Azure was basically a combination of code and configuration. This does still apply for the cloud service.
A cloud service represents the PaaS service model in Azure, where you can deploy your multi-tier applications, using multiple roles and have a flexible model to scale your stateless applications.
Each role (Web or/and Worker Role) has its own code and configuration file.
So from a developer’s perspective, they only need to concentrate on their code, and let Windows Azure’s eco-system take care of the underlying architecture for the infrastructure and maintain performance, patching of the operating system and general maintenance in case of a failure.
Based on the SLA’s available in Azure, you must specify at least two instances of each role to assure you meat a satisfied SLA. This will apply to both failures and when you’re servicing your service.
This is to guarantee external connectivity to your internet-facing roles 99.95% of the time.
If you have worked with System Center 2012 – Virtual Machine Manager, you may be aware of the service concept where you can deploy distributed applications, use load balancing and scale out the stateless instances, and specify upgrade domains. Windows Azure has something similar, and provides you with two environments.
The staging environment is where you can test your cloud service before you put it into your production environment. When you are satisfied with your service, you can easily do a VIP swap (swapping the virtual IP address that’s associated with the two environments).
I’ll blog more about Azure over the next weeks.
<Return to section navigation list>
Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
My Configuring Windows Azure Services for Windows Server post of 8/14/2012 begins:
Contents:
Introduction
The Microsoft Hosting site describes a new multi-tenanted IaaS offering for hosting service providers that the Windows Server team announced at the Worldwide Partners Conference (WPC) 2012, held in Houston, TX on 7/8 through 7/12/2012:
The new elements of Windows Azure Services for Windows Server 2008 R2 or 2012 (WAS4WS) are the Service Management Portal and API (SMPA); Web Sites and Virtual Machines are features of Windows Azure Virtual Machines (WAVM), the IaaS service that the Windows Azure team announced at the MEET Windows Azure event held in San Francisco, CA on 6/7/2012.
Licensing Requirements
Although Hosting Service Providers are the target demographic for WAS4WS, large enterprises should consider the service for on-site, self-service deployment of development and production computing resources to business units in a private or hybrid cloud. SMPA emulates the new Windows Azure Management Portal Preview, which also emerged on 6/7/2012.
When this post was written, WAS4WS required a Service Provider Licensing Agreement:
Licensing links:
Any Microsoft partner can join the Microsoft Hosting Community (I’ve tried to join but don’t receive the promised email response.):
More details are available on the Microsoft partner-hosted services page.
Full disclosure: OakLeaf Systems is a Microsoft Partner.
Memo to Microsoft: Provide licensing for organizations that aren’t commercial hosting services to use WAS4WS for creating internal (private) or hybrid clouds for business units without the SPLA requirement. This version should support Active Directory and, optionally, Windows Azure Active Directory (WAAD), as well as Remote Desktop Services (RDS) and Remote Web Access (RWA). Presumably, RDS and RWA would require purchasing Windows Server Azure 2008 R2 Remote Desktop Services (5-User Client Access License) CALs, US$749.00 from the Microsoft Store
Note: WAS4WS isn’t related to the elusive Windows Azure Platform Appliance (WAPA), which Microsoft introduced in July, 2010 and later renamed the Windows Azure Appliance (see Windows Azure Platform Appliance (WAPA) Announced at Microsoft Worldwide Partner Conference 2010 of 6/7/2010 for more details.) To date, only Fujitsu has deployed WAPA to a data center (see Windows Azure Platform Appliance (WAPA) Finally Emerges from the Skunk Works of 6/7/2011.) WAS4WS doesn’t implement Windows Azure Storage (high-availability tables and blobs) or other features provided by the former Windows Azure App Fabric, but is likely to be compatible with the recently announced Service Bus for Windows Server (Service Bus 1.0 Beta.)
Prerequisites
System Requirements
From the 43-page Getting started Guide: Web Sites, Virtual Machines, Service Management Portal and Service Management API July 2012 Technical Preview (PDF):
The Technical preview is intended to run on a single Hyper-V host with 7 virtual machines. In addition to the virtual machines required for the software, it is expected that there will be a separate server (or servers) in the datacenter running Microsoft SQL Server, MySQL Server, and a File Server (Windows UNC) or NAS device hosting web content.
Hyper-V Host server for Service Management Portal and Web Sites VMs:
- Dual Processor Quad Core
- Operating System: Windows Server 2008 R2 SP1 Datacenter Edition With Hyper-V (64bit) / Windows Server 2012 with Hyper-V (64 bit)
- RAM: 48 GB
- 2 Volumes:
First Volume: 40GB or greater (host OS).Second Volume: 100GB or greater (VHDs).
- Separate SQL server(s) for Web Sites configuration databases and users/web sites databases running Microsoft SQL Server 2008 R2.
- Separate MySQL server version 5.1 for users/web sites databases.
- Either a Windows UNC share or a NAS device acting as a File server to host web site content.
Note: The SQL Server, MySQL Server, and File Server can coexist with each other, and the Hyper-V host machine, but should not be installed in the same VMs as other Web Sites roles. Use separate SQL Server computers, or separate SQL instances, on the same SQL Server computer to isolate the Web Sites configuration databases from user/web sites databases.
A system meeting the preceding requirements is required to meet the high-end (three Web workers and two load balancers) of the following architecture:
Service Management Portal and Web Sites-specific server role descriptions:
- Web Workers – Web Sites-specific version of IIS web server which processes client’s web requests.
- Load Balancer(s) – IIS web server with Web Sites-specific version of ARR which accepts web requests from clients, routes requests to Web Workers and returns web worker responses to clients.
- Publisher – The public version of WebDeploy and an Web Sites-specific version of FTP which provide transparent content publishing for WebMatrix, Visual Studio and FTP clients.
- Service Management Portal / Web Sites Controller – server which hosts several functions:
o Management Service – Admin Site: where administrators can createWeb Sites clouds, author plans and manage user subscriptions.
o Management Service – Tenant Site: where users can signup and
create web sites, virtual machineand databases.
o Web Farm Framework to provision and manage server Roles.
o Resource Metering service to monitor webservers and site resource
usage.
- Public DNS Mappings. (DNS management support for the software is coming in a future release. The recommended configuration for this technical preview is to use a single domain. All user-created sites would have unique host names on the same domain.)
Software Requirements
- Download (as 1 *.exe and 19 *.rar files), extract to a 20+ GB VHD, and install System Center 2012 Service Pack 1 CTP2 – Virtual Machine Manager – Evaluation (VHD) on a Windows Server 2008 R2 (64-bit) or Windows Server 2012 host with the Hyper-V role enabled.
- Download and install System Center 2012 Service Pack 1 Community Technology Preview 2 – Service Provider Foundation Update for Service Management API to the VM created from the preceding VHD.
- Download and run the Web Platform Installer (WebPI, Single-Machine Mode) for the 60 additional components required to complete the VM configuration.
Note: This preview doesn’t support Active Directory for VMs; leave the VMs you create as Workgroup members.
Tip: Before downloading and running the WebPI, click the Configure SQL Server (do this first) button on the desktop (see below) and install SQL Server 2008 R2 Evaluation Edition with mixed-mode (Windows and SQL Server authentication) by giving the sa account a password. Logging in to SQL Server as sa is required later in the installation process (see step 1 in the next section).
The post continues with a detailed, illustrated Configuring the Service Management Portal/Web Sites Controller section.
I’m lobbying for a version of the Service Management Portal that doesn’t require a Service Provider Licensing Agreement (SPLA) for deploying enterprise private clouds with do-it-yourself provisioning for business units. See my recent Will a version of the Management Portal and API be available for Enterprise Users? thread in the Web Sites and Virtual Machines on Windows Server for Hosting Service Providers for more details.
Kevin Kell described Learning Tree’s Implementing a Private Cloud Solution courseware for AWS and Windows Server/System Center in an 8/7/2012 post:
Last week I attended Learning Tree’s “Implementing a Private Cloud Solution” course at our Reston Education Center. It is a great course for anyone seeking in-depth technical details on how to build their own on-premises private cloud. The course also covers using a hosted private cloud solution and building secure connections to your own data center.
This course is not for the faint of heart! It is also not for the technically challenged! When you show up Tuesday morning you need to be prepared to work very hard for the next four days. The course author, Boleslav Sykora, has put together a fast paced session that gives you as much technical detail as you would ever want on the subject. It is the type of course where you will want to come early and stay late each and every day so you can work through all the extensive bonus exercises that are offered. I loved it and I think you will too!
We feature building two private clouds, one using Eucalyptus and another using Microsoft System Center, completely from scratch. There is a lot of Linux command line stuff and quite a bit of detailed networking configuration. This is exactly the reality of what is involved if you want to build your own private cloud. Over the four days you come to understand that private cloud computing is not some mystical, magical hype but is an evolution of solid fundamental concepts that have been around for some time. This course will appeal to technical professionals who want to gain real experience implementing solutions that will define the future of the on-premises data center.
For those who would prefer not to bother with the complexity of an internal private cloud implementation there are many hosted solutions to choose from. Probably the best known is Amazon’s Virtual Private Cloud (VPC). Once you use VPC on Amazon you will likely never go back to using EC2 without it.
In fact as I write this blog I am on a train heading to New York. There I will teach Learning Tree’s “Cloud Computing with Amazon Web Services” course. That, also, is a great course!
Because there are many private cloud implementations based on the Amazon EC2 model and API (particularly Eucalyptus) Amazon has kind of become the de facto standard for how Infrastructure as a Service (IaaS) is done. Even if you believe you would never use a public cloud for a production system there is much to be learned about cloud computing from Amazon. Beyond that the public cloud is a great place to do testing, development and proof-of-concept before investing the time and capital required to build your own private cloud. Public clouds such as Amazon can also become part of a hybrid solution that features the best of what private clouds and public clouds have to offer. Learning Tree’s Amazon Web Services course gives you hands-on experience with many aspects of Amazon’s cloud and shows you how to build solutions using the various services offered there.
So if you are a hardcore techie who wants to have end-to-end control over all aspects of a cloud solution come to Learning Tree’s private cloud course. If you would like to understand how to leverage the Amazon public cloud or to understand the service models of arguably the most dominant cloud provider in the world then come to Learning Tree’s Amazon Web Services course. Either way I hope to see you soon!
<Return to section navigation list>
Cloud Security and Governance
No significant articles today.
<Return to section navigation list>
Cloud Computing Events
Brian Gracely (@bgracely) said he’s Looking Forward to VMworld 2012 in a 7/12/2012 post:
It’s that time again – VMworld. For the 2011 event in the US, I wrote some before and after thoughts on the state of technology, new trends and the event.
Given the actions that VMware has taken recently (new CEO, Software Defined Data Center vision, Nicira acquisition, DynamicOps acquisition, Cetas acquisition, CloudFoundry + OpenStack), I’m going to go out on a limb and say that VMworld 2012 will be one of the most highly watched events that IT has seen for a long time (all Apple announcements not withstanding). It’s not a reach to say that every aspect of VMware will be questioned:
New Leadership – Pat Gelsinger (CEO) takes over a business that must learn how to regain it’s technology leadership in a space that will face intense commoditization (hypervisor). His Intel experience should be an excellent fit for this task. It must also determine what role it will play in Public and Hybrid clouds, with speculation growing that it may launch public services.
- New Technology – From VMware’s perspective, in order to not only deliver great Enterprise services but also compete with Amazon, Google, Microsoft and Rackspace (or other OpenStack SPs), it needed to be able to control the automation and management of every aspect that impacted a VM. The acquisition of Nicira was the next logical step, as existing networks were never designed for the mobility and dynamic nature of today’s environments.
- Evolving Technology – Considering the possibility that more and more customers would adopt a Hybrid Cloud model (private/public, or multi-public), it’s interesting to see VMware finally acknowledge heterogenous environments with both the DynamicOps and Nicira acquisitions. Support for multiple hypervisors, multiple Cloud providers and various elements of open-source are all potentially in play. It’ll be interesting to see how VMware plans to blend in the different elements.
- New Revenue Sources - CloudFoundry seems to be gaining momentum with the announcements by several Cloud providers to launch services on top of this open-source PaaS platform (Tier 3, Uhuru, etc.), but how will VMware monetize beyond the ESX hypervisor and vSphere tools? vFabric is making progress and has some very interesting functionality, but is the knowledge getting out to the market? Where are the vFabric evangelists like there are for vSphere?
Simon Wardley (@swardley) had some interesting perspectives on VMware in his blog today. As you can see, he highlights many of the questions that people have about VMware’s future strategies, partnerships and technology direction. Some of the insight may seem extreme at first glance, but given Simon’s open-source background and incredible ability to analyze strategic models, I wouldn’t discard anything he says. [Disclosure - I am currently employed by EMC, but none of my comments should be interpreted to have insider knowledge of VMware strategy or plans.]
For me, these are the key areas that I’d like to see greater clarity:
- How does the vCloud Director (vCD) and DynamicOps integration come together? Unlike vCenter, which did an excellent job of opening up the ability for ecosystem integration (network, storage, management, virtual-appliances), vCD has always been very closed. Networking – closed. Security – closed. Storage – limited. Multi-Cloud choice – limited. So will DynamicOps be used to allow hybrid cloud management, or just used to manage 3rd-party hypervisors? Or will it eventually replace vCD?
- VMware has historically always shown technology previews at VMworld, typically 12-18 months out, which signals where they expect to embed functionality that currently resides in 3rd-party HW/SW into VMware platforms. With their new vision of Software Defined Data Center, I’d expect to see this included an expanding list of functionality. It will be very interesting to see how this impacts their partner ecosystem, or if they signal a greater level of participation in open-source efforts. vFabric has had this for a while with SpringSource integration. They inherit some from Nicira (OpenStack Quantum, Open vSwitch) and DynamicOps (XenServer management) but also via work with Puppet Labs + Razor, CloudFoundry, and CloudFoundry + OpenStack.
- While Software Defined Networking (SDN) is a cool concept, networking is very difficult to get right. The early buzz is that Nicira customers are pleased with the technology, but how SDN fits into non-greenfield environments will be the more interesting question. Understanding about SDN is still in the early stages, and many networking-gurus still aren’t sure what the killer use-cases are. VM Mobility is an immediate need in the Enterprise, but will it be a compelling enough use-case for companies to change their networking model vs. existing (and semi-new) overlay models such as VXLAN? How quickly will Enterprise networking teams be able to deal with SDN, or will this be the tipping point for Server teams to take back greater control over the connectivity of applications?
- The vision of Software Defined Data Center is a powerful concept, and one that I believe has many of the pieces in place to succeed with today. It’s the next evolution of intelligent software delivering IT differentiation on consistent hardware. It’ll drive a new set of integration rules for software elements, and it’ll open up new opportunities for hardware to deliver greater speed and capacity to move the massive amounts of data being generated by users and applications.
- VMware has started building their Big Data story with the Cetas acquisition, open-source Project Serengeti and new elements of vFabric / GemFire / SQLFire. Unlike CloudFoundry, which seems to have a clear Cloud vision, it’s not immediately obvious how VMware is trying to shape this segment of the market. They are showing that virtualization can play a role in Big Data (easier setup of environments; self-service consumption), but is there a bigger play they will attempt to capture with a more unified vision? Software Defined Data Center plays a role in infrastructure setup and operations, but how does it play in the Big Data middleware stack that is being created? And how does it tie into CloudFoundry, which also supports MongoDB, MySQL, Redis.
In the past, VMworld was often perceived to be an infrastructure show. Now they have the attention of application developers, DBAs, OpenStack, networking teams, cloud providers and a breadth of partners trying to figure out their role in the new world VMware is trying to build. How many of these questions are answered will determine not only the future of VMware, but have far reaching impact on many segments of the IT ecosystem. And I didn’t even speculate on what might happen if some of the other rumors turn out to be even partially true. It’s a fun time to be involved with aspects of VMware and VMworld.
Microsoft’s Servers and Tools Business (@stbtalent) will hold a Cloud Computing Hiring Event – September 27 & 28th, according to an 8/13/2012 post:
Microsoft’s Server & Tools Business (STB) is a place where brilliant minds can collaborate with other brilliant minds. Where your work environment gives you the flexibility determine your own career path. Microsoft Server & Tool Business is the place where talent and determination win. Where industry leaders lead and legends are born. It’s a place where every day is an opportunity to make the extraordinary happen because Ordinary Doesn’t Work Here. #ODWH
The SQL Azure team will be interviewing for full-time Software Developer Engineer roles on September 27th & 28th. Required skills needed for consideration:
- Completed Degree (Bachelors or Masters or PhD) prior to 2011* in Computer Science or related field *we are unable to consider full-time students*
- 4+ years coding with C, or C++, or C# and systems development experience
- 1+ years’ experience with one or more of the following areas: computer architecture, distributed systems, enterprise server development, query processing, query, cloud computing, operating systems
- Travel assistance is available for those selected for onsite interviews – Microsoft schedules and pays for your trip!
- Apply for event consideration!
Apparently, the Talent Team didn’t receive the SQL Azure -> Windows Azure SQL Database memo.
<Return to section navigation list>
Other Cloud Computing Platforms and Services
Matthew Lodge (@mathewlodge) invited users to Try Your Own vCloud in Minutes on 8/15/2012:
Today, we’re announcing that we’re introducing a new service that allows you to get your own vCloud IaaS service in minutes, called vCloud Service Evaluation. We heard from many customers that they came to vcloud.vmware.com to learn more about vCloud services, but that it wasn’t easy to sign up with a credit card, kick the tires, and learn by doing. vCloud Service Evaluation will provide a quick, easy and low-cost way for you to learn about the advantages of a vCloud through hands-on testing and experimentation.
You can sign up for the beta here: www.vmware.com/go/vcloudbeta. We’ll be sending out invites to those who sign up the week of August 27th, and those of you who are going to VMworld in San Francisco can see and try the service at the cloud services pod within the VMware booth.
You’ll need a credit card to use the service. It makes the service self-funding, and we can keep things simple, avoiding complex “service quotas” and other artificial restrictions – and also offer Windows VMs. We learned that customers have widely differing requirements for tests and proofs of concept. So, instead of annoying restrictions, you pay a small amount for what you use – a 1Gb Linux VM with one vCPU is $0.04/hour – and you are free to run the VMs you need until you are done. Once you have entered your card details, you’ll get your credentials within 15 minutes. If we need to verify anything, you’ll get a call.
To keep costs down, we commissioned a VMware vCloud service provider to build and operate the service on our behalf. We’re giving you a vanilla example of how a vCloud Powered service – delivered by a VMware vCloud service provider – would work. It’s worth pointing out that vCloud service providers offer significantly more in terms of cloud functionality. vCloud Service Evaluation has all the basics like a catalog of useful VM templates, virtual networking, persistent storage, external IP addresses, firewalls, load balancers, the vCloud API etc., but you’ll get a lot more in a production vCloud service.
To find that production vCloud service, head to vcloud.vmware.com: the gateway to the world’s largest network of certified compatible public cloud services, including more than 145 vClouds in 28 countries.
To get you started quickly, vCloud Service Evaluation offers a variety of pre-built content templates (at no charge) including WordPress, Joomla!, Sugar CRM, LAMP stack, Windows Server and a mix of web and application stacks and OSes. You can also Bring Your Own VM (BYOVM). That’s right, you can BYOVM and put it into your own private catalog for deployment. You can do that either by uploading it directly into vCloud Director, or you can run the vCloud Connector VMs into your account (they’re in the public catalog) and use that to transfer your VMs from vSphere or any other vCloud.
Here’s what the main console looks like:
The service evaluation also allows you to run the VMware vCloud Director® interface.
We also learned that while we had some great information on vmware.com, but that it was hard to find stuff relevant to vCloud – and it wasn’t clear where to ask questions. So we put all the “how to” guides in one place, added some new ones, and also provided a Community site (message boards) where you can ask questions and get answers from experts at VMware and our partners.
Finally, email, chat and telephone support is available Monday through Friday for billing enquiries and to report any technical problems. “How do I…?” questions are best asked (and answered) on the Communities site.
We hope you find vCloud Service Evaluation a simple, low-cost way to learn about VMware vCloud, and look forward to getting your feedback on the service.
Chris Talbot reported Red Hat Unveils OpenStack Distribution in an 8/14/2012 post to the TalkinCloud blog:
Another open source vendor has tossed its hat into the OpenStack ring. A red hat, to be more precise. Now in preview release, Red Hat‘s (NYSE: RHT) own OpenStack distribution based on the open source OpenStack framework for building and management public, private and hybrid IaaS clouds.
The news that Red Hat was planning on launching its own OpenStack distribution broke back in April when a GigaOm report let the news slip. Red Hat joining the OpenStack community seems like a case of “if you can’t beat ‘em, join ‘em.” It’s no secret that the vendor was facing increasing competition from the OpenStack community, and now it really is official the company that helped build the Linux empire back in the good ol’ days has seen the OpenStack light.
Of course, Red Hat has had its share of contributions to the OpenStack community and was recognized as the third overall contributor to OpenStack at the OpenStack Summit 2012 in April.
The company is currently looking for feedback from its early customers and expects to launch a fully supported OpenStack release in 2013. That gives Red Hat at least several months to gather customer feedback, test and tweak its distribution.
“Our current productization efforts are focused around hardening an integrated solution of Red Hat Enterprise Linux and OpenStack to deliver an enterprise-ready solution that enables enterprises worldwide to realize infrastructure clouds,” said Brian Stevens, CTO and vice president of worldwide engineering at Red Hat, in a prepared statement.
It should be interesting to see what Red Hat has up its sleeve and how it plans to leverage the rest of its hybrid cloud portfolio of products, which include Red Hat Enterprise Linux, Red Hat Enterprise Virtualization, Red Hat CloudForms, Red Hat Storage and Red Hat OpenShift PaaS.
As partners and customers kick the tires of Red Hat’s OpenStack distribution, they should keep in mind it’s in its early stages and is currently unsupported by the vendor. Based on Red Hat Enterprise Linux 6, the preview version of the OpenStack distribution is available for download now.
Read More About This Topic
Jeff Barr (@jeffbarr) reported AWS Direct Connect – New Locations and Console Support in an 8/13/2012 post:
Did you know that you can use AWS Direct Connect to set up a dedicated 1 Gbps or 10 Gbps network connect from your existing data center or corporate office to AWS?
New Locations
Today we are adding two additional Direct Connect locations so that you have even more ways to reduce your network costs and increase network bandwidth throughput. You also have the potential for a more consistent experience. Here is the complete list of locations:
- CoreSite 32 Avenue of the Americas, New York – Connect to US East (Northern Virginia). New.
- Terremark NAP do Brasil - Connect to South America (Sao Paulo). New.
- CoreSite One Wilshire, Los Angeles, CA – Connect to US West (Northern California).
- Equinix DC1 – DC6 & DC10, Ashburn, VA – Connect to US East (Northern Virginia).
- Equinix SV1 & SV5, San Jose, CA – Connect to US West (Northern California).
- Equinix SG2, Singapore – Connect to Asia Pacific (Singapore).
- Equinix TY2, Tokyo – Connect to Asia Pacific (Tokyo).
- TelecityGroup Docklands, London – Connect to EU West (Ireland).
If you have your own equipment running at one of the locations listed above, you can use Direct Connect to optimize the connection to AWS. If your equipment is located somewhere else, you can work with one of our APN Partners supporting Direct Connect to establish a connection from your location to a Direct Connection Location, and from there on to AWS.
Console Support
Up until now, you needed to fill in a web form to initiate the process of setting up a connection. In order to make the process simpler and smoother, you can now start the ordering process and manage your Connections through the AWS Management Console.
Here’s a tour. You can establish a new connection by selecting the Direct Connect tab in the console:
After you confirm your choices you can place your order with one final click:
You can see all of your connections in a single (global) list:
You can inspect the details of each connection:
You can then create a Virtual Interface to your connection. The interface can connected to one of your Virtual Private Clouds or it can connect to the full set of AWS services:
You can even download a router configuration file tailored to the brand, model, and version of your router:
Get Connected
And there you have it! Learn more about AWS Direct Connect and get started today.
<Return to section navigation list>
http://oakleafblog.blogspot.com/2012/08/windows-azure-and-cloud-computing-posts_15.html
