Windows Azure and Cloud Computing Posts for 4/14/2012+

A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

• Windows Azure Blob, Drive, Table, Queue and Hadoop Services
• SQL Azure Database, Federations and Reporting
• Marketplace DataMarket, Social Analytics, Big Data and OData
• Windows Azure Access Control, Identity and Workflow
• Windows Azure VM Role, Virtual Network, Connect, RDP and CDN
• Live Windows Azure Apps, APIs, Tools and Test Harnesses
• Visual Studio LightSwitch and Entity Framework v4+
• Windows Azure Infrastructure and DevOps
• Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds
• Cloud Security and Governance
• Cloud Computing Events
• Other Cloud Computing Platforms and Services

---

Azure Blob, Drive, Table, Queue and Hadoop Services

Wade Wegner (@WadeWegner) discussed a Return Empty Set Instead of ResourceNotFound from Table Storage issue in a 4/14/2012 post:

This past week I’ve been working on a little project – amazing how less email equates to more time for other endeavors – and I was surprised when I received a DataServiceQueryException when querying table storage in the local storage emulator. I was querying based on partition and row keys and, if no data matched the statement, I received an HTTP 404: Resource Not Found exception.

I was initially puzzled. Shouldn’t I receive an empty set or null instead?

Of course, I had forgotten that this is by design. The DataServiceContext will throw a DataServiceQueryException if there’s no data to return. To receive an empty set it’s necessary to set the IgnoreResourceNotFoundException property to true.

Here’s a simplified version of the code:

    string connectionString = "UseDevelopmentStorage=true";

    var context = CloudStorageAccount.Parse(connectionString)
        .CreateCloudTableClient().GetDataServiceContext();

    context.IgnoreResourceNotFoundException = true;

    var results = context.CreateQuery<TableEntity>("tableName")
        .Where(e => e.PartitionKey == partitionKey && e.RowKey == rowKey).AsTableServiceQuery();

    var key = results.FirstOrDefault();

Problem solved. No DataServiceQueryException!

Something to keep in mind when working with the Windows Azure table storage service. I almost didn’t blog about but decided that it was worth a few minutes effort. Probably something to add to your Windows Azure development checklist (you have one, right?).

---

M Sheik Uduman Ali (@udooz) analyzed Synchronous, Async and Parallel Programming Performance in Windows Azure in a 4/5/2012 post (missed when published):

This post discusses the performance benefits of effectively using .NET TPL when doing I/O bound operations.

Intent

When there is a need for non-synchronous programming pattern (asynchronous and/or parallel) in Azure applications, the pattern of choice must be based on the target VM size we have chosen for that app and the type of operation particular part does.

Detail

.NET provides TPL (Task Parallel Library) to write non-synchronous programming much easier way. The asynchronous API enables to perform I/O bound and compute-bound asynchronous operations which lets the main thread to do the remaining operations without waiting for the asynchronous operations to complete. Refer http://snip.udooz.net/Hbmib2 for details. The parallel API enables to effectively utilizes the multicore processors on your machine to perform data intensive or task intensive operations. Refer http://snip.udooz.net/HTLrVv for details.

When writing azure applications, we may need to interact with many external resources like blob, queues, tables, etc. So, it is very obvious to think asynchronous or parallel programming patterns when the amount of I/O operations are higher. In these cases, we should be more cautious on selecting asynchronous and parallel. The extra-small instance provides shared CPU power, the small instance provides single core and medium or above provide multicore. Hence, asynchronous pattern would be the better option for extra-small and small instances. For problem those are highly parallel in nature, then the application should be placed on Medium or above instance with parallel pattern.

To confirm the above statement, I did a small proof of concept which has high I/O operation. The program interacts with Azure blob to get large number of blobs to get data to solve a problem. I’ve taken a small amount of Enron Email dataset from http://www.cs.cmu.edu/~enron/ which contains email messages for various Enron users on their respective Inbox folder as shown in figure 1 and figure 2.

The above figure shows the “inbox” for the user “benson-r”. Every user has approximately more than 200 email messages. A message contains the following content:

Message-ID: <21651803.1075842014433.JavaMail.evans@thyme>
Date: Tue, 5 Feb 2002 11:06:50 -0800 (PST)
From: robert.stalford@enron.com
To: jay.webb@enron.com
Subject: online power option change request
Cc: andy.zipper@enron.com
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
======= OTHER HEADERS=======
Jay,
It was ..... ====== remaining message body ======

The program going to solve how many times particular user written email to this user. The email messages reside in a blob container with appropriate blob directory. Hence, the pseudo code is some thing like:

for every user
get the blob sub-directory for the user from the blob container
create new dictionary // key - sender email ID, value - count
for every blob in the sub-directory
get blob content
parse the “From” value from the message
if the “From” value already exists on dictionary
increment the value by 1
else
add From field value as key and value as 1 into the dictionary
write the result

I apply “sync, async and parallel” along with normal Task.StartNew and Task.StartNew + ContinueWith programming patterns on “fetching and parsing email messages” logic (more chatty I/O).

The Code

The normal procedural flow is shown below:

// rootContainer is CloudBlobDirectory represents "maildir" container
var mailerInbox = rootContainer.GetSubdirectory(mailerFolder + "/inbox");
foreach (var blob in mailerInbox.ListBlobs())
{
//don't see the subfolders if any
if (blob is CloudBlobDirectory) continue;
var email = mailerInbox.GetBlobReference(blob.Uri.ToString()).DownloadText();
//parsing From field
var match = Regex.Match(email, @"From\W*(\w[-.\w]*@[-a-z0-9]+(\.[-a-z0-9]+)*)");
if (match.Groups.Count > 0)
{
var key = match.Groups[1].Value;
//estimate is a Dictionary contains From email id and the count
if (estimate.ContainsKey(key))
estimate[key] = estimate[key]++;
else
estimate.Add(key, 1);
}
}
var sb = new StringBuilder();
foreach (var kv in estimate)
{
sb.AppendFormat("{0}: {1}\n", kv.Key, kv.Value);
}
//writing the result to a blob
var result = mailerInbox.GetBlobReference("result_normal_" + attempt);
result.UploadText(sb.ToString());

The parallel version is shown below:

var mailerInbox = rootContainer.GetSubdirectory(mailerFolder + "/inbox");
Parallel.ForEach(mailerInbox.ListBlobs(), blob =>
{
if (!(blob is CloudBlobDirectory))
{
var email = mailerInbox.GetBlobReference(blob.Uri.ToString()).DownloadText();
var match = Regex.Match(email, @"From\W*(\w[-.\w]*@[-a-z0-9]+(\.[-a-z0-9]+)*)");
if (match.Groups.Count > 0)
{
var key = match.Groups[1].Value;
// used ConcurrentDictionary
cestimate.AddOrUpdate(key, 1, (k,v) => v++);
}
}
});
//the result writing part is here, similar to normal version

The asynchronous version is:

var mailerInbox = rootContainer.GetSubdirectory(mailerFolder + "/inbox");
var tasks = new Queue();
foreach (var blob in mailerInbox.ListBlobs())
{
if (blob is CloudBlobDirectory) continue;
// blobStorage is a wrapper for Azure Blob storage REST API
var webRequest = blobStorage.GetWebRequest(blob.Uri.ToString());
tasks.Enqueue(Task.Factory.FromAsync(webRequest.BeginGetResponse,
webRequest.EndGetResponse, TaskCreationOptions.None)
.ContinueWith(t =>
{
var response = t.Result;
var stream = new StreamReader(response.GetResponseStream());
var emailMsg = stream.ReadToEnd();
stream.Close();
response.Close();
var match = regex.Match(emailMsg);
if (match.Groups.Count > 0)
{
var key = match.Groups[1].Value;
cestimate.AddOrUpdate(key, 1, (k, v) => v++);
}
}));
}
Task.WaitAll(tasks.ToArray());

The major difference in the “fetching and parsing” part is, instead of managed API, I have used REST API with a wrapper so that I can access the Blob asynchronously. In addition the above, I have used normal TPL tasks in two different way. In the first way, I just processed “fetching and parsing” stuff as shown below:

foreach (var blob in mailerInbox.ListBlobs())
{
if (blob is CloudBlobDirectory) continue;
string blobUri = blob.Uri.ToString();
tasks.Enqueue(Task.Factory.StartNew(() =>
{
var email = mailerInbox.GetBlobReference(blobUri).DownloadText();
var match = Regex.Match(email, @"From\W*(\w[-.\w]*@[-a-z0-9]+(\.[-a-z0-9]+)*)");
if (match.Groups.Count > 0)
{
var key = match.Groups[1].Value;
cestimate.AddOrUpdate(key, 1, (k, v) => v++);
}
}));
}

Task.WaitAll(tasks.ToArray());

Another one way, I have used ContinueWith option with the Task as shown below:

foreach (var blob in mailerInbox.ListBlobs())
{
if (blob is CloudBlobDirectory) continue;
string blobUri = blob.Uri.ToString();
tasks.Enqueue(Task.Factory.StartNew(() =>
{
return mailerInbox.GetBlobReference(blobUri).DownloadText();
}).ContinueWith(t =>
{
var match = regex.Match(t.Result);
if (match.Groups.Count > 0)
{
var key = match.Groups[1].Value;
cestimate.AddOrUpdate(key, 1, (k, v) => v++);
}
}, TaskContinuationOptions.OnlyOnRanToCompletion));
}
Task.WaitAll(tasks.ToArray());

Results

I’ve hosted the work role and storage account at “Southeast Asia”. On every VM size, I’ve made 6 runs and removed the first time result. I have given 12 concurrent connection in the ServicePointManager for all the testing. I did not change this value in medium and large instances. All the results are in millisecond.

Extra Small

…

Small

…

Medium

…

Large

…

Surprisingly, irrespective of the VM size, when an operation is I/O bound, asynchronous pattern outshines all the other approaches followed by Parallel.

Final Words

Hence, the “asynchronous” approach won the I/O bound operation (shown as a diagram also here).

Let me come up with one more test which covers on which area Parallel approach will shine. In addition to these, when you have lesser I/O and want smooth multithreading, Task and Task + ContinueWith may help you.

What do you think? Share your thoughts!

I highly thank Steve Marx and Nuno for validating my approach and the results which are actually improved my overall testing strategy.

The source code is available at http://udooz.net/file-drive/doc_download/23-mailanalyzerasyncpoc.html

No significant articles today.

---

<Return to section navigation list>

SQL Azure Database, Federations and Reporting

No significant articles today.

---

<Return to section navigation list>

MarketPlace DataMarket, Social Analytics, Big Data and OData

Glenn Gailey (@ggailey777) explained Upgrading WCF Data Services Projects to WCF Data Services 5.0 in a 4/16/2012 post:

Now that WCF Data Services 5.0 has been shipped with support for OData v3, it’s time to start updating existing applications to take advantage of some of the new features of OData v3. For a complete list of new OData v3 features that are supported in WCF Data Services 5.0, see the post WCF Data Services 5.0 RTM Release.

Note: If you do not need any of the new OData v3 functionalities, you don’t really need to upgrade to WCF Data Services 5.0. If you do upgrade, you can still take advantage of the OData v2 behaviors in WCF Data Services 5.0. For more information, see the section OData Protocol Versions in Data Service Versioning (WCF Data Services).

Side-by-Side Installation

This release installs side-by-side with the previous versions of WCF Data Services that are in the .NET Framework. This is achieved by renaming the WCF Data Services assemblies from System.Data.Services.*.dll to Microsoft.Data.Services.*.dll, which makes it easier to target the version of WCF Data Services without having to use the multi-targeting in Visual Studio (trust me this is a good thing). This new out-of-band version of WCF Data Services is installed in the program files directory rather than with the reference assemblies. This means that you can find the new assemblies in the following installation path on an x64 computer:

%programfiles(x86)%\Microsoft WCF Data Services\5.0\bin

As you would expect, the WCF Data Services client and server libraries are in the .NETFramework subdirectory and the Silverlight client assemblies are in the SL directory (plus, the Silverlight assemblies have SL appended to the file name).

Common Upgrade Tasks

There are two things that you need to do before you can upgrade a Visual Studio project to use WCF Data Services 5.0 libraries.

1. Install the new 5.0 release, which you can get from the Download Center page WCF Data Services 5.0 for OData v3.
2. Remove any existing references to existing assemblies the name of which start with System.Data.Services.

At this point, you can upgrade either data service or client projects.

Upgrading a Data Service Project

Use the following procedure to upgrade an existing WCF Data Services instance to 5.0 and use OData v3 support:

1. Remove references to System.Data.Services.dll and System.Data.Services.Client.dll.
2. Add a new reference to Microsoft.Data.Services.dll and Microsoft.Data.Services.Client.dll assemblies (found in the installation location described above).
3. Change the Factory attribute in the .svc file markup to:

   Factory="System.Data.Services.DataServiceHostFactory, 
   Microsoft.Data.Services, Version=5.0.0.0, 
   Culture=neutral, PublicKeyToken=31bf3856ad364e35"

4. Change the value of DataServiceBehavior.MaxProtocolVersion to DataServiceProtocolVersion.V3.

Upgrading an OData Client Project

Really, the only way that I have found to be able to correctly upgrade an existing client application to use the WCF Data Services 5.0 version of the client libraries is as follows:

1. Remove the existing references to System.Data.Services.Client.dll.
2. Delete the existing service references to OData services.
3. Re-add the service reference using the Add Service Reference dialog in Visual Studio.

Heads-up for JSON-based Applications

OData v3 introduces a change in the Accept header values used to request a JSON response from the data service. This change doesn’t technically break OData v2 clients because it requires that the data service use the OData v3. However, you may start to get HTTP 415 responses after you upgrade the client or service to OData v3 and request that the data service use the OData v3 protocol with the old application/json value. For a good, detailed explanation of the nuances of this, see the post What happened to application/json in WCF DS 5.0?

As you may recall from my earlier post Getting JSON Out of WCF Data Services, you have to do some extra work enable the data service to handle the $format=json query option. Remember that when upgrading such a data service to support OData v3, you must also change the JSON Accept header value inserted by your code to application/json;odata=verbose.

---

Tony Baer (@TonyBaer) stood in on 4/16/2012 for Andrew Brust (@andrewbrust) as the author of Fast Data hits the Big Data fast lane for ZDNet’s Big Data blog:

Of the 3 “V’s” of Big Data – volume, variety, velocity (we’d add “Value” as the 4th V) – velocity has been the unsung ‘V.’ With the spotlight on Hadoop, the popular image of Big Data is large petabyte data stores of unstructured data (which are the first two V’s). While Big Data has been thought of as large stores of data at rest, it can also be about data in motion.

“Fast Data” refers to processes that require lower latencies than would otherwise be possible with optimized disk-based storage. Fast Data is not a single technology, but a spectrum of approaches that process data that might or might not be stored. It could encompass event processing, in-memory databases, or hybrid data stores that optimize cache with disk.

Fast Data is nothing new, but because of the cost of memory, was traditionally restricted to a handful of extremely high-value use cases. For instance:

• Wall Street firms routinely analyze live market feeds, and in many cases, run sophisticated complex event processing (CEP) programs on event streams (often in real time) to make operational decisions.
• Telcos have handled such data in optimizing network operations while leading logistics firms have used CEP to optimize their transport networks.
• In-memory databases, used as a faster alternative to disk, have similarly been around for well over a decade, having been employed for program stock trading, telecommunications equipment, airline schedulers, and large destination online retail (e.g., Amazon).

Hybrid in-memory and disk have also become commonplace, especially amongst data warehousing systems (e.g., Teradata, Kognitio), and more recently among the emergent class of advanced SQL analytic platforms (e.g., Greenplum, Teradata Aster, IBM Netezza, HP Vertica, ParAccel) that employ smart caching in conjunction with a number of other bells and whistles to juice SQL performance and scaling (e.g., flatter indexes, extensive use of various data compression schemes, columnar table structures, etc.). Many of these systems are in turn packaged as appliances that come with specially tuned, high-performance backplanes and direct attached disk.

Finally, caching is hardly unknown to the database world. Hot spots of data that are frequently accessed are often placed in cache, as are snapshots of database configurations that are often stored to support restore processes, and so on.

So what’s changed?

The usual factors: the same data explosion that created the urgency for Big Data is also generating demand for making the data instantly actionable. Bandwidth, commodity hardware and, of course, declining memory prices, are further forcing the issue: Fast Data is no longer limited to specialized, premium use cases for enterprises with infinite budgets.

Not surprisingly, pure in-memory databases are now going mainstream: Oracle and SAP are choosing in-memory as one of the next places where they are establishing competitive stakes: SAP HANA vs. Oracle Exalytics. Both Oracle and SAP for now are targeting analytic processing, including OLAP (by raising the size limits on OLAP cubes) and more complex, multi-stage analytic problems that traditionally would have required batch runs (such as multivariate pricing) or would not have been run at all (too complex, too much delay). More to the point, SAP is counting on HANA as a major pillar of its stretch goal to become the #2 database player by 2015, which means expanding HANA’s target to include next generation enterprise transactional applications with embedded analytics.

Potential use cases for Fast Data could encompass:

• A homeland security agency monitoring the borders requiring the ability to parse, decipher, and act on complex occurrences in real time to prevent suspicious people from entering the country
• Capital markets trading firms requiring real-time analytics and sophisticated event processing to conduct algorithmic or high-frequency trades
• Entities managing smart infrastructure which must digest torrents of sensory data to make real-time decisions that optimize use of transportation or public utility infrastructure
• B2B consumer products firms monitoring social networks may require real-time response to understand sudden swings in customer sentiment

For such organizations, Fast Data is no longer a luxury, but a necessity.

More specialized use cases are similarly emerging now that the core in-memory technology is becoming more affordable. YarcData, a startup from venerable HPC player Cray Computer, is targeting graph data, which represents data with many-to-many relationships. Graph computing is extremely process-intensive, and as such, has traditionally been run in batch when involving Internet-size sets of data. YarcData adopts a classic hybrid approach that pipelines computations in memory, but persisting data to disk. YarcData is the tip of the iceberg – we expect to see more specialized applications that utilize hybrid caching that combine speed with scale.

But don’t forget, memory’s not the new disk

The movement – or tiering – of data to faster or slower media is also nothing new. What is new is that data in memory may no longer be such a transient thing, and if memory is relied upon for in situ processing of data in motion or rapid processing of data at rest, memory cannot simply be treated as the new disk. Excluding specialized forms of memory such as ROM, by nature anything that’s solid state is volatile: there goes your power… and there goes your data. Not surprisingly, in-memory systems such as HANA still replicate to disk to reduce volatility. For conventional disk data stores that increasingly leverage memory, Storage Switzerland’s George Crump makes the case that caching practices must become smarter to avoid misses (where data gets mistakenly swapped out). There are also balance of system considerations: memory may be fast, but is its processing speed well matched with processor? Maybe solid state overcomes I/O issues associated with disk, but may still be vulnerable to coupling issues if processors get bottlenecked or MapReduce jobs are not optimized.

Declining memory process are putting Fast Data on the fast lane to mainstream. But as the technology is now becoming affordable, we’re still early in the learning curve for how to design for it.

---

Clemens Vasters (@clemensv) asked “REST API” or “HTTP API”? in a 4/13/2012 post:

I just wrote this email on a private mailing list and thought it may make sense to share it. The context of the discussion was overuse of the term “REST” in a document discussing an HTTP API:

REST is a set of architectural principles. REST describes how state flows and describes the shape of relationships between the parties in a distributed system. HTTP is a protocol with a variety of stacks supporting it, and the REST principles were born out of developing HTTP. There could, in theory, a broad variety of protocols that also embody REST architecture, but there are, in fact, very few (if any) that aren’t just variations of HTTP.

“The client sends …”, “The server receives …”, “The server provides an interface for …” are all statements about implementation and, thus, HTTP. It commonly starts making talking about REST specifically when debating whether a system is actually following the principles according to the 5.3.3 “Data View” section in [1], since everything up to that point in Fielding’s dissertation you get generally for free with HTTP.

[1] http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

Bottom line: HTTP APIs are HTTP APIs. REST is about how things hang together. The terms aren’t interchangeable. In most technical discussions about interfaces or methods or URIs and most other implementation details, HTTP API is the right term.

No significant articles today.

---

<Return to section navigation list>

Windows Azure Service Bus, Access Control, Identity and Workflow

Jim O’Neil (@jimoneil) announced on 4/16/2012 that he will appear on Mike Benkovich’s Windows Azure Soup to Nuts: Service Bus (April 17) Webcast":

My colleague, Mike Benkovich, has been running a weekly webcast called “Cloud Computing: Soup to Nuts”, covering various parts of the Windows Azure platform for the developer audience. It’s a great way to get your ‘head in the clouds’ and make use of the 90-day Windows Azure Free Trial or your MSDN benefits (you did know you get free time in the cloud with MSDN, right?).

Tomorrow, Tuesday April 17th, I’ll be the guest presenter for the ninth segment of his series: “Getting Started with the Service Bus” from 2 – 3 p.m. ET.

No man is an island, and no cloud application stands alone! Now that you’ve conquered the core services of web roles, worker roles, storage, and Microsoft SQL Azure, it’s time to learn how to bridge applications within the cloud and between the cloud and on premises. This is where the Service Bus comes in—providing connectivity for Windows Communication Foundation and other endpoints even behind firewalls. With both relay and brokered messaging capabilities, you can provide application-to-application communication as well as durable, asynchronous publication/subscription semantics. Come to this webcast ready to participate from your own computer to see how this technology all comes together in real time.

Did you catch that last part of the abstract ? I’m going to need some help with the demos, and that’s where you come in. Intrigued? Register for the webcast!

To catch up on the rest of Mike’s series, check out his Soup to Nuts page on BenkoTips!

No significant articles today.

---

<Return to section navigation list>

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

No significant articles today.

---

<Return to section navigation list>

Live Windows Azure Apps, APIs, Tools and Test Harnesses

The Windows Azure Team (@WindowsAzure) announced the availability of the Windows Media Services Preview on 4/16/2012:

Build workflows for the creation, management, and distribution of media with Windows Azure Media Services. Media Services offer the flexibility, scalability, and reliability of a cloud platform to handle high quality media experiences for a global audience. Some common uses of Media Services include:

End to End Workflows

Build comprehensive media workflows entirely in the cloud. From uploading media to distributing content, Media Services provide a range of pre-built, ready-to-use, first- and third-party services that can be combined to meet your specific needs. Capabilities include upload, storage, encoding, format conversion, content protection, and delivery.

Hybrid Workflows

Easily integrate Media Services with tools and processes you already use. For example, encode content on-site then upload to Media Services for transcoding into multiple formats and deliver through a third-party CDN. Media Services can be called individually via standard REST API’s for easy integration with external applications and services.

Cloud Support for Windows, Xbox, iOS, and Android

Creating, managing and delivering media across multiple devices has never been easier. Media Services provide everything you need to deliver content to a variety of devices, from Xbox and Windows PCs, to MacOS, iOS and Android.

Capabilities and Benefits of Media Services

Media Services includes cloud-based versions of many existing technologies from the Microsoft Media Platform and our media partners. Whether enhancing existing solutions or creating new workflows, you can easily combine and manage Media Services to create custom workflows that fit every need:

Ingest: Upload your media assets to Media Services storage using standard HTTP transfers or built-in third-party agents for faster UDP transfers with added security.

Encoding: Work with a range of standard codecs and formats, including industry-leading Smooth Streaming, HTTP Live Streaming, MPEG-DASH and Flash. You can choose the Windows Azure Media Encoder or a built-in third-party encoder.

Format Conversion: Convert entire libraries or individual streams with total control over input and output.

Content Protection: Easily encrypt live or on-demand video and audio with standard MPEG Common Encryption and Microsoft PlayReady, the industry’s most accepted DRM for premium content. Add watermarking to your media for an extra layer of protection.

On-Demand Streaming: Seamlessly deliver content via Windows Azure CDN or a third-party delivery network. Automatically scale to deliver high quality video experiences around the globe.

Live Streaming: Easily create and publish live streaming channels, encoding with Media Services or pushing from an external feed. Take advantage of built-in features like server-side DVR and instant replay.

For more information on these services and how to build solutions around them, visit the Windows Azure Developer Center.

Pricing and Metering for Media Services

The upcoming preview of Windows Azure Media Services will be available at no cost (charges for associated Windows Azure features like Storage, Egress, and CDN may apply). To sign up for the preview, click here.

The announcement coincided with the second day of the National Association of Broadcasters (NAB) show in Las Vegas, NV. You can read more technical details about the Windows Azure Media Services Preview for developers here.

---

Mary Jo Foley (@maryjofoley) also reported in her Microsoft delivers preview of its Windows Azure-hosted media services platform post of 4/16/2012 to ZDNet’s All About Microsoft blog:

… Microsoft officials also announced at NAB that the company will be working with Akamai and Deltatrae to deliver high definition streaming video of the London 2012 Olympic Games this summer across multiple countries.

---

Scott Guthrie (@scottgu) rang in with Announcing Windows Azure Media Services on 4/16/2012:

I’m excited to share news about a great new cloud capability we are announcing today – Windows Azure Media Services.

Windows Azure Media Services

Windows Azure Media Services is a cloud-based PaaS solution that enables you to efficiently build and deliver media solutions to customers. It offers a bunch of ready-to-use services that enable the fast ingestion, encoding, format-conversion, storage, content protection, and streaming (both live and on-demand) of video. It also integrates and exposes services provided by industry leading partners – enabling an incredibly deep media stack of functionality that you can leverage.

You can use Windows Azure Media Services to deliver solutions to any device or client – including HTML5, Silverlight, Flash, Windows 8, iPads, iPhones, Android, Xbox, and Windows Phone devices. Windows Azure Media Services supports a wide variety of streaming formats – including Smooth Streaming, HTTP Live Streaming (HLS), and Flash Media Streaming.

One of the unique aspects of Windows Azure Media Services is that all of its features are exposed using a consistent HTTP REST API. This is true both for the media services we’ve built, as well as the partner delivered media services that are enabled through it. This makes it incredibly easy to automate media workflows and integrate the combined set of services within your applications and media solutions. Like the rest of Windows Azure, you only pay for what you use with Windows Azure Media Services – making it a very cost effective way to deliver great solutions.

…

Windows Azure Media Services uses the same award-winning media backend that has been used to power some of the largest live sporting events ever broadcast on the web – including the 2010 Winter Olympics, 2010 FIFA World Cup, 2011 Wimbledon Championships, and 2012 NFL SuperBowl. Using Windows Azure Media Services you’ll now be able to quickly standup and automate media cloud solutions of your own that are capable of delivering amazing solutions to an equal sized audience.

Learn More

We are introducing Windows Azure Media Services at the 2012 National Association of Broadcasters (NAB) Show this week, and attendees can stop by the Microsoft booth there to meet the team and see live demonstrations of it in action.

You can also visit windowsazure.com/media to learn more about the specific features it supports, and visit the windowsazure.com media dev center to learn more about how to develop against it. You can sign-up to try out the preview of Windows Azure Media Services by sending email to mediaservices@microsoft.com (along with details of the scenario you’d like to use it for).

We are really excited about the capabilities Windows Azure Media Services provides, and are looking forward to watching the solutions that will soon be built on it.

---

Benjamin Guinebertière (@benjguin) described How to request/buy a certificate and use it in Windows Azure in a 4/14/2012 bilingual post. From the English version:

Some domain registrars may let you request an SSL certificate for your domain. It is also possible to buy a certificate from a certificate authority. This post shows a way to request such a real or production certificate (not a test certificate) and use it in Windows Azure.

In this example I use Gandi registrar. With each domain they offer an SSL certificate; so let’s see how to request it and use it in Windows Azure. The main steps are:

1. create a request from within IIS
2. send the request to Gandi
3. confirm the request in a bunch of e-mail and Web interfaces
4. retrieve the request response and put it into IIS
5. export the certificate from the IIS machine as a .pfx file
6. upload the .pfx file to Windows Azure portal
7. use the certificate in a simple sample Windows Azure App.

Create a request from within IIS

In this sample, the domain I registered with www.gandi.net was “appartement-a-vendre-courbevoie.fr” and we’ll create a certificate for myapp.appartement-a-vendre-courbevoie.fr so that we can expose an ssl application at https://myapp.appartement-a-vendre-courbevoie.fr.

We’ll first create a certificate request from within IIS. IIS is used as a tool that will create an unsigned certificate (with its private key) before sending it (without the private key) to the certificate authority who will sign the certificate.

Start IIS Manager, and go to the server certificate feature

Create a certificate request

In next screen the most important is to have Common name corresponding exactly to the URL that the certificate will be used with.

This generates a certificate request that looks like this

Send the request to Gandi

Before requesting the certificate, Gandi requires you to have an e-mail adress that corresponds to admin@<yourdomain>. Then, you can request the certificate. Here are the steps.

Confirm the request

Here are the steps to confirm the request

Retrieve the request response

Let’s now retrieve the result

Export the certificate

Let’s now export the certificate from the local machine to a .pfx file.

Upload the .pfx file to Windows Azure portal

Let’s send the .pfx file to Windows Azure

Use the certificate in a simple sample Windows Azure App.

NB: In some configurations, I’ve seen the necessity to have the certificate stored at current user’s level, not only at local machine level. Let’s first copy the certificate from the local machine store to the current user store.

Let’s now use the certificate in a Visual Studio 2010 project and deploy it to Windows Azure.

Let’s deploy to a bunch of extra small machines to show that the certificate is deployed automatically by Windows Azure to each instance. Note that SSL channel ends on each VM in the Web farm as I showed in this previous post.

By the way, using 6 extra small machines is the same price as 1 small machine.

(…)

(…)

In order to be able to access the App. from the domain name corresponding to the certificate, a CNAME entry must be added in the DNS; then myapp.appartement-a-vendre-courbevoie.fr matches sslapp.cloudapp.net

Here is the result

---

Richard Conway (@azurecoder) described Automating the generation of service certificates in Windows Azure on 4/13/2012:

I was prompted to write this having seen some of the implementations of the generation of service certificates online. Some poor explanations so I though I’d plug the gap. First let us cover some definitions. We interact with our subscription through a management certificate.

The management certificate needs to be uploaded to the subscription through the portal. This is the only function that we can’t automate. Obvious why, everybody has probably seen the chicken and the egg here already. Anyway, Microsoft have provided a .publishsettings file and Uri which eases the pain of automating this process because the fabric will instamagically update your subscription when you use your live id to login and download a publishsettings file. Y voila you have management access.

A service certificate is something different though. Service certificates are bound to an individual hosted service and don’t entail management of anything. They actually allow you to perform any operation which involves a certificate for that particular hosted service. Under the seams that certificate is being added to the Personal store on each of the role instances within that service.

Service certificates are immensely important for two essential functions: SSL and Remote Desktop.

Management Portal Showing Service Certificates

SSL is intrinsic to the role instance since it is part of IIS which is present on each of the web roles. Remote Desktop requires a plugin but equally uses the service certificate for authentication purposes.

I wanted to highlight one great way of generating service certificates. There are several ways to do this but we’ll focus on a single one although we can use makecert, powershell and Microsoft provide a test app called CertificateGenerator (essentially a COM Callable Wrapper) amongst others. This way uses Bouncy Castle, a great library which is available through nuget. Simply:

Bouncy Castle from Nuget

> Install-Package BouncyCastle

at the Package Manager Console prompt and it is installed.

Let’s start by determining all of our using statements:

using Org.BouncyCastle.Asn1.X509;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Crypto.Generators;
using Org.BouncyCastle.Crypto.Prng;
using Org.BouncyCastle.Math;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.X509;

And then our method signature:

public static X509Certificate2 Create(string name, DateTime start, DateTime end, string userPassword, bool addtoStore = false)

In order to create our certificate as a minimum we need a name, a validity period and as we are protecting a private key we need a private key password (more on this later!). Additionally we may want to add this to a local certificate store which the System.Cryptography assembly allows us to do fairly easily.

We always start any asymmetric cryptographic operation with the a private-public key pair. To generate keys we can use the following:

// generate a key pair using RSA
var generator = new RsaKeyPairGenerator();
// keys have to be a minimum of 2048 bits for Azure
generator.Init(new KeyGenerationParameters(new SecureRandom(new CryptoApiRandomGenerator()), 2048));
var cerKp = generator.GenerateKeyPair();

Two properties that an X509v3 certificate has are a serial number and a subject name (and issuer name). The representation of what this looks like is canonical so we use terms such as “Common Name” (CN) or “Organisational Unit” (OU) to define details about the party the certificate represents and who the authority is that is vouching for them.

To create a subject name we use the X509Name as below and to generate a serial number which is a unique reference to our certificate we generate a large random prime:

// create the CN using the name passed in and create a unique serial number for the cert var certName = new X509Name("CN=" + name); var serialNo = BigInteger.ProbablePrime(120, new Random());

After doing this we can create an X509v3CertificateGenerator object which will encapsulate and create the certificate for us:

// start the generator and set CN/DN and serial number and valid period
var x509Generator = new X509V3CertificateGenerator();
x509Generator.SetSerialNumber(serialNo);
x509Generator.SetSubjectDN(certName);
x509Generator.SetIssuerDN(certName);
x509Generator.SetNotBefore(start);
x509Generator.SetNotAfter(end);

Once we’ve set the basic and essential properties we can focus on what the cert actually does:

// add the server authentication key usage
var keyUsage = new KeyUsage(KeyUsage.KeyEncipherment);
x509Generator.AddExtension(X509Extensions.KeyUsage, false, keyUsage.ToAsn1Object());
var extendedKeyUsage = new ExtendedKeyUsage(new[] {KeyPurposeID.IdKPServerAuth});
x509Generator.AddExtension(X509Extensions.ExtendedKeyUsage, true, extendedKeyUsage.ToAsn1Object());

Two types of property that the certificate has are Key Usage and Extended Key Usage which tell us all about its purpose to life. It’s rasion D’etre (it’s getting that time of night where I think I can actually speak French!)

In this case the certificate we create will need to be able to do two things.

1. Prove to a client that it has authority to verify the server and
2. Encrypt a key during a key exchange process

X509 Certificate with KU/EKU properties

Both of these are common to SSL (TLS).

The rest is fairly straightforward. We can set a signature algorithm. Note the use of Sha1 which by extension is the thumbprint algorithm in our certificate which is an integrity check to prove that the cert hasn’t been tampered with. It’s important to be aware that Azure will only support this thumbprint algorithm.

// algorithm can only be SHA1 ??
x509Generator.SetSignatureAlgorithm("sha1WithRSA");
// Set the key pair
x509Generator.SetPublicKey(cerKp.Public);
Org.BouncyCastle.X509.X509Certificate certificate = x509Generator.Generate(cerKp.Private);

When this is done we will want to do common tasks with this and generally end up with our familiar X509Certificate2 exposed by the System.Cryptography.X509Certificates namespace and used in all common crypto tasks. Well the means to do this are fairly easy and provided by Bouncy Castle.

// export the certificate bytes
byte[] certStream = DotNetUtilities.ToX509Certificate(certificate).Export(X509ContentType.Pkcs12, userPassword);

Also not the use of PKCS#12 (Public Key Cryptographic Standard) which defines the private and uses a form of password-based encryption (PBE) to ensure that only with the password can I access the private key. As we can just use our password and now treat the X509Certificate2 class as a container for our cert with private key.

var cert = new X509Certificate2(certStream, userPassword);

Adding the certificate to the store is fairly easy. You would first start by opening the store you want to engage:

///
<summary> /// Returns the My LocalMachine store
/// </summary>
private static X509Store ReturnStore()
{
  var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
  store.Open(OpenFlags.OpenExistingOnly | OpenFlags.ReadWrite);
  return store;
}

After that all it takes is a bit addition using your X509Certificate2 object and then closing the store to release the handle.

One thing to note is that this certificate is self-signed. This doesn’t have to be the case; I could easily build a PKI here using this simple technique. Of course the code would like slightly differently (maybe we’ll cover this in a follow-up post) as would the issuer name.

I thought I’d write this post to offer readers another way to generate certificates. Six years ago when I was involved as the CTO in a startup that produced epassport software I would get immersed into the underlying details of these standards. Most of the time we would use OpenSSL which is an absolute gem of a library but Bouncy Castle comes a pretty close second in terms of functionality and upkeep. Have a play and enjoy!

The next generation of the Azure Fluent Management library uses the above code in order to automate the setup of SSL for a webrole and remote desktop. There has been a lot of refactoring on this recently to help us streamline deployments and we hope to release this in the coming week.

---

<Return to section navigation list>

Visual Studio LightSwitch and Entity Framework 4.1+

Michael Washington (@ADefWebserver) asserted LightSwitch And OData: Like Putting Out A Fire With A Telephone in a 4/14/2012 post:

Visual Studio LightSwitch is a relatively young technology. Its supporters, have various reasons for supporting it. For some it has allowed them to realize their dream application, for others it has allowed them to finally complete that long suffering project.

For myself, my primary motivation is that it allows me to actually complete projects in a reasonable amount of time (and time is money). In most cases I can complete a project 95%+ faster than if I coded the exact same requirements in ASP.NET Web Forms.

The Silverlight Match

Last year, however, there has been a disquiet among my fellow LightSwitch enthusiasts. Silverlight’s largest detractor was no longer outsiders but Microsoft:

• Our strategy with Silverlight has shifted
• No plug-ins allowed on Windows on ARM

The problem is that LightSwitch, at the time, only created Silverlight applications. Out ‘in the field’ people would commonly ask, “isn’t Silverlight dead?”. How can we promote LightSwitch in our organizations and to our colleagues and fellow developers if ‘Silverlight is dead’ ?

LightSwitch: The House Is On Fire

For many of my fellow LightSwitch supporters the house was on fire and they were wondering what the LightSwitch team was going to do to put it out. Most agreed that LightSwitch needed to output HTML5 pages to remain competitive.

To use the House Is On Fire as a metaphor, the supporter was watching the house burn and wanted a fire hose, fast.

On February 29th 2012 the LightSwitch team gave us… OData and said HTML5 support won’t (yet) be included in LightSwitch.

This was not the fire hose people were looking for. Think about it, your house is burning down and instead of giving you a fire hose, someone gives you… well… a telephone. How are you suppose to put the fire out with a telephone?

I Was Not Scared – Just Annoyed

During the past year I was not scared for the future of LightSwitch, I was annoyed. Many… ok most people did not see what I see. The brilliance of the LightSwitch architecture. LightSwitch does not just create applications, what it mainly does is allow you to define an application, its entities, both local and external, its business rules and objects, and its security. It also allows you to define the UI (user interface). I deliberately use the word define at this point because it is the second part of LightSwitch, the publishing process that actually implements the application and the UI.

When you publish a LightSwitch application, that is when the Silverlight application is created. However, LightSwitch was designed from the ground up to be a two part process. I have always been confidant that the reason it is a two part process is that the LightSwitch team knew years ago, when the product was being designed, that it needed to be designed to publish and output in other technologies.

To return to the House Is On Fire metaphor, I have faith only because I can recognize that the city has built fire hydrants and there is one nearby.

So yeah, I was a bit shocked too when I was staring at a telephone rather than a fire hose!

LightSwitch Team – Playing The Long Game

One thing I have come to accept over the past two years interacting with the LightSwitch team is that they really are smarter than the average person. They think ahead, way ahead. The proof is in the LightSwitch application and its extensibility that can handle any challenge I have thrown at it. The LightSwitchHelpWebsite.com has over 50 sample applications that I have created over the past 2 years, proving over and over again that it can create any professional enterprise application.

So even I had to stop and think, back to the House Is On Fire metaphor, is a telephone what I really need? With a telephone:

• I can call for help
• I can get information from other people
• Perhaps the fire is bigger than I thought?
  • I can call for an airplane water drop

So perhaps the most important thing is not to create HTML5 web pages (the fire hose), but to first improve communication:

• When the HTML5 ‘yet’ comes, wont the LightSwitch team need a solid core built on OData to communicate between the HTML5 pages and LightSwitch?
• By implementing OData now, doesn’t that allow me to create HTML5 pages, JQuery Mobile apps, Native Android apps, Windows Phone 7 apps, now? (and later communicate with technologies that have yet to be created?)

When your house is burning down, maybe what you really do need is a telephone.

Also See:

• Blog – Betting The House On LightSwitch
• Blog – LightSwitch: It’s About The Money… (It’s Always About The Money)
• Blog – Oh, And It’s Also All About Productivity
• Blog – LightSwitch: "There is a commitment to explore creation of apps other than Silverlight…"

OData:

• A Full CRUD LightSwitch JQuery Mobile Application
• A Full CRUD DataJs and KnockoutJs LightSwitch Example Using Only An .Html Page
• Calling LightSwitch 2011 OData Using Server Side Code
• Using The OData Explorer with LightSwitch OData
• Learn How To Make OData Calls In LightSwitch 2011
• Accessing Your Visual Studio 2011 LightSwitch Application Using OData
• More …

---

Return to section navigation list>

Windows Azure Infrastructure and DevOps

Ryan Dunn (@dunnry) explained Getting Diagnostics Data From Windows Azure in a 4/16/2012 post:

Assuming you know what to monitor and you have configured your deployments to start monitoring, now you need to actually get the data and do something with it.

First, let’s briefly recap how the Diagnostics Manager (DM) stores data. Once it has been configured, the DM will start to buffer data to disk locally on the VM using the temporal scratch disk*. It will buffer it using the quota policy found in configuration. By default, this allocates 4GB of local disk space to hold diagnostics data. You can change the quota with a little more work if you need to hold more, but most folks should be served just fine with the default. Data is buffered as FIFO (first in, first out) in order to age out the oldest data first.

Scheduled versus OnDemand

Once the data is buffering locally on the VM, you need to somehow transfer the data from the VM to your cloud storage account. You can do this by either setting a Scheduled or OnDemand transfer. In practice, I tend to recommend always using Scheduled transfers and ignoring the OnDemand option (it ends up being a lot easier).

But, for completeness, here is an example of setting an OnDemand transfer:

void Main()
{
    var account = new CloudStorageAccount(
        new StorageCredentialsAccountAndKey("dunnry", "yourkey"),
        true
        );
        
    var mgr = new DeploymentDiagnosticManager(account, "6468a8b749a54c3...");
    
    foreach (string role in mgr.GetRoleNames())
    {
        var ridm = mgr.GetRoleInstanceDiagnosticManagersForRole(role);
        
        var options = new OnDemandTransferOptions()
        {
            From = DateTime.UtcNow - TimeSpan.FromMinutes(10),
            To = DateTime.UtcNow,
            NotificationQueueName = "pollme"
        };
        
        var qc = account.CreateCloudQueueClient();
        var q = qc.GetQueueReference("pollme");
        q.CreateIfNotExist();
        
        foreach (var i in ridm)
        {
            //cancel all pending transfers
            foreach (var pt in i.GetActiveTransfers())
            {
                i.CancelOnDemandTransfers(pt.Key);
            }
            
            var key = i.BeginOnDemandTransfer(DataBufferName.Logs, options);
            //poll here... why bother...
        }
    }
}

It’s not exactly straightforward, but essentially, you need to specify the time range to transfer and optionally a queue to notify when completed. You must ensure that all outstanding OnDemand transfers are canceled and then you can begin the transfer and ideally you should also cancel the transfer when it is completed. In theory, this gives you some flexibility on what you want transferred.

As with most things in life, there are some gotchas to using this code. Most of the time, folks forget to cancel the transfer after it completes. When that happens, it prevents any updates to the affected data source. This can impact you when you try to set new performance counters and see an error about an OnDemand transfer for instance. As such, you end up writing a lot of code to detect and cancel pending transfers first before doing anything else in the API.

Using Scheduled transfers ends up being easier in the long run because you end up getting the same amount of data, but without having the pain of remembering to cancel pending transfers and all that. Here is similar code (you should adapt for each data source you need to transfer):

void Main()
{
    var account = new CloudStorageAccount(
        new StorageCredentialsAccountAndKey("dunnry", "yourkey"),
        true
        );
        
    var mgr = new DeploymentDiagnosticManager(account, "6468a8b749a54c3...");
    
    foreach (string role in mgr.GetRoleNames())
    {
        var ridm = mgr.GetRoleInstanceDiagnosticManagersForRole(role);
        
        foreach (var idm in ridm)
        {
            var config = idm.GetCurrentConfiguration()

                ?? DiagnosticMonitor.GetDefaultInitialConfiguration();
            config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(5);
            
            //set other scheduled intervals here...

            idm.SetCurrentConfiguration(config);
        }
    }
}

This ends up being the technique we use for AzureOps.com. When you setup your subscription with us, we detect the diagnostics connection string and allow you to change your data source settings. For Performance Counters, we force the transfer to 5 minutes to today (a good compromise) and allow you to choose the interval for other sources (i.e. Traces, Windows Event Logs). When you use a provider like AzureOps, it is usually best to stream the data in in relatively small chunks as opposed to say transferring once an hour. Firstly, we won’t be able to do anything with your data until we see it and you probably want to be notified sooner than 1 time an hour. Secondly, when you set long transfer period times, there is a risk that you exceed the buffer quota and start to lose data that was never transferred. In practice, we have not observed any noticeable overhead by transferring more often. When in doubt, pick 5 mins.

Whew! If you have made it this far, you now have a reasonable set of performance counters and trace information that is both being collected on your VMs in Windows Azure as well as being persisted to your storage account. So, essentially, you need to now figure out what to do with that data. That will be the subject of the next post in this series.

*if you are interested, RDP into an instance and check the resource drive (usually C:) under /Resources/Directory/<roleuniquename>/Monitor to see buffered data.

---

Tony Bailey (@cloudminion) recommended that you Join the Windows Azure partner ecosystem in a 4/16/2012 update to his LinkedIn profile:

I’m wondering why OakLeaf Systems isn’t listed in the Windows Azure Partner List for the U.S. Will update this post when I find out.

---

Joe Panettieri (@joepanettieri) asked Will Open Source Clouds Surround Microsoft Windows Azure? in a 4/15/2012 post to the TalkinCloud blog:

Over the past year, I’ve watched closely as Microsoft’s two flagship cloud platforms — Office 365 and Windows Azure — gained anecdotal momentum. Each week, I hear from more channel partners and ISVs (independent software vendors) that are embracing Office 365 and Windows Azure, respectively.

But here’s the challenge from Microsoft and its channel partners: It seems like the cloud computing world has increasingly gone open source. The examples:

• The OpenStack Foundation launched last week with the backing of Dell, IBM, Hewlett-Packard and other giants that have vowed to fund the open source cloud computing organization. Much of the OpenStack industry will gather this week at a major conference in San Francisco.
• Citrix Systems is working with Apache to open source CloudStack. Plus, CloudStack apparently will be interoperable with Amazon’s cloud APIs.
• Eucalyptus, another open source cloud platform, has also vowed to maintain API compatibility with Amazon Web Services.
• VMware’s Cloud Foundry (platform as a service) also has an open source project to which developers and contribute.

Yes, Microsoft recently created an open source subsidiary. And Windows Azure supports a range of open source development tools, with Apache Hadoop for Azure a work in progress. But I keep wondering if Microsoft can keep pace as numerous IT giants — again, Dell, HP and IBM in particular — continue to line up behind major cloud open source initiatives.

For smaller channel partners, I still see plenty of potential upside in the Windows Azure world. I believe ISVs such as CA Technologies (ARCserve), Symantec (Backup Exec) and others have been launching their apps in the Azure cloud.

Still, I hear 10 times the chatter around open source clouds. I wonder if all that open source discussion is hype — or reality.

Read More About This Topic

• OpenStack Launches Fifth OS Release ‘Essex’
• Sentri Buys Clearway for Microsoft Office 365 Cloud Expertise
• Microsoft Cloud Essentials, Cloud Accelerate Show Growth
• Office 365 & Windows Azure Cloud Pricing: How Low Can It Go?
• Leap Year Bug Causes Massive Microsoft Windows Azure Outage

---

Wely Lau (@wely_live) posted Comparing IAAS and PAAS: A Developer’s Perspective on 4/15/2012:

In my previous article, I discussed the basic concepts behind Cloud Computing including definitions, characteristics, and various service models. In this article I will discuss service models in more detail, and in particular the comparison between IAAS and PAAS from a developer’s standpoint.

I’m using two giant cloud players for illustrative purposes: Amazon Web Service representing IAAS and Windows Azure Platform representing PAAS. Nonetheless, please be informed that the emphasis is on the service models and not the actual cloud players.

Figure 1: IAAS VS PAAS

Infrastructure as a Service (IAAS)

IAAS refers to the cloud service model that provides on-demand infrastructure services to the customer. The infrastructure may refer to rentable resources such as computation power, storage, load-balancer, and etc.

As you can see on the left-hand side of Table 1, the IAAS provider will be responsible for managing physical resources, for example network, servers, and clustered machines. Additionally, they typically will also manage virtualization technology enabling customers to run VMs (virtual machines). When it comes to the Operating System (OS), it is often arguable whether it’s managed by the provider or customer. In most cases, the IAAS provider will be responsible for customer VM Images with a preloaded OS but the customer will need to subsequently manage it. Using AWS as an example, AMI (Amazon Machine Image) offers customers several types of Operating Systems such as Windows Server, Linux SUSE, andLinux Red Hat. Although the OS is preloaded, AWS will not maintain or update it.

Other stacks of software including middleware (such as IIS, Tomcat, Caching Services), runtime (JRE and .NET Framework), and databases (SQL Server, Oracle, MySQL) are normally not provided in the VM Image. That’s because the IAAS provider won’t know and won’t care what customers are going to do with the VM. Customers are responsible for taking care of them. When all of the above mentioned software has been settled, customers will finally deploy the application and data on the VM.

Step-by-step: Setting-up an Application on IAAS Environment

To convey a comprehensive explanation, I am going to illustrate the steps involved when setting up an application in an IAAS environment. For that, I’m borrowing a slide from a presentation by Mark Russinovich, at the BUILD conference. This illustration explains how a typical IAAS provisioning model works.

Figure 2: Setting up an App

Considering a common scenario when you have finished developing a multi-tier application, you as the developer will need to deploy it to the cloud. The application will need to be hosted on a Web Server and an RDBMS database. For IAAS, here are the typical steps:

1. Preparing Database Servers

Select the VM Images from the VM Images library. The VM Image will then get provisioned and launched. If DBMS software is not provided, you will need to install DBMS on your own.

2. Preparing Web / Application Servers

Select VM Images from the library to get provisioned and launched. If the web/app server/runtime aren’t installed, you’ll need to install them by yourself.

3. Provisioning a Database and Its Objects

The next step is about provisioning the database, including configuring the data files, log files, security, etc. Then you create the tables and add data to it.

4. Deploying Your Application

Next you take the application that you’ve developed and deploy it to the Web Server.

5. Configuring load-balancer

When you need to host your application on multiple instances, you may also need to configure things such as the IP Address for each instance and load balancer.

6. Managing Your VMs and DMBS

The final step is about managing the VMs. For example, when there’s an update or service pack on the OS, the IAAS provider will not automatically do it for you. Instead, you may need to do it by yourself.

Platform as a Service (PAAS)

Now, let’s jump into another cloud spectrum, “PAAS”, to see how it differs. In PAAS, the provisioning model is about an on-demand application hosting environment. Not only managing the component like an IAAS provider would, a PAAS provider will also help customers manage additional responsibilities such as OS, Middleware, Runtime, and even Databases, as can be seen on the right-hand side of Table 1.

In other words, you can think of PAAS as renting a stack of software, hardware, and infrastructure. Customer will just need to bring the application and data and they are ready to go.

Step-by-step: Setting-up an Application on PAAS Environment

For PAAS, given that the database server, VM, and web server VM are readily provisioned, you just need to do two steps, as illustrated by another slide from Mark Russinovich.

Figure 3: Provision and Deploy

1. Database Provisioning

You might need to indicate where (which region) your virtual DB Server is provisioned, but you don’t have to install a bunch of DBMS software on your own. You will need to provision the database, create tables, and add data.

2. Deploying Your Application

This is a similar step applicable to IAAS, you will still need to deploy your application on the PAAS cloud environment.

How about the load-balancer? Take Windows Azure as example, it will all automatically be configured and ready to take the traffic, and everything else will be automatically managed. You don’t have to worry about IP Addresses or a load-balancer.

How about maintaining VMs? The DBMS and Web Server VM will be maintained by the provider. For example:

• If the VM where your application is hosted has any hardware issues, the provider should be able to detect the failure and rectify it immediately to make sure that your application will stay up and running. In Windows Azure, Fabric Controller will be the component handling these kinds of issues.
• If there are new updates or patches on the Operating System, the provider will make sure that the VM your application sits on is always updated. For example: Windows Azure uses “Guest OS Version” to differentiate service updates. Of course you can also choose to stick to one version or auto-update.

Figure 4: Configuration

Summary

To summarize, we have investigated different service models and provisioning steps of IAAS and PAAS solutions. PAAS providers indeed take on much more responsibility for your solution than an IAAS provider would. On the other side, IAAS may offer more flexibility at lower level (example: public IP addresses, load-balancer, etc.).

There’s no one-size-fits-all here. As a developer or architect, you should understand a customer’s need and determine the correct model to get the best possible outcome.

This post was also published at A Cloud Place blog.

---

<Return to section navigation list>

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

Lori MacVittie (@lmacvittie) asserted “Oh, it certainly helps, but it’s not a requirement” in an introduction to her Hybrid Architectures Do Not Require Private Cloud post of 4/16/2012 to F5’s DeveloperCentral blog:

Taking advantage of cloud-hosted resources does not require forklift re-architecture of the data center. That may sound nearly heretical but that’s the truth, and I’m not talking about just SaaS which, of course, has never required anything more than an Internet connection to “integrate” into the data center.

I’m talking about IaaS and integrating compute and storage resources into the data center, whether it’s cloud-based or traditional or simply highly virtualized.

Extending the traditional data center using hybrid model means being able to incorporate (integrate) cloud-hosted resources as part of the data center. For most organizations this means elasticity – expanding and contracting capacity by adding and removing remote resources to a data center deployed application. Flexibility and cost savings drive this model, and the right model can realize the benefits of cloud without requiring wholesale re-architecture of the data center.

That’s something that ought to please the 50% of organizations that, according to a 2011 CIO survey, are interested in cloud specifically to increase capacity and availability. Bonus: it also serves to address other top drivers identified in the same survey of reducing IT management and maintenance as well as IT infrastructure investment.

Really Big Bonus? Most organizations probably have the means by which they can achieve this today.

LEVERAGING CLOUD RESOURCES FROM A TRADITIONAL DATA CENTER

Scalability requires two things: resources and a means to distribute load across them. In the world of application delivery we call the resources “pools” and the means to distribute them an application delivery controller (load balancing service, if you prefer).

The application delivery tier, where the load balancing service resides topologically in the data center, is responsible for not only distributing load across resources but for being able to mitigate failure without disrupting the application service. That goes for elasticity, too. It should be possible to add and remove (intentionally through provisioning processes or unintentionally through failure) resources from a given pool without disruption the overall application service.

This is the primary business and operational value brought to an organization by load balancing services: non-disruptive (or seamless or transparent if you prefer more positive marketing terminology) elasticity.

Yes, the foundations of cloud have always existed and they’re in most organizations’ data centers today.

Now, it isn’t that hard to imagine how this elasticity can extend to integrate cloud-hosted resources. Such resources are either non-disruptively added to/removed from the load balancing service’s “pool” of resources. The application delivery controller does not care whether the resources in the pool are local or remote, traditional or cloud, physical or virtual. Resources are resources.

So whether the data center is still very traditional (physical-based), has moved into a highly virtualized state, or has gone all the way to cloud is really not relevant to the application delivery service. All resources can be operationally managed consistently by the application delivery controller.

To integrate cloud-based resources into the architecture requires only one thing: connectivity.

The connectivity between a data center and the “cloud” is generally referred to as a cloud bridge (or some variation thereof). This cloud bridge has the responsibility of connecting the two worlds securely and providing a network compatibility layer that “bridges” the two networks, implying a transparency that allows resources in either environment to communicate without concern for the underlying network topology. How this is accomplished varies from solution to solution, and there are emerging “virtual network encapsulation” technologies (think VXLAN and GRE) that are designed to make this process even smoother.

Once a connection is established, and assuming network bridging capabilities, resources provisioned in “the cloud” can be non-disruptively added to the data center-hosted “pools” and from there, load is distributed as per the load balancing service’s configuration for the resource (application, etc… ).

THE ROAD to CLOUD

There seems to be a perception in the market that you aren’t going to get to hybrid cloud until you have private cloud, which may explain the preponderance of survey respondents who are focused on private cloud with must less focus on public cloud. The road to “cloud” doesn’t require that you completely revamp the data center to be cloud-based before you can begin taking advantage of public cloud resources. In fact, a hybrid approach that integrates public cloud into your existing data center provides an opportunity to move steadily in the direction of cloud without being overwhelmed by the transformation that must ultimately occur.

A hybrid traditional-cloud based approach allows the organization to build the skill sets necessary, define the appropriate roles that will be needed, and understand the fundamental differences in operational models required to implement the automation and orchestration that ultimately brings to the table all the benefits of cloud (as opposed to just the cheaper resources).

Cloud is a transformational journey – for both IT and the business – but it’s not one that can be taken overnight. The pressure to “go cloud” is immense, today, but IT still needs the opportunity to evaluate both the data center and cloud environments for appropriateness and to put into place the proper policies and governance structure around the use of cloud resources. A strategy that allows IT to begin taking advantage of cloud resources now without wholesale rip-and-replace of existing technology provides the breathing room IT needs to ensure that the journey to cloud will be a smooth one, where the benefits will be realized without compromising on the operational governance required to assure availability and security of network, data, and application resources.

---

<Return to section navigation list>

Cloud Security and Governance

Brian Hitney recommended the Windows Azure Trust Center in a 4/16/2012 post:

The Windows Azure team recently posted about the Windows Azure Trust Center. One of the most frequent conversations that comes up when discussing moving applications to the cloud revolves around security and compliance, and it’s also one of the most challenging conversations to have.

What makes it particularly challenging is the fact that the responsibility of compliance is typically shared between the hardware, platform, and software.

The site has a few sections that in particular drill down into security, privacy, and compliance related information. Definitely good information to refer to when evaluating a move into the cloud!

---

<Return to section navigation list>

Cloud Computing Events

Microsoft’s 30 To Launch (@UserCommunity) group announced a Your App. Our Cloud. One Journey program on 4/16/2012:

The world needs your app, and the cloud can supercharge it. Join 30 to Launch | Windows Azure – 30 days of specific technical and business guidance. When you graduate, submit your idea or app for the opportunity to win US$4,000 in cash. Get free cloud access for MSDN subscribers or startups.

Tried to sign up for the program with my existing Windows Azure Table Storage demo app and received the following message:

#FAIL!

---

Nathan Totten (@ntotten) and Nick Harris (@cloudnick) produced Cloud Connection Episode 76 – Meet our new additions to the Windows Azure Technical Evangelist Team on 4/13/2012:

Join Nate and Nick each week as they cover Windows Azure. You can follow and interact with the show at @CloudCoverShow.

In this episode, we would like you to meet our new additions to the Windows Azure Technical Evangelism Team namely Haishi Bai, Cory Fowler, Jon Galloway and Brady Gaster. Learn about their careers prior to Microsoft and find out what areas of Windows Azure they will be focusing on.

In the News:

• Windows Azure Trust Center Launched
• Announcing New Datacenter Options for Windows Azure
• Announcing the refresh of the Service Bus EAI and EDI Labs

In the Tip of the Week, Nathan [pictured at right] discusses a blog post by Sheik that looks at sync, async and parallel programming performance in WIndows Azure.

---

Aidan Finn (@joe_elway) reported MMS 2012, Here I Come on 4/13/2012:

I’ll be making my way to Las Vegas tomorrow morning and arriving there in the evening for my very first trip to the Microsoft Management Summit (MMS) 2012. MMS, if you don’t know it, it sometimes referred to as the System Center conference. While there might be presentations on other products, they will be few. The focus is purely on System Center 2012, automation, security, compliance, and the private cloud.

This event is so sold out that even the wait list was allegedly closed. There’s a huge amount of interest. But don’t despair if you couldn’t attend; the content will be made available online in a few ways:

As usual, the keynotes (by Brad Anderson, Corporate Vice President of the Management and Security Division at Microsoft) will be streamed live:

• Day One Keynote: Microsoft Private Cloud. Built for the Future. Ready Now. (Tuesday, April 17, 16:15-17:45AM GMT) – Cloud computing and the delivery of true IT as a Service is one of the most profound industry shifts in decades. Join Brad Anderson, Corporate Vice President of Microsoft’s Management and Security Division, as he shares Microsoft’s vision for cloud computing and shows how System Center 2012, as part of the Microsoft private cloud, will enable you to deliver the promise of cloud computing in your organization today.
• Day Two Keynote: A World of Connected Devices (Wednesday, April 18, 16:15-17:45AM) – Clouds and cloud-connected devices are changing the world of work and our daily interactions. Tech-savvy and always-connected, people want faster, more intuitive technology, uninterrupted services, and the freedom to work anywhere, anytime, on a variety of devices. Join Brad Anderson, Corporate Vice President of the Management and Security Division at Microsoft to learn how System Center 2012 and Windows Intune can help IT embrace this new reality today, and in the future, by making the right intelligent infrastructure investments.

The sessions will be available, I’d guess on Channel 9, 2 days after their original presentation. I might be blogging live, as I usually do at conferences like this. So keep a watch here!

The sessions I’m most interested in are possibly Orchestrator and Service Manager because they are the ones I have little or no experience of. A conference like this is also a great opportunity to network. I’ve a few meetings and side presentations lined up for the week, and I’m really looking forward to meeting up with fellow MVPs and others who I have “met” over the past years. And if I get the chance, I want to make some large holes in paper targets.

I’m surprised that Microsoft scheduled MMS for the same week as the giant NAB Show.

---

<Return to section navigation list>

Other Cloud Computing Platforms and Services

Ted Samson (@tsamson_IW) asserted “Company unveils array of new services, including a MySQL offering optimized for the cloud” in a deck for his Rackspace launches new OpenStack-based cloud portfolio article of 4/16/2012 for InfoWorld’s Cloud Computing blog:

Rackspace has unveiled an "early access" limited-availability edition of its updated public cloud environment, built on the open source cloud platform OpenStack. Dubbed simply Rackspace Cloud, its features include cloud servers, databases, block storage, networks, and monitoring, as well as a new control panel.

April has proven an eventful month for OpenStack, starting on a low note when Citrix announced plans to all but drop support for the platform in favor of focusing on CloudStack. Since then, a major new version of OpenStack called Essex was released, Hewlett-Packard unveiled a hybrid cloud built on OpenStack, and a group of major IT heavyweights, including IBM, Cisco Systems, and Dell, announced support for a forthcoming OpenStack Foundation.

RackSpace’s new OpenStack-based cloud environment will eventually replace its current platform, which was built on virtual platform technology the company acquired from Slicehost. Although similar to what today’s cloud infrastructure looks like, the Slicehost software was "never designed for the scale we’re at today," Rackspace CTO John Engates told InfoWorld. "It wasn’t distributed in nature, and it had to go through central pipelines."

The new cloud platform, Engates said, can scale to many thousands of servers, making it a viable choice for not just enterprises but service providers.

The Cloud Servers component is based on the latest OpenStack Compute release and, according to Engates, offers a level of scalability suitable for even service providers. Cloud Servers is accessible via the new OpenStack API as well as through a control panel. The Cloud Databases service also provides automated management of common database tasks.

The Cloud Databases component provides API access to a massively scalable, highly available MySQL database. Rackspace built servers and storage for the service designed specifically for running databases in the cloud, Engates said. Databases don’t fare well in today’s cloud environments, he said, because the highly virtualized infrastructures aren’t optimized to meet database’s high I/O demands. "Customers find that performance isn’t all they wish it would be — across all clouds, not just Rackspace."

The Cloud Monitoring service lets customers monitor both infrastructure and applications. It supports both Rackspace’s and competitors’ cloud servers. The service is agentless and thus incapable of, say, polling servers for CPUs, disk, and memory. ..

Read more.

---

Jeff Barr (@jeffbarr) reported the availability of a Microsoft SharePoint Server on AWS Reference Architecture White Paper in a 4/13/2012 post:

We have just published the Microsoft SharePoint Server on AWS Reference Architecture White Paper.

This white paper discusses general concepts regarding how to use SharePoint services on AWS and provides detailed technical guidance on how to configure, deploy, and run a SharePoint Server farm on AWS. It illustrates reference architecture for common SharePoint Server deployment scenarios and discusses their network, security, and deployment configurations so you can run SharePoint Server workloads in the cloud with confidence.

This white paper is targeted to IT infrastructure decision-makers and administrators. After reading it, you should have a good idea on how to set up and deploy the components of a typical SharePoint Server farm on AWS.

Here’s what you will find inside:

• SharePoint Server Farm Reference Architecture
• Common SharePoint Server Deployment Scenarios
  • Intranet SharePoint Server Farm
  • Internet Website or Service Based on SharePoint Server
• Implementing SharePoint Server Architecture Scenarios in AWS
  • Network Setup:
    • Amazon VPC setups for Intranet and Public Website Scenarios
    • AD DS Setup and DNS Configuration
  • Server Setup and Configuration
    • Mapping SharePoint Server Roles and Servers to Amazon EC2 AMIs and Instance Types
    • SharePoint and SQL Server Configurations
  • Security
    • Security Groups
    • Network ACLs
    • Windows Instance Security
    • Administrator Access
    • Data Privacy
  • Deployment
  • Monitoring and Management
  • Backup and Recovery

---

<Return to section navigation list>

http://oakleafblog.blogspot.com/2012/04/windows-azure-and-cloud-computing-posts_16.html