You are here : Windows Azure Blog » OakLeaf Systems » Windows Azure and Cloud Computing Posts for 2/24/2012+

Windows Azure and Cloud Computing Posts for 2/24/2012+

Posted On Sunday, March 4, 2012 By rss. Under OakLeaf Systems

A compendium of Windows Azure, Service Bus, EAI & EDI Access Control, Connect, SQL Azure Database, and other cloud-computing articles.

Note: This post includes articles published on 2/24/2012 through 3/3/2012 while I was attending the Microsoft Most Valuable Professionals (MVP) Summit 2012 in Seattle, WA.

Note: This post is updated daily or more frequently, depending on the availability of new articles in the following sections:

Azure Blob, Drive, Table, Queue and Hadoop Service

Denny Lee (@dennylee) posted BI and Big Data–the best of both worlds! on 2/29/2012:

As part of the excitement of the Strata Conference this week, Microsoft has been talking about Big Data and Hadoop. It started off with Dave Campbell’s question: Do we have the tools we need to navigate the New World of Data?. And some of the tooling call outs specific to Microsoft include references to PowerPivot, Power View, and the Hadoop JavaScript framework (Hadoop JavaScript– Microsoft’s VB shift for Big Data).

As noticed by GigaOM’s article Microsoft’s Hadoop play is shaping up, and it includes Excel; the great call out is:

to make Hadoop data analyzable via both a JavaScript framework and Microsoft Excel, meaning many millions of developers and business users will be able to work with Hadoop data using their favorite tools.

Big Data for Everyone!

The title of the Microsoft BI blog post says it the best: Big Data for Everyone: Using Microsoft’s Familiar BI Tools with Hadoop – it’s about helping make Big Data accessible to everyone by use of one of the most popular and powerful BI tools – Excel.

So what does accessible to everyone mean – in the BI sense? It’s about being to go from this (which is a pretty nice view of Hive query against Hadoop on Azure Hive Console)

and getting it Excel or PowerPivot.

The most important call out here is that you can use PowerPivot and Excel to merge data sets not just from Hadoop, but also bring in data sets from SQL Server, SQL Azure, PDW Oracle, Teradata, Reports, Atom feeds, Text files, other Excel files, and via ODBC – all within Excel! (thanks @sqlgal for that reminder!)

From here users can manipulate the data using Excel macros and PowerPivot DAX language respectively. Below is a screenshot of data extracted from Hive and placed into PowerPivot for Excel.

But even more cooler – data visualization wise – your PowerPivot for Excel workbook (once uploaded to SharePoint 2010 with SQL Server 2012) and you can create an interactive Power View report.

For more information on how to get PowerPivot and Power View to connect to Hadoop (in this case, its Hadoop on Azure but conceptually they are the same), please reference the links below:

How To Connect Excel to Hadoop on Azure via HiveODBC

Connecting PowerPivot to Hadoop on Azure – Self Service BI to Big Data in the Cloud

Connecting Power View to Hadoop on Azure

Connecting Power View to Hadoop on Azure [Video]

So what’s so Big about Big Data?

As noted by in the post What’s so Big about Big Data?, we call out that Big Data is important because of the sheer amount of machine generated data that needs to be made sense of.

As noted by Alexander Stojanovic (@stojanovic), the Founder and General Manager of Hadoop on Windows and Azure:

It’s not just your “Big Data” problems, it’s about your BIG “Data Problems”

To learn more, check out the my 24HOP (24 Hours of PASS) session:

Tier-1 BI in the Age of Bees and Elephants

In this age of Big Data, data volumes become exceedingly larger while the technical problems and business scenarios become more complex. This session dives provides concrete examples of how these can be solved. Highlighted will be the use of Big Data technologies including Hadoop (elephants) and Hive (bees) with Analysis Services. Customer examples including Klout and Yahoo! (with their 24TB cube) will highlight both the complexities and solutions to these problems.

Making this real, a great case study showcasing this includes the one at Klout, which includes a great blog post: Big Data, Bigger Brains. And below is a link to Bruno Aziza (@brunoaziza) and Dave Mariani’s (@dmariani) YouTube video on how Klout Leverages Hadoop and Microsoft BI Technologies To Manage Big Data.

Avkash Chauhan (@avkashchauhan) explained Primary Namenode and Secondary Namenode configuration in Apache Hadoop in a 2/27/2012 post:

Apache Hadoop Primary Namenode and secondary Namenode architecture is designed as below:

Namenode Master:

The conf/masters file defines the master nodes of any single or multimode cluster. On master, conf/masters that it looks like this:

localhost

This conf/slaves file lists the hosts, one per line, where the Hadoop slave daemons (datanodes and tasktrackers) will run. When you have both the master box and the slave box to act as Hadoop slaves, you will see same hostname is listed in both master and slave.

On master, conf/slaves looks like as below:

localhost

If you have additional slave nodes, just add them to the conf/slaves file, one per line. Be sure that your namenode can ping to those machine which are listed in your slave.

Secondary Namenode:

If you are building a test cluster, you don’t need to set up secondary name node on a different machine, something like pseudo cluster install steps. However if you’re building out a real distributed cluster, you must move secondary node to other machine and it is a great idea. You can have Secondary Namenode on a different machine other than Primary NameNode in case the primary Namenode goes down.

The masters file contains the name of the machine where the secondary name node will start. In case you have modified the scripts to change your secondary namenode details i.e. location & name, be sure that when the DFS service starts its reads the updated configuration script so it can start secondary namenode correctly.

In a Linux based Hadoop cluster, the secondary namenode is started by bin/start-dfs.sh on the nodes specified in conf/masters file. Initially bin/start-dfs.sh calls bin/hadoop-daemons.sh where you specify the name of master/slave file as command line option

Start Secondary Name node on demand or by DFS:

Location to your Hadoop conf directory is set using $HADOOP_CONF_DIR shell variable. Different distributions i.e. Cloudera or MapR have setup it differently so have a look where is your Hadoop conf folder.

To start secondary name node on any machine using following command:

$HADOOP_HOME/bin/hadoop –config $HADOOP_CONF_DIR secondarynamenode

When Secondary name node is started by DFS it does as below:

$HADOOP_HOME/bin/start-dfs.sh starts SecondaryNameNode

>>>> $bin”/hadoop-daemons.sh –config $HADOOP_CONF_DIR –hosts masters start secondarynamenode

In case you have changed secondary namenode name say “hadoopsecondary” then when starting secondary namenode, you would need to provide hostnames, and be sure these changes are available to when starting bin/start-dfs.sh by default:

$bin”/hadoop-daemons.sh –config $HADOOP_CONF_DIR –hosts hadoopsecondary start secondarynamenode

which will start secondary namenode on ALL hosts specified in file ” hadoopsecondary “.

How Hadoop DFS Service Starts in a Cluster:

In Linux based Hadoop Cluster:

Namenode Service : start Namenode on same machine from where we starts DFS .

DataNode Service : looks into slave file and start DataNode on all slaves using following command :

#>$HADOOP_HOME/bin/hadoop-daemon.sh –config $HADOOP_CONF_DIR start datanodeSecondaryNameNode Service: looks into masters file and start SecondaryNameNode on all hosts listed in masters file using following command

#>$HADOOP_HOME/bin/hadoop-daemon.sh –config $HADOOP_CONF_DIR start secondarynamenode

Alternative to backup Namenode or Avatar Namenode:

Secondary namenode is created as primary namenode backup to keep the cluster going in case primary namenode goes down. There are alternative to secondary namenode available in case you would want to build a name node HA. Once such method is to use avatar namenode. An Avatar namenode can be created by migrating namenode to avatar namenode and avatar namenode must build on a separate machine.

Technically when migrated Avatar namenode is the namenode hot standby. So avatar namenode is always in sync with namenode. If you create a new file to master name node, you can also read in standby avatar name node real time.

In standby mode, Avatar namenode is a ready-only name node. Any given time you can transition avatar name node to act as primary namenode. When in need you can switch standby mode to full active mode in just few second. To do that, you must have a VIP for name node migration and a NFS for name node data replication.

The Namenode Master paragraph is a bit mysterious.

SQL Azure Database, Federations and Reporting

Cihan Biyikoglu (@cihangirb) started a Scale-First Approach to Database Design with Federations: Part 1 – Picking Federations and Picking the Federation Key series on 2/29/2012:

Scale-out with federations means you build your application with scale in mind as your first and foremost goal. Federations are build around this idea and allow application to annotate their schema with additional information for declaring its scale-out intent. Generating your data model and your database design is basic parts of the app design. At this point of the app lifecycle, it is also time to pick your federations and federations key and work in the scale-first principals into your data and database design.

In a series of posts, I’ll walk through the process of designing, coding and deploying applications with federations. If you’d like to design a database with scalability in mind using sharding [as] the technique, this post will also help you get there as well… This is a collection of my personal experience designing a number of sharded databases in past life and experience of some of the customers in the preview program we started back in June 2011, plus the experiences of customers who are going live with federations.

In case you missed earlier posts here is a quick definition of federation and federation key;

Federation is an object defined to scale out parts of your schema. Every database can have many federations. Federations use federation members which are regular sql azure databases to scale out your data in one or many tables and all associated rich programming properties like views, indexes, triggers, stored procs. Each federation has a name and a federation key which is also called a distribution scheme. Federation key or federation distribution scheme defines 3 properties;

A federation key label, help self document the meaning of the federation key like tenant_id or product_id etc. ,

A data domain to define the distribution surface for your data. In v1, data domains supported are INT, BIGINT, UNIQUEIDENTIFIER (guid) and VarBinary – up to 900 bytes.

A distribution style, to define how the data is distributed to the data domain. At this point distribution style can only be RANGE.

Lets assume you are designing the AdventureWorks (AW) database. You have a basic idea on the entities at play like Stores, Customers, Orders, Products, SalesPeople etc and thinking about how you’d like to scale it out…

Picking your Federations:

Picking your federations and federations keys is much like other steps of data modeling and database design processes like normalization. You can come up with multiple alternatives and they optimize for various situations. You can get to your final design only by knowing the transactions and queries that are critical to your app.

Step #1: Identify entities you want to scale out: You first identify the entities (collection of tables) in your database that is going to be under pressure and needs scale out. These entities are your candidate federations in your design.

Step #2: Identify tables that make up these entities: After identifying these central entities you want to scale out like customer, order and product, it is fairly easy to traverse the relationships and identify the group of objects tied to these entities with relationships, access patterns and properties to these set of entities. Typically these are groups of tables with one-to-one or on-to-many relationships.

Picking your Federation Key:

Step #3: Identify the Federation Key: Federation key identify the key used for distributing the data and define the boundary of atomic units. Atomic units are rows across all scaled out tables (federated tables) that share the same federation key instance. One important rule in federation is that atom cannot be SPLIT. Ideal federation keys have the following properties;

Atomic Units is the target of most query traffic & transaction boundaries.

Distributes workload equally to all members equally – decentralize load to many atomic units as opposed to concentrating the load.

Atomic units cannot be split, so largest atomic unit does not exceed the boundaries of a single federation member.

Walking through AW

The database design for AW is something I hope you are already familiar with. You can fid details here on the schema for SQL Azure; http://msftdbprodsamples.codeplex.com/releases/view/37304. For this app we want to be able to handle Mil customers, 100 Mil orders, and Mil products. These are the largest entities in our database. I’ll add a few more details on the workload to help guide our design; here are our most popular transactions and queries;

‘place an order’, ‘track/update orders’, ‘register/update a customer’, ‘get customer orders’,’get top ordering customers’, ‘register/update products’, ’get top selling products’

and here are some key transactions and consistent query requirements;

‘place an order’, ‘import/export orders for a customer and stores’, ‘monthly bill for customers’

Step #1: We have the classic sales database setup with customer, order and product in AW. In this example, we expect orders to be the parts that will be most active, the tables will be the target of most of our workload. We expect many customers and also handle cases where there are large product catalogs.

Step #2: In AW, Store and customer tables are used to identify the customer entity. SalesTerritory, SalesOrdersHeader and SalesOrderDetail tables contain properties of orders. Customer and Order entities have one-to-many relationship. On the other hand, Product entity has a many-to-many back to Order and to Customer entity. When scaling out, you can align one-to-one and one-to-many relationships together but not many-to-many relationships. Thus we can only group Customers and Orders together but not products.

Step #3: Given the Customer (store and customer tables) and Order (SalesOrderHeader, SalesrderDetail, SalesTerritory) we can think of a few setups here for the federation key.

- StoreID as the Federation Key:

Well that would work for all transactions so #1 of ideal federation key principal taken care of! That is great.

However stores may have varying size and may not distribute the load well if there the customer and order traffic variance between stores is too wide. Not so great on #2 principal of ideal federation keys.

StoreID as a federation key will mean all customers and all their orders in that store will be a single atomic unit (AU). If you have stores that could get large enough to challenge the computational capacity of a single federation member, you will hit the ‘split the atom’ case and get stuck because you cannot.

So StoreID may be too coarse a granule to equally distribute load and may be too large an AU if a store gets ‘big’. By the way, TerritoryID is a similar alternative and has a very similar set of issues so same argument applies to that as well.

- OrderID as the Federation Key:

OrderID certainly satisfy #2 and #3 of the ideal federation key principals but has an issue with #1 so lets focus on that.

That could work as well but is too fine a granule for queries that are orders per customer in the system. It also won’t align with transactional requirements of import/export of customer and store orders. Another important note; with this setup, we will need a separate federation for the Customer entity. It means that queries that may be common like ‘get all orders of a customer’ or ‘get order of customer dated X/Y/Z’ will need to hit all or at least multiple members. Also with this setup we lose ability to transact multiple order from a customer. We may do that when we are importing or exporting a customers orders.

Fan-out is not necessarily bad. It promotes parallel executions and can provide great efficiencies. However efficiencies are lost when we hit all members and can’t fast-eliminate members that don’t have any data to return and when cost of parallelization overwhelms the processing of the query. With OrderID as the federation key, queries like ‘get orders of a customer’ or ‘get top products per customer’ will have to hit all members.

- OrderDetailID as the Federation Key:

The case of OrderDetailID has the same issues as OrderID case above but amplified on principal #1. With this setup, we will lose transactional boundary to place a single order in a transaction. In this case, there will be more queries that will need to be fanned-out like ‘get all customer orders’ or ‘get an order’… Makes assembling order a full fan-out query which can get quite expensive.

- CustomerID as the Federation Key:

With CustomerID #2 and #3 will not likely be an issue. The only case is where a customer gets so large that it overwhelms a member and computational capacity of AU. For most cases, CustomerID can be a great way to decentralize the load getting around issues StoreID or TerritoryID would create.

This setup also satisfy almost all of #1 as well except 2 cases; one is ‘get top selling products across all customers’. However that case isn’t satisfied in any of the other alternatives either. This setup does focus the DML (INSERT/UPDATE/DELETE) transactions and satisfy both multiple order and single order placement transaction to work seamlessly. So looks like a good choice for #1 standpoint. Second is import/export at store boundary; for example import of stores all customers and orders will not be possible with this setup in a transaction. Some people may be able to live with consistency at the customer level and be ok relaxing consistency at the store level. you need to ask yourself; Can you work with eventual consistency without transactions at the store level by depending on things like datetime stamps or some other store level sequence generator. if you can this is the right choice.

How about Product Entity?

We have not touched on how to place products in this case. To remind you the issue; there is a many-to-many relationship in customers vs orders thus federation aligned with orders and/or customers Well, you have 3 choices when it comes to products.

- Products as a Central Table: In this case, you’d leave Product entity in the root. That would risk making root and products a bottleneck. Especially if you have a fast updating catalog of products and you don’t build caching facilities to minimize hitting the root for product information for popular queries and transaction in the system. The advantage to this setup is that product entity can be maintained in a single place with full consistency.

- Product as a Reference Table: In this case, you would place a copy of the products entity in each federation member. This would mean you need to pay more for storing this redundant information. This would mean updating the product catalog will have to be done across many copies of the data and will mean you need to live with eventual consistency on product entity across members. That is at any moment in time, copies of products in member 1 & 2 may not be identical. Upside is this would give you good performance like local joins.

- Product as a separate Federation: In this case, you have a separate federation with a key like productID that holds product entity = all the tables associated with that. You would set up products in a fully consistent setup so there would not be redundancy and you would be setting up products with great scale characteristics. You can independently decide how many nodes to engage and choose to expand if you run out of computational capacity for processing product queries. Downside compared to the reference table option is that you no longer enjoy local joins.

To tie all this together, given the constraints most people will choose customerID as the fed key and place customers and orders in the same federation and given the requirement to handle large catalogs, most people will choose a separate federation for products.

In this case, I choose a challenging database schema to show the highlight the variety of options and for partition-able workloads life isn’t this complicated. However for sophisticated schemas with variety of relationships, there pros and cons to evaluate much like other data and database design exercises. It is clear that doing design with federations puts ‘scale’ as the top goal and concern and that may mean that you evaluate some compromises compromises on transactions and query processing.

In part 2, I’ll cover the schema changes that you need to make with federations which is step 2 in designing o migrating an existing database design over to federations.

Steve Jones (@way0utwest) asserted The Cloud is good for your career in a 2/29/2012 post to the Voice of the DBA blog:

I think most of us know that the world is not really a meritocracy. We know that the value of many things is not necessarily intrinsic to the item; value is based on perception. That’s a large part of the economic theory of supply and demand. The more people want something, the more it should cost.

I ran across a piece that surveyed some salaries and it shows that people working with cloud platforms are commanding higher salaries, even when the underlying technologies are the same. It seems crazy, but that’s the world we live in. Perceptions drive a lot of things in the world, especially the ones that don’t seem to make sense.

Should you specialize in cloud technologies? In one sense, the platform is not a lot different than what you might run in your local data center. Virtualized machines, connections and deployments to remote networks, and limited access to the physical hardware. There are subtle differences, and learning about them and working with them, could be good for your next job interview, when the HR person or technology-challenged manager asks you about Azure.

Part of your career is getting work done, using your skills and talents in a practical and efficient manner. However a larger part of it, IMHO, is the marketing of your efforts and accomplishments. The words you choose, the way you present yourself, these things matter. I would rather be able to talk about SQL Azure as a skill, and then relate that to local Hyper-V installations of SQL Server than the other way around.

If you are considering new work, or interesting in the way cloud computing might fit into an environment, I’d urge you to take a look at SQL Azure or the Amazon web services. You can get a very low cost account for your own use, and experiment. You might even build a little demo for your next interview that impresses the person who signs the paychecks.

Cihan Biyikoglu (@cihangirb) asked What would you like us to work on next? in a 2/28/2012 post:

10 weeks ago we shipped federations in SQL Azure and it is great to see the momentum grow. This week, I’ll be spending time with our MVPs in Redmond and we have many events like this where we get together with many of you to talk about what you’d like to see us deliver next on federations. I’d love to hear from the rest of you who don’t make it to Redmond or to one of the conferences to talk to us; what would you like us to work on next?

If you like what you see today, what would make it even better? if you don’t like what you see today, why? what experience would you like to see taken care of, simplified? Could be big or small. Could be NoSQL or NewSQL or CoSQL or just plain vanilla SQL. Could be in APIs, tools or in Azure or in SQL Azure gateway or the fabric or the engine? Could be the obvious or the obscure…

Open field; fire away and leave a comment or simply tweet #sqlfederations and tell us what you’d like us to work on next…

Peter Laudati (@jrzyshr) posted Get Started with SQL Azure: Resources on 2/28/2012:

Earlier this month, SQL Azure prices were drastically reduced, and a new 100mb for $5 bucks a month pricing level was introduced. This news has certainly gotten some folks looking at SQL Azure for the first time. I thought I’d share some resources to help you get started with SQL Azure.

Unlike the new “Developer Centers” for .NET, Node.js, Java, and PHP on WindowsAzure.com, there does not appear to be a one-stop shop for finding all of the information you’d need or want for SQL Azure. The information is out there, but it is spread around all over the place.

I’ve tried to organize these into three high-level categories based on the way one might think about approaching this platform:

What do I need to know about SQL Azure to get started?

Can I migrate my data into it?

How do I achieve scale with it?

Note: This is by no means an exhaustive list of every SQL Azure resource out there. You may find (many) more that I’m not aware of. You also may come across documentation & articles that are older and possibly obsoleted by new features. Be wary of articles with a 2008 or 2009 date on the by line.

Getting Started

Understanding Storage Services in Windows Azure

Storage is generally provided “as-a-service” in Windows Azure. There is no notion of running or configuring your own SQL Server ‘server’ in your own VM. Windows Azure takes the hassle of managing infrastructure away from you. Instead, storage services are provisioned via the web-based Windows Azure management portal, or using other desktop-based tools. Like in a restaurant, you essentially look at a menu and order what you’d like.

Storage services in Windows Azure are priced and offered independently of compute services. That is, you do not have to host your application in Windows Azure to use any of the storage services Windows Azure has to offer. For example, you can host an application in your own datacenter, but store your data in Azure with no need to ever move your application there too. Exploring Windows Azure’s storage services is an easy (and relatively low-cost) way to get started in the cloud.

There are currently three flavors of storage available in Windows Azure:

Local storage (in the compute VMs that host your applications)

Non-relational storage (Blobs, Tables, Queues – a.k.a. “Windows Azure Storage”)

Relational storage (SQL Azure)

For a high-level overview of these, see: Data Storage Offerings in Windows Azure

Note: This post is focused on resources for SQL Azure only. If you’re looking for information on the non-relational storage services (Blobs, Tables, Queues), this is post “is not the droids you’re looking for”.

Getting Started With SQL Azure

I started my quest to build this post at the new WindowsAzure.com (“new” in December 2011). Much of the technical content for the Windows Azure platform was reorganized into this new site. Some good SQL Azure resources are here. Others are still elsewhere. Let’s get started…

Start here:

What is SQL Azure? – This page on WindowsAzure.com explains what SQL Azure is and the high-level scenarios it is good for.

Business Analytics – This page on WindowsAzure.com explains SQL Azure Reporting at a high-level.

Overview of SQL Azure – This whitepaper is linked from the “Whitepapers” page on WindowsAzure.com. It appears to be older (circa 2009), but appears to provide a still relevant overview of SQL Azure

SQL Azure Migration Wizard (Part 1): SQL Azure – What Is It? – This screencast on Channel 9 by my colleagues Dave Bost and George Huey may have “Migration Wizard” in the title, but it goes here in the first section. These guys provide a good high-level overview of what SQL Azure is.

Get your hands dirty with the equivalent of a “Hello World” example:

How To Use SQL Azure – Dive right in. This article walks you through setting up a simple SQL Azure database and then connecting to it from a .NET application.

Managing SQL Azure Servers and Databases Using SQL Server Management Studio – SQL Azure can be managed via many different tools. One of the most popular is SQL Server Management Studio (SSMS). This article walks you through the basics of doing that.

BIG FLASHING NOTE #1:

You must have SQL Server Management Studio 2008 R2 SP1 to manage a SQL Azure database! SSMS 2008 and SSMS 2008 R2 are just NOT good enough. If you don’t have SSMS 2008 R2 SP1, it will cause a gap in the space time continuum! The errors you will receive if you don’t have SSMS 2008 R2 SP1 are obscure and not obvious indicators of the problem. You may be subject to losing valuable hours of your personal time seeking the correct solution. Be sure you have the right version.

BIG FLASHING NOTE #2:

You CAN run SQL Server Management Studio 2008 R2 SP1 even if you’re NOT running SQL Server 2008 R2 SP1. For example, if you need to still run SQL Server 2008 R2, 2008, or older edition, you can install SSMS 2008 R2 SP1 side-by-side without impacting your existing database installation. Disclaimer: Worked on my machine.

BIG FLASHING TIP:

How can I get SQL Server Management Studio 2008 R2 SP1?

Unfortunately, I found it quite difficult to parse through documentation to find the proper download for this. Searching for the “ssms 2008 r2 sp1 download” on Google or Bing will give will present you with Microsoft Download center pages that have multiple file download options. I present you with two options here:

Microsoft SQL Server 2008 R2 SP1 – Express Edition – This page contains multiple download files to install the Express edition of SQL Server. The easiest thing to do here is download either SQLEXPRWT_x64_ENU.exe or SQLEXPRWT_x86_ENU.exe depending on your OS-version (32 vs 64 bit). These files contain both the database and the management tools. When you run the installation process, you can choose to install ONLY the management tools if you don’t want to install the database on your machine.

Microsoft SQL Server 2008 R2 SP1 – This page contains multiple download files to install just SP1 to an existing installation of SQL Server 2008 R2. If you already have SQL Server Management Studio 2008 R2, you can run SQLServer2008R2SP1-KB2528583-x86-ENU.exe or SQLServer2008R2SP1-KB2528583-x64-ENU.exe, depending on your OS version (32 or 64 bit) to upgrade your existing installation to SP1.

The reason I call so much attention to this issue is because it is something that WILL cause you major pain if you don’t catch it. While some documents call out that you need SSMS 2008 R2 SP1, many do not provide the proper download links and send you on a wild goose chase looking for them. Thank me. I’ll take a bow.

The next place I recommend spending time reading is the SQL Azure Documentation on MSDN.

Content here is broken down into three high-level categories:

SQL Azure Database – This is the top of a treasure trove of good content.

SQL Azure Reporting

SQL Azure Data Sync

You can navigate the tree on your own, but some topics of interest might be:

SQL Azure Overview – I’m often asked what’s the difference between SQL Azure & SQL Server. This sheds some light on that.

Guidelines & Limitations – This gets a little more specific on SQL Server features NOT supported on SQL Azure.

Development: How-to Topics – There’s a smorgasbord of “How To” links here on how to connect to SQL Azure from different platforms and technologies.

Administration – All the details you need to know to manage your SQL Azure databases. See the “How-to” sub-topic for details on things like backing up your database, importing/exporting data, managing the firewall, etc.

Accounts & Billing in SQL Azure – Detailed info on pricing & billing here. (Be sure to see my post clarifying some pricing questions I had.)

Tools & Utilities Support – Many of the same tools & utilities you use to manage SQL Server work with SQL Azure too. This is a comprehensive list of them and brief overview of what each does.

Windows Azure Training Kit – No resource list would be complete without the WATK! The WATK contains whitepapers, presentations, demo code, and labs that you can walkthrough to learn how to use the platform. This kit has grown so large, it has its own installer! You can selectively install only the documentation and sample labs that you want. The SQL Azure related content here is great!

Migrating Your Data

The resources in the previous section should hopefully give you a good understanding of how SQL Azure works and how to do most basic management of it. The next task most folks want to do is figure out how to migrate their existing databases to SQL Azure. There are several options for doing this.

Start here: Migrating Databases to SQL Azure – This page in MSDN provides a high-level overview of the various options.

Three migration tools you may find yourself interested in:

SQL Azure Migration Wizard – The SQLAzureMW is an open source project on CodePlex. It was developed by my colleague George Huey. This is a MUST have tool in your toolbox!
SQLAzureMW is designed to help you migrate your SQL Server 2005/2008/2012 databases to SQL Azure. SQLAzureMW will analyze your source database for compatibility issues and allow you to fully or partially migrate your database schema and data to SQL Azure.

SQL Azure Migration Wizard (SQLAzureMW) is an open source application that has been used by thousands of people to migrate their SQL database to and from SQL Azure. SQLAzureMW is a user interactive wizard that walks a person through the analysis / migration process.

SQL Azure Migration Assistant for MySQL

SQL Azure Migration Assistant for Access

Channel 9 SQL Azure Migration Wizard (Part 2): Using the SQL Azure Migration Wizard – For people who are new to SQL Azure and just want to get an understanding of how to get a simple SQL database uploaded to SQL Azure, George Huey & Dave Bost did a Channel 9 video on the step by step process of migrating a database to SQL Azure with SQLAzureMW. This is a good place to to get an idea of what’s involved.

Tips for Migrating Your Applications To The Cloud – MSDN Magazine article by George Huey & Wade Wegner covering the SQLAzureMW.

Overview of Options for Migrating Data and Schema to SQL Azure – I found this Wiki article on TechNet regarding SQL Azure Migration. It appears to be from 2010, but with updates as recent as January 2012. The information here appears valid still.

Scaling with SQL Azure

Okay, you’ve figured out how to get an account and get going. You’re able to migrate your existing databases to SQL Azure. Now it’s time to take it to the next level: Can you scale?

Just because you can migrate your existing SQL Server database to SQL Azure doesn’t mean it will scale the same. SQL Azure is a multi-tenant “database-as-a-service” that is run in the Azure datacenters on commodity hardware. That introduces a new set of concerns regarding performance, latency, competition with other tenants, etc.

I recommend watching this great video from Henry Zhang at TechEd 2011 in Atlanta, GA:

Microsoft SQL Azure Performance Considerations and TroubleshootingIn this talk, Henry goes deep on how SQL Azure is implemented under the covers, providing you a better understanding of how the system works. He covers life in a mutli-tenant environment, including throttling, and how to design your databases for it. (Henry’s talk is an updated version of one delivered by David Robinson at TechEd Australia in 2010.)

There is a 150GB size limit on SQL Azure databases (recently up from 50GB). So what do you do if you’re relational needs are greater than that limit? It’s time to learn about art of sharding and SQL Azure Federation. While SQL Azure may take away the mundane chores of database administration (clustering/replication/etc), it does introduce problems which require newer skillsets to solve. This is a key example of that.

Start off by watching this video by Chihan Biyikoglu from TechEd 2011 in Atlanta, GA:

Building Scalable Database Solutions Using Microsoft SQL Azure Database Federations – Chihan Biyikoglu

In this talk Chihan explains what a database federation is, and how they work in SQL Azure.

Note: This talk is from May 2011 when SQL Azure Federations were only available as a preview/beta. The SQL Azure Federations feature was officially released into production in December 2011. So there may be variances between the May video and current service feature. He released a short updated video here.

Next read George Huey’s recent MSDN Magazine Article:

SQL Azure: Scaling Out with SQL Azure Federation
In this article, George covers the what, the why, and the how of SQL Azure Federations, and how SQL Azure Migration Wizard and SQL Azure Federation Data Migration Wizard can help simplify the migration, scale out, and merge processes. This article is geared to architects and developers who need to think about using SQL Azure and how to scale out to meet user requirements. (Chihan, from the previous video, was a technical reviewer on George’s article!)

Follow that up George Huey as a guest on Cloud Cover Episode #69:

Channel 9 Episode 69 – SQL Azure Federations with George Huey – George covers a lot of the same information with Wade Wegner in a Channel 9 Cloud Cover session on SQL Azure Federations. As the article above, this video covers what SQL Azure Federations is and the process of migrating / scaling out using SQL Azure Federations and some of the architectural considerations that need to be considered during the design process.

As a follow-up to the SQL Azure Migration Wizard, George has also produced another great tool:

SQL Azure Federation Data Migration Wizard (SQLAzureFedMW)
SQL Azure Federation Data Migration Wizard simplifies the process of migrating data from a single database to multiple federation members in SQL Azure Federation.
SQL Azure Federation Data Migration Wizard (SQLAzureFedMW) is an open source application that will help you move your data from a SQL database to (1 to many) federation members in SQL Azure Federation. SQLAzureFedMW is a user interactive wizard that walks a person through the data migration process.

That about wraps up my resource post here. Questions? Feedback? Leave it all below in the comments! Hope this helped you on your way to learning SQL Azure.

MarketPlace DataMarket, Social Analytics, Big Data and OData

Glenn Gailey (@ggailey777) posted Blast from the Past: Great Post on Using the Object Cache on the Client on 3/2/2012:

In the course of trying to answer a customer question on how to leverage the WCF Data Services client for caching, I came across a great blog post on the subject—which I had almost completely forgotten about:

Working with Local Entities in Astoria Client

(This post—by OData developer Peter Qian—is, in fact, so old that is refers to WCF Data Services by its original code name “Astoria.”)

The customer was looking for a way to maintain a read-only set of relatively static data from an OData service in memory so that this data could be exposed by his web app. As Peter rightly points out, the best thing to do is use a NoTracking merge option when requesting the objects. In this case, the object data is available in the Entities collection of the DataServiceContext and can be exposed in various ways. The entity data stored by the context is wrapped in an EntityDescriptor that includes the entity’s tracking and metadata, so some fancier coding is involved to expose this cached data as an IQueryable<T>, which have the LINQ—iness that we all really want.

Just re-read the post yourself, and see if you agree with me that it’s a rediscovered gem for using the WCF Data Services client.

Tip:

Remember to try and avoid the temptation to use IQueryable<T> collections on the client as the data source for another OData feed (using the reflection provider). This kind of republishing using the WCF Data Services client can get you into a lot of problems. This is because the WCF Data Services client library does not support the full set of LINQ queries used by an OData service.

David Campbell asked Do we have the tools we need to navigate the New World of Data? in a 2/29/2012 post to the SQL Server Team blog:

Last October at the PASS Summit we began discussing our strategy and product roadmap for Big Data including embracing Hadoop as part of our data platform and providing insights to all users on any data. This week at the Strata Conference, we will talk about the progress we have been making with Hortonworks and the Hadoop ecosystem to broaden the adoption of Hadoop and the unique opportunity for organizations to derive new insights from Big Data.

In my keynote tomorrow, I will discuss a question that I’ve been hearing a lot from the customers I’ve been talking to over the past 18 months, “Do we have the tools we need to navigate the New World of Data?” Organizations are making progress at learning how to refine vast amounts of raw data into knowledge to drive insight and action. The tools used to do this, thus far, are not very good at sharing the intermediate results to produce “Information piece parts” which can be combined into new knowledge.

I will share some of the innovative work we’ve been doing both at Microsoft and with members of the Hadoop community to help customers unleash the value of their data by allowing more users to derive insights by combining and refining data regardless of the scale and complexity of data they are working with. We are working hard to broaden the adoption of Hadoop in the enterprise by bringing the simplicity and manageability of Windows to Hadoop based solutions, and we are expanding the reach with a Hadoop based service on Windows Azure. Hadoop is a great tool but, to fully realize the vision of a modern data platform, we also need a marketplace to search, share and use 1^st and 3^rd party data and services. And, to bring the power to everyone in the business, we need to connect the new big data ecosystem to business intelligence tools like PowerPivot and Power View.

There is an amazing amount of innovation going on throughout the ecosystem in areas like stream processing, machine learning, advanced algorithms and analytic languages and tools. We are working closely with the community and ecosystem to deliver an Open and Flexible platform that is compatible with Hadoop and works well with leading 3^rd party tools and technologies enabling users of non-Microsoft technologies to also benefit from running their Hadoop based solutions on Windows and Azure.

We have recently reached a significant milestone in this journey, with our first series of contributions to the Apache Hadoop projects. Working with Hortonworks, we have submitted a proposal to the Apache Software Foundation for enhancements to Hadoop to run on Windows Server and are also in the process of submitting further proposals for a JavaScript framework and a Hive ODBC Driver.

The JavaScript framework simplifies programming on Hadoop by making JavaScript a first class programming language for developers to write and deploy MapReduce programs. The Hive ODBC Driver enables connectivity to Hadoop data from a wide range of Business Intelligence tools including PowerPivot for Excel.

In addition, we have also been working with several leading Big Data vendors like Karmasphere, Datameer and HStreaming and are excited to see them announce support for their Big Data solutions on our Hadoop based service on Windows Server & Windows Azure.

Just 10 years ago, most business data was locked up behind big applications. We are now entering a period where data and information become “first class” citizens. The ability to combine and refine these data into new knowledge and insights is becoming a key success factor for many ventures.

A modern data platform will provide new capabilities including data marketplaces which offer content, services, and models; and it will provide the tools which make it easy to derive new insights taking Business Intelligence to a whole new level.

This is an exciting time for us as we also prepare to launch our data platform that includes SQL Server 2012 and our Big Data investments. To learn more on what we are doing for Big Data you can visit www.microsoft.com/bigdata.

Windows Azure Access Control, Service Bus and Workflow

Alan Smith reported PDF and CHM versions of Windows Azure Service Bus Developer Guide Available & Azure Service Bus 2-day Course in a 3/1/2012 post:

I’ve just added PDF and CHM versions of “Windows Azure Service Bus Developer Guide”, you can get them here.

The HTML browsable version is here.

I have the first delivery of my 2-day course “SOA, Connectivity and Integration using the Windows Azure Service Bus” scheduled for 3-4 May in Stockholm. Feel free to contact me via my blog if you have any questions about the course, or would be interested in an on-site delivery. Details of the course are here.

Alan Smith announced availability of this Windows Azure Service Bus Developer Guide in a 2/29/2012 post:

I’ve just published a web-browsable version of “Windows Azure Service Bus Developer Guide”. “The Developers Guide to AppFabric” has been re-branded, and has a new title of “Windows Azure Service Bus Developer Guide”. There is not that much new in the way of content, but I have made changes to the overall structure of the guide. More content will follow, along with updated PDF and CHM versions of the guide.

Windows Azure VM Role, Virtual Network, Connect, RDP and CDN

Avkash Chauhan (@avkashchauhan) described Windows Azure CDN and Referrer Header in a 2/27/2012 post:

The Windows Azure Azure CDN, like any other CDNs, attempts to be a transparent caching layer. The CDN doesn’t care who the referring site might be. Like any other CDN, Windows Azure CDN keep things transparent and have no concern of what the referring site is. So it is correct to say that Windows Azure CDN does not have any dependency on Referrer Header. Any client solution created using referrer header will have no direct impact on how Windows Azure CDN works.

If you try to control CDN access based on be referrer, that may be a good idea because it is very easy to use tools like wget or curl to create a http request which has your referrer URL. Windows Azure CDN does not publicly support any mechanism for authentication or authorization of requests. If a request is received, it’s served. There is no way for Windows Azure CDN to reject requests based on referrer header or on any other header.

Live Windows Azure Apps, APIs, Tools and Test Harnesses

Liam Cavanagh (@liamca) continued his series with What I Learned Building a Startup on Microsoft Cloud Services: Part 6 – Securing the service and building trust on 2/28/2012:

I am the founder of a startup called Cotega and also a Microsoft employee within the SQL Azure group where I work as a Program Manager. This is a series of posts where I talk about my experience building a startup outside of Microsoft. I do my best to take my Microsoft hat off and tell both the good parts and the bad parts I experienced using Azure.

I knew my next step was to get an SSL certificate for my domain so that I could secure communications through https and encrypt the user’s connection information that is stored in the Cotega system database. Before I did this I needed to create a domain name. Unfortunately Windows Azure does not provide domain name registration or SSL certificates so I decided to look into using GoDaddy which is also where this blog is hosted. I found a domain name I liked through the GoDaddy auctions for about $10 called Cotega. I was also pretty lucky because there was a deal on the GoDaddy SSL Certificate for $12 / year. If I were to do this again and there was not a deal on the GoDaddy SSL certificate, I would probably take a closer look at FreeSSL’s free SSL certificate as I have read about other startups who chose this option.

Integrating SSL Certificates and CNames

Since I am hosting my web site on DiscountASP.NET I needed to load the certificate there. To configure the domain name, you need to set up a CName. A CName is basically a redirection that happens when someone enters www.cotega.com into a browser. In the background the CName links www.cotega.com to my DiscountASP.NET hosted IP address. This also works with the SSL certificate that is hosted with my web site so that https and http both leverage this IP address.

If I had of known at the start that I was going to host my web site on DiscountASP.net, I would have also used them to provide my domain name. Luckily I was able to use my previously created GoDaddy SSL Certificate and domain name with only minor configuration changes on DiscountASP.net.

Self Signed Certificates

The other thing that I wanted to use a certificate for is the encryption of users connection strings that are stored in the Cotega system database. For this I wanted to have a really high level of encryption so I chose to create an unsigned 2,048 bit MD5 X.509 certificate using the makecert.exe utility. You would not want to use a makecert generated certificate for your SSL traffic as browsers will reject an unsigned certificate. If you want to learn more about this, there is a great overview of the issues here. However, for encrypting the connection information within my MVC controller, this was a perfect solution because I control the entire encryption process which means the “man in the middle attack” does not apply but I could implement a really high level of encryption for my users.

Working with Customers to Building Trust

One of the interesting differences I learned in building services at Microsoft vs. Cotega is in the area of trust. At both Microsoft and Cotega, I spend a lot of time making sure a user’s information is secured. However, at Microsoft I did not have to spend as much time explaining this to customers. I think this comes from their past experiences with Microsoft where they have learned of Microsoft privacy and security policies and have grown over time to trust them. However, at Cotega, the service and company is new and has not built the level of trust that Microsoft has. As a result, I found that it was important for me to work closely with these customers to ensure that I can build that level trust which has primarily come from one-on-one discussions. A side benefit of this close relationship with customers is the early feedback I get on the service. But I will talk about that more later…

Ryan Dunn (@dunnry) described Setting Up Diagnostics Monitoring In Windows Azure in a 2/27/2012 post:

In order to actually monitor anything in Windows Azure, you need to use the Diagnostics Manager (DM) that ships out of box. SaaS providers like AzureOps rely on this data in order to tell you how your system is behaving. The DM actually supports a few data sources that it can collect and transfer:

Performance Counters

Trace logs

IIS Logs

Event Logs

Infrastructure Logs

Arbitrary logs

One of the most common issues I hear from customers is that they don’t know how to get started using the DM or they think they are using it and just cannot find the data where they think they should. Hopefully, this post will clear up a bit about how the DM actually works and how to configure it. The next post will talk about how to get the data once you are setup.

Setting UP The Diagnostics Manager

Everything starts by checking a box. When you check the little box in Visual Studio that says "Enable Diagnostics", it actually modifies your Service Definition to include a role plugin. Role plugins are little things that can add to your definition and configuration similar to a macro. If you have ever used Diagnostics or the RDP capability in Windows Azure, you have used a role plugin. For most of their history, these plugins have been exclusively built by Microsoft, but there is really nothing stopping you from using it yourself (that is another topic).

If we check our SDK folder in the plugins directory in the ‘diagnostics‘ folder, you will actually find the magic that is used to launch the DM.
<?xml version="1.0" ?> <RoleModule xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" namespace="Microsoft.WindowsAzure.Plugins.Diagnostics"> <Startup priority="-2"> <Task commandLine="DiagnosticsAgent.exe" executionContext="limited" taskType="background" /> <Task commandLine="DiagnosticsAgent.exe
/blockStartup" executionContext="limited" taskType="simple" /> </Startup> <ConfigurationSettings> <Setting name="ConnectionString" /> </ConfigurationSettings> </RoleModule>
.csharpcode, .csharpcode pre {font-size:small;color:black;font-family:consolas, "Courier New", courier, monospace;background-color:#ffffff;} .csharpcode pre {margin:0em;} .csharpcode .rem {color:#008000;} .csharpcode .kwrd {color:#0000ff;} .csharpcode .str {color:#006080;} .csharpcode .op {color:#0000c0;} .csharpcode .preproc {color:#cc6633;} .csharpcode .asp {background-color:#ffff00;} .csharpcode .html {color:#800000;} .csharpcode .attr {color:#ff0000;} .csharpcode .alt {background-color:#f4f4f4;width:100%;margin:0em;} .csharpcode .lnum {color:#606060;}

Here, we can see that the DM is implemented as a pair of startup tasks. Notice, it is using a task type of background (the other is blocking until it gets going). This means that the DM exists outside the code you write and should be impervious to your code crashing and taking it down as well. You can also see that the startup tasks listed here will be run with a default priority of -2. This just tries to ensure that they run before any other startup tasks. The idea is that you want the DM to start before other stuff so it can collect data for you.

You can also see in the definition the declaration of a new ConfigurationSettings with a single Setting called ‘ConnectionString’. If you are using Visual Studio when you import the Diagnostics plugin, you will see that the tooling automatically combines the namespace with the settings name and creates a new Setting called Microsoft.Windows.Plugins.Diagnostics.ConnectionString. This setting will not exist if you are building your csdef or cscfg files by hand. You must remember to include it.

Once you have the plugin actually enabled in your solution, you will need to specify a valid connection string in order for the DM to operate. Here you have two choices:

Running in emulation, it is valid to use "UseDevelopmentStorage=true" as ConnectionString.

Before deploying to cloud, you must remember to update that to a valid storage account (i.e. "DefaultEndpointsProtocol=https;AccountName=youraccount;AccountKey=nrIXB.")

Common Pitfalls

It seems simple enough, but here come the first set of common pitfalls I see:

Forgetting to set the ConnectionString to a valid storage account and deploying with ‘UseDevelopmentStorage=true’. This has become less of a factor in 1.6+ SDK tooling because you will notice the checkbox that says, "Use publish storage account as connection string when you publish to Windows Azure". However, tooling will not help you here for automated deploys or when you forget to check that box.

Using "DefaultEndpointsProtocol=http" in the connection string (note the missing ‘s’ from ‘https’). While it is technically possible to use the DM with an http connection, it is not worth the hassle. Just use https and save yourself the hassle of troubleshooting this later.

Setting an invalid connection string. Hard to believe, but I see it all the time now on AzureOps. This usually falls into two categories: deleting a storage account, and regenerating a storage key. If you delete a storage account, but forget to remove that as the ConnectionString, things won’t work (shocking, I know). Further, if you decide to regenerate the primary or secondary storage keys and you were using them, stuff won’t work here either. Seems obvious, but you won’t actually get any warning on this. Stuff won’t work and you will have to figure that out yourself. A good 3rd party provider (like AzureOps) will let you know however.

Forgetting to co-locate the diagnostics storage account with the hosted service. This one might not show itself until you see the bill. The diagnostics agent can be pretty chatty. I have seen GBs of data logged in a single minute. Forgetting to co-locate that would run you a pretty hefty bandwidth bill in addition to slowing you down.

Best Practices

Setting up the Diagnostics Manager is not terribly hard, but easy to get wrong if you are not familiar with it. There are some other subtle things you can do here that will shoot yourself in the foot however. Here are some things you can do that will make your life easier:

Always separate your diagnostics storage account from other storage accounts. This is especially important for production systems. You do not want diagnostics competing with your primary storage account for resources. There is an account wide, 5000 transactions per second limit across tables, queues, and blobs. When you use a single storage account for both, you could unintentionally throttle your production account.

If possible, use a different diagnostics storage account per hosted service. If that is not practical, at least try to separate storage accounts for production versus non-production systems. It turns out that querying diagnostics data can be difficult if there are many different systems logging to the same diagnostics tables. What I have seen many times is someone use the same diagnostics account for load testing against non-production systems as their production system. What happens is that the amount of data for the non-production system can greatly exceed the production systems. The query mechanism for then finding production data is akin to finding a needle in the haystack. It can take a very long time in some cases to query even for simple things.

Don’t’ use the Anywhere location. This applies to all storage accounts and all hosted services. This might seem obvious, but I see it all the time. It is possible to use Anywhere location with affinity groups and avoid pitfall #4, but it is not worth the hassle. Additionally, if you have a 3rd party (like AzureOps) that is monitoring your data, we cannot geo-locate a worker next to you to pull your data. We won’t know where you are located and it could mean big bandwidth bills for you.

At this point, if you have enabled the DM, and remembered to set a valid connection string, you are almost home. The last thing to do is actually get the data and avoid common pitfalls there. That is the topic for the next post.

No significant articles today.

Visual Studio LightSwitch and Entity Framework 4.1+

Jan Van der Haegen (@janvanderhaegen) described What’s new in Visual Studio LightSwitch 11 (LS VS vNext Beta) – a hacker’s view… in a 2/29/2012 post:

I need to write down my random thoughts on what just happened…

Rumors got out a couple of days ago, but in case you missed it: Visual Studio LightSwitch 11 was released in public beta a couple of hours ago!

Download it – it’s no longer a separate download, but is included in the full Visual Studio 11 consumer preview.

Find out what’s new on the official LightSwitch team blog.

There will probably be more official content later today, and over the next few weeks, but let’s dive into the “what’s new” post together while the installer is running… My blog post won’t list the new features, the official announcement covered that briefly but with just enough detail, instead, lets reflect a bit on how this will change our lives as LightSwitch newbies, experts, and hackers alike…

The LightSwitch team has been listening!

Some of us, including myself, had the amazing honor to meet Beth Massi in person, others used the Microsoft Connect site, to provide feedback to the team.

And they have been listening!

And not only have they been listening when we addressed them directly, but they have been listening when we blogged, tweeted, or even posted popular LightSwitch extensions that solve some common problems in LightSwitch v1… They have been listening! So let’s continue, more than ever, to connect to the LightSwitch team, connect to each other, and enjoy how LightSwitch product and community becomes better and better over the years.

The result, is directly visible in LightSwitch 11. The ability to define data relationships on external data, an easy way to add static text and images, new business types (web address and percentage), assigning roles on Active Directory groups to easy the user management pains, …

All of this is mainly for end-user-developers, which of course, includes us experts and hackers as well! Really, besides exploring the technological wonders of LightSwitch, we use it to write applications too, right?

Welcome LightSwitch as a first class citizen!

What’s more, today marks the availability of Visual Studio 11 Beta

This, might be one of the most influencing novelties confirmed today. LightSwitch is no longer (solely?) available as a separate download, but is included in the full-blown Visual Studio 11. LightSwitch v1 primarily targeted “end-user-developers”, or at least – that’s how it often felt, and by including it in the full VS 11 package, Microsoft is clearly signalling that they haven’t, and will never, forget about us experts & hackers.

It’ll be a lot easier to convince other professional developers to give LightSwitch a second try, to convince them that it’s not “Access version 2″ (yes, one person literally asked me that… I politely giggled at his ignorance then, but am laughing out loud now!), if it’s already installed on their machines!

The fact that there is a next version of LightSwitch, and that it’s included in the full VS 11 suite, is also a clear acknowledgment that LightSwitch has a future, and that Microsoft is working hard on that future. Who would have guessed that a billion dollar company wouldn’t release a product without a bigger plan?

In the middle of all this “HTML5 and Metro”-hyping, a lot of people also wondered if LightSwitch wasn’t “dead before it even started”, because of using SilverLight as a front-end technology. To be honest, the business doesn’t care much about front-end technology, but we, as professional developers, want to stay trendy and hip, and in a way, they gave us that with this release… (Read on…)

Getting your data out of a LightSwitch application!

LightSwitch v1, included quite some options to get your ‘legacy’ data in a LightSwitch application – design it yourself, connect to an existing database, SharePoint, custom WCF RIA services, … but apart from some custom excell exports, you could not get your data back out easily.

First, LightSwitch in VS11 has embraced OData. [...] adds first-class support for connecting your business applications to OData feeds. [...] But, we thought we’d spice things up more and turn our OData support on its head – in VS11, LightSwitch also makes is extremely easily to produce and deploy your own data as OData services. This is both true for tables you define in your project as well as external data sources you connect to. What this means is that the LightSwitch server pipeline [...] is no longer a closed black box – other apps can now leverage the simplicity and power of LightSwitch via the OData feeds it exposes.

So many ideas pop into my head…

Instead of LightSwitch connecting to other application’s data, other applications can now connect with greatest easy, to our LightSwitch application’s data. For me, this takes LightSwitch away from the “tool that can connect to enterprise solutions”, right up there on the pedestal as the “application that is part of an enterprise solution”.

If you have some legacy apps, but don’t have the need to re-write them as a LightSwitch application, you can abuse LightSwitch to create an OData service for it. Create a new LightSwitch application, connect to the database of the legacy app, don’t make any screens but instead, publish the LightSwitch OData service only. Convert-legacy-to-future? Fastest tool to accomplish the job, right here!

If you are creating an application in another technology, it’s now really easy to access the data from your LightSwitch application using an industry standard. PhP, JAVA, HTML5 (JavaScript), whatever your flavor, can interact with my LightSwitch application, without bypassing my business logic. (You could already connect to the database of the LightSwitch application, if you wanted…)

And last but not least… My personal favorite… Since the LSML contains a definition of your screens (in other words: how your data should be presented) and OData services are available to interact with the data… What’s stopping us, as a community, to create a Windows Phone, WinRT, or HTML5 framework, based on the output created with the LightSwitch built-in editor?

In case you missed my tweet last week, my younger brother picked “using HTML5 as a front end for LightSwitch” as a topic for his (college) thesis. His life just got a LOT easier…

So… Who’ll be the first to accomplish such a project? Or does anyone want to start an open source project? Who’s ready to dream with me, and make those dreams come true, thanks to the awesomeness, that is LightSwitch 11…

Now, if you will excuse me, the installer just prompted me to reboot my pc… I’ll probably be in the LightSwitch 11 zone during the next 48 hours straight, but please, leave a comment and tell me how you feel, or what your opinion is, on the beautiful gift we got today? Merry Christmas, everyone!

Beth Massi (@bethmassi) reported Visual Studio 11 Beta Released! with LightSwitch in a 2/29/2012 post:

We are extremely happy to announce the release of Microsoft® Visual Studio® 11 Beta! This release also includes the next version of LightSwitch!

Download Visual Studio 11 Beta

Please see the LightSwitch Team Blog and the LightSwitch Beta Resources page on the Developer Center for information on new features available in LightSwitch. We’ll be rolling out more in-depth content and training in the coming weeks so keep checking back. My personal favorite is the new OData support so you know I’ll be writing a lot about that in the next few days.

Read Jason Zander’s blog for information on new capabilities in Visual Studio 11 and .NET Framework 4.5. Beta comes with a “Go Live” license which means you can now start using Visual Studio LightSwitch for production projects!

We also created a new forum for Beta. Please post questions and feedback in the LightSwitch in Visual Studio 11 Beta Forum.

The Entity Framework Team posted EF4.3.1 and EF5 Beta 1 Available on NuGet on 2/29/2012:

Today we are pleased to announce the release of two new versions of the EntityFramework NuGet package.

The 4.3.1 version includes some bug fixes for the 4.3 package we released a few weeks ago. We are also making the first beta of EF 5 available as a pre-release package.

What’s in 5.0.0-beta1

This is the first preview of EF 5 and includes all the features and bug fixes in EF 4.3.1 in addition to a number of new features that are available in applications targeting .NET Framework 4.5. (You will need Visual Studio 11 Beta to work with the .NET Framework 4.5)

Enum support allows you to have enum properties in your entity classes. This new feature is available for Model, Database and Code First.

Table-Valued functions in your database can now be used with Database First.

Spatial data types can now be exposed in your model using the DbGeography and DbGeometry types. Spatial data is supported in Model, Database and Code First.

The Performance enhancements that we recently blogged about are included in EF 5 Beta 1.

Visual Studio 11 includes LocalDb rather than SQLEXPRESS. The EntityFramework NuGet package will now check which database is available when the package is installed and use a configuration file setting to set the default database server that Code First databases will be created on. If SQLEXPRESS is running, it will be used. If SQLEXPRESS is not available then LocalDb will be registered as the default instead.

The following new features are also available in the Entity Model Designer in Visual Studio 11 Beta:

Multiple-diagrams per model allows you to have several diagrams that visualize subsections of your overall model.

Shapes on the design surface can now have coloring applied.

Batch import of stored procedures allows multiple stored procedures to be added to the model during model creation.

Known Issues

We will keep this list updated with known issues for this release:

There is a known issue using the Code First Migrations commands in Visual Studio 2010. We are working on an updated package that addresses this issue. You will receive the following error when using migrations commands from EF 5 Beta 1 in Visual Studio 2010:

Could not load file or assembly ‘Microsoft.VisualStudio.Shell.10.0, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a’ or one of its dependencies. The system cannot find the file specified.

There is a known issue in MVC4 applications in Visual Studio 11 that is described in more detail here.

Getting Started

You can get EF 5 Beta 1 by installing the latest pre-release version of the EntityFramework NuGet package.

These existing walkthroughs provide a good introduction to using the Code First, Model First & Database First workflows available in Entity Framework:

Code First Walkthrough

Model & Database First Walkthrough

We have created walkthroughs for the new features in EF 5:

EF 5.0 Enums Walkthrough

EF 5.0 Table-Valued Functions Walkthrough

EF 5.0 Spatial Walkthrough

Compatibility

This version of the NuGet package is fully compatible with Visual Studio 11 Beta and can be used for applications targeting .NET 4.0 and 4.5.

The new features listed above will only be available when writing an application that targets .NET 4.5.

EF 5 will work with Visual Studio 2010 but there is currently a known issue that prevents the Code First Migrations commands from being used in Visual Studio 2010. See the Known Issues section (above) for more details.

What’s in 4.3.1?

A few weeks back we released EF 4.3, which included the new Code First Migrations feature and a few bug fixes. Today we are releasing the 4.3.1 patch release that includes some bug fixes to the 4.3 release.

This is a fully supported go-live release.

The changes in 4.3.1 compared to 4.3 include:

Templates for using DbContext with Model First and Database First have been renamed from ADO.NET DbContext Generator to EF 4.x DbContext Generator. We made this rename to easily distinguish between the EF 5.x DbContext Generator that is to be used for EF 5 applications.

Added the ability to enable Code First Migrations against an existing database. Julie Lerman blogged about using an empty initial migration to capture the initial state of the model when it is in sync with an existing database. Once this migration is generated Code First Migrations will correctly detect and scaffold changes to your model. You can now use the –IgnoreChanges to tell Code First Migrations not to scaffold any code for this initial migration (i.e. ‘Add-Migration InitialMigration –IgnoreChanges).

Visual Studio 11 includes LocalDb rather than SQLEXPRESS. The EntityFramework NuGet package will now check which database is available when the package is installed and use a configuration file setting to set the default database server that Code First databases will be created on. If SQLEXPRESS is running, it will be used. If SQLEXPRESS is not available then LocalDb will be registered as the default.

Fixed issue where TPC mapping in Code First was generating “Unknown Columnname” exceptions when accessing the database.

Fixed issue where hard coded column data types were not honored in generated databases. If you used the Column data annotation or HasColumnType Fluent API method to hard code a column data type (i.e. xml, money, etc.), this data type was not used when creating the database.

Fixed issue preventing decimal columns from being configured with the ‘Identity’ store generated pattern.

We now provide a better exception message when context/database initialization fails due to connectivity issues with the database.

Enabled support for configsource in configuration files when using Code First Migrations.

Fixed issues using Moq with DbContext – as reported in this Stack Overflow question.

Getting Started

You can get EF 4.3.1 by installing the latest version of the EntityFramework NuGet package.

Check out the EF 4.3 release post for details and walkthroughs for EF 4.3.

Compatibility

This version of the NuGet package is fully compatible with Visual Studio 2010 and Visual Studio 11 Beta. It is designed to be used when writing .NET 4.0 applications.

If writing a .NET 4.5 application we recommend using the 5.0.0-beta1 version of the package instead. You can use the package in .NET 4.5 applications but you should be aware of the following known issue:

Entity Framework 4.1 thru 4.3 included additional data annotations in the System.ComponentModel.DataAnnotations namespace in the EntityFramework assembly. In .NET 4.5 these annotations were moved to be part of the .NET Framework in the System.ComponentModel.DataAnnotations.Schema namespace of the System.ComponentModel.DataAnnotations.dll assembly. If you are using EF 4.x and targeting .NET 4.5 this results in two data annotations with the same name in different assemblies. Because the annotations in the .NET Framework are in a different namespace we were not

able to use type forwarding to avoid this conflict.

If you are not using the affected data annotations there is no impact on your code. If you are using the data annotations in a C# project you can use the extern modifier to ensure your code uses the annotations from EntityFramework.dll (http://msdn.microsoft.com/en-us/library/e59b22c5(v=VS.80).aspx). If you use the new annotations from the System.ComponentModel.DataAnnotations.dll assembly in .NET 4.5

Support

We are seeing a lot of great Entity Framework questions (and answers) from the community on Stack Overflow. As a result, our team is going to spend more time reading and answering questions posted on Stack Overflow.

We would encourage you to post questions on Stack Overflow using the entity-framework tag. We will also continue to monitor the Entity Framework forum.

EF5.0.0_2D00_beta1_5F00_License.rtf

Return to section navigation list>

Windows Azure Infrastructure and DevOps

My (@rogerjenn) Uptime Report for my Live OakLeaf Systems Azure Table Services Sample Project: February 2012 of 3/3/2012 begins as follows:

My live OakLeaf Systems Azure Table Services Sample Project demo runs two small Windows Azure Web role instances from Microsoft’s South Central US (San Antonio, TX) data center. Here’s its uptime report from Pingdom.com for February 2012:

30 minutes of the reported downtime occurred during Windows Azure Compute’s infamous “Leap Year” outage of 2/29 to 3/1/2012. Here are Pingdom’s reports for that outage:

PingdomAlert DOWN: Azure Tables (oakleaf.cloudapp.net) is down since 02/29/2012 09:58:21AM.

PingdomAlert UP: Azure Tables (oakleaf.cloudapp.net) is UP again at 02/29/2012 10:28:21AM, after 30m of downtime.

Pingdom doesn’t cover my SQL Azure Reporting Systems Preview Demo, but it appeared to be up during repeated manual tests during the outage period.

See my Microsoft’s Official Response to the Windows Azure Outage of 2/29/2012 post of 2/29/2012 for Microsoft initial explanation of the source of the outage.

Following is detailed Pingdom response time data for the month of February 2012:

This is the ninth uptime report for the two-Web role version of the sample project. Reports will continue on a monthly basis.

The post continues with a description of my live OakLeaf Systems Azure Table Services Sample Project demo.

Joseph Feldman wrote Research: Cloud ROI Calculations for InformationWeek::Reports, who published it for download on 3/2/2012:

The Kung Fu Panda Phenomenon

While there’s an air of inevitability around cloud computing, the ROI calculation has stayed in flux. A lot depends on whether we’re talking SaaS, PaaS or IaaS; some are easier to wrap your arms around than others. Because of SaaS’s very granular cost methodology, it’s actually pretty straightforward to calculate investment and who benefits, and thus peg return on investment. For example, especially for commoditized enterprise applications like email, it’s fairly trivial to figure out how much personnel, storage, servers, software and licenses cost to come up with a per-seat price for in-house email. Then you can compare apples to apples.

Not so with platform and infrastructure as a service. They certainly have a leg up on conventional networks in terms of cost granularity and transparency. After all, you know exactly what something costs when you run it for an hour. Easy, right? Except now you’ve got to compare it to in-house infrastructure, and that’s where things get complicated, because we tend not to have a good handle on those costs. If you throw private clouds in the mix–and you should, because they’re the future–comparisons get even more difficult, because now you have to compare the cost of doing things the conventional way (for argument’s sake, straight virtualization with SANs for most infrastructure needs) versus the cost of reinventing your staff and building a private cloud versus the ongoing costs of public cloud. It’s easy if you’re a startup building a greenfield IT infrastructure from scratch, but much more complicated for an established enterprise.

The 374 respondents to our latest InformationWeek Cloud ROI Survey are using more public cloud services across the board compared with the last time we ran this poll, in April 2010. But that’s not to say we have more cost clarity. (R3890312)

Survey Name InformationWeek 2012 Cloud ROI Survey

Survey Date November 2011

Region North America

Number of Respondents 374

Purpose To determine the extent of cloud computing ROI analysis in the enterprise.

Download

Table of Contents

3 Author’s Bio

4 Executive Summary

5 Research Synopsis

6 The Kung Fu Panda Phenomenon

8 Not in My Data Center

9 The Prius Infrastructure

12 Search and Destroy

14 But What About the Bottom Line?

16 Worries, Benefits and Transparency

18 Life Cycle, Life Cycle, Life Cycle

20 Reality-Based ROI

22 Appendix

26 Related Reports

About the Author

Jonathan Feldman serves as director of information technology services for a city in North Carolina. The city has won several technology innovation awards during his tenure, including the International Economic Development Council New Media Award. He has also directed professional services in the private sector, providing security and network infrastructure services to the military, healthcare, financial services and law enforcement markets. …

Write to him at jf@feldman.org.

David Linthicum (@DavidLinthicum) asserted “Hype, misuse, and misplaced aspirations have clouded the term’s definition. Here’s what it’s really about” in a deck for his Redefining cloud computing — again post of 3/2/2011 to InfoWorld’s Cloud Computing blog:

Back in August, I declared the term "cloud computing" officially meaningless because of its extensive overuse and misuse. No matter what a vendor sold, it was somehow "cloud computing." These days, when somebody wants me to define "cloud computing," I fight the urge to eject them from the conference room. It’s so widely defined, and thus so vague, that providing a crisp definition is nearly impossible.

More disturbing, there seems to be an increasing overuse of cloud computing concepts as saviors for all past IT mistakes. Pushing cloud computing as the way to solve all, or even most, computing problems reveals those who make such statements as less than credible.

So what the heck is cloud computing? The definition I use is also the National Instutute for Standards and Technology’s take on the concept and technology. It’s been repeated so many times in so many presentations over the last few years that I’m not going to say it again here. However, the NIST definition is not the end of the matter: So many design and architectural patterns are emerging around the concept of cloud computing that it’s difficult to fit everything into NIST’s definition. I’m counting well past 100.

Again, what is cloud computing?

The concept of cloud computing is about the ability for organizations to stop solving all IT problems by themselves. It’s certainly about sharing resources, such as storage and compute services, but it really should be more about sharing solutions and pushing risk out of the business. Unfortunately, the focus today is on the tactical side of sharing computing resources. I hope in a few years that it will be more about the impact to a business rather than how the technology is defined.

Perhaps the best definition is around how cloud computing, or whatever you want to call it, will redefine how we consider and use technology to make us better at doing whatever we do, not how we should change around the addition of yet another technology concept. We’ve been there, done that, and have the T-shirt.

I’ll let you know when I find just the right definition. For now, at least I know what I’m looking for.

Geva Perry (@gevaperry) posted More on Bottom-Up Adoption of Cloud Computing on 3/2/2012:

A quick follow up on my previous post on cloud computing adoption patterns. I have been guest blogging on Compuware’s CloudSeluth blog and have written a post on this topic titled Cloud and Bottom-Up Adoption. In it I reiterate some of the points I wrote about in the last post and which I presented in my CloudConnect keynote, but I added another angle, comparing the first attempt of creating a true public cloud, the Sun Grid, to the first successful attempt: Amazon Web Services.

The main difference between the two? You guessed it.

While the former (Sun) targeted the traditional IT customer, the CIO, Amazon went after developers, and that was the secret to their success.

For more, check out the full post here [or below].

Geva Perry (@gevaperry) asserted Cloud Computing Adoption is Bottom-Up in a 3/1/2012 post to the CloudSleuth blog:

One of the less mentioned cloud computing-related phenomenon is the pattern of its adoption in the market, and in particular, the enterprise.

Many might be surprised to know that Amazon’s greatest innovation with AWS wasn’t the notion of utility computing, on demand or a pay-per-use pricing model — all of that already existed. Its greatest innovation was that it targeted developers and not IT executives.

What few remember is that almost two years before Amazon launched its Elastic Compute Cloud (EC2) service in November 2006, Sun Microsystems launched the first true public cloud. The Sun Grid, which is what the utility computing service was called, had many of the key elements that were later introduced by EC2: on-demand access to compute resources, elasticity and a $1/CPU hour payment model. There was one big problem with the Sun Grid: it failed miserably.

It’s no news that when it comes to introducing technological innovation to the market, timing is everything. But there is a saying among the VCs on Sand Hill Road that despite the so-called “first mover advantage” it is actually better to be slightly late to market than slightly early. So a simple analysis of the Sun Grid failure could say they were just ahead of their time, but I think the problem was more fundamental.

One of the major differences between the Sun Grid and Amazon EC2 was the audiences each of the companies targeted. While the former pitched their IaaS to their traditional customers, such as Wall Street CIOs, the latter went after a non-traditional target market: developers.

Amazon’s cloud was appealing to developers because of its origin within the company. Contrary to popular belief, it didn’t emerge within the company to take advantage of unused computing capacity. That would have been an IT-oriented move. It was developed at the demand of the company’s software programmers who needed a more agile, flexible way to access computing resources and wanted API access.

Naturally, what worked for Amazon’s web-savvy developers would work for thousands of similar developers outside the company. But there were additional forces at play in AWS’s success.

One such force was the growth of developer influence. This began with open source software. Developers would be the ones within the enterprise to choose the winners and losers in the application stack, because they could now download for free and without licensing issues any open source software component they needed. For more on this check out Stephen O’Grady of Redmonk’s post: Meet the New Kingmakers; Same as the Old Kingmakers.

As with open source software, with cloud and AWS too, once developers started successfully using it for proof-of-concept and testing, it quickly crept into production environments.

This pattern of adoption isn’t only what drove infrastructure-as-a-service adoption with services such as EC2. It is also what helped early software-as-a-service offerings such as Salesforce.com catch on.

In his book, Behind the Cloud, Salesforce.com founder and CEO Marc Benioff writes about one of their first enterprise accounts, Sungard: “The salespeople were buying it on their own credit cards and going around their managers to purchase an account…” As he explains, Salesforce had a competitive advantage because although their enterprise customer already made a top-down decision to use Siebel’s CRM software, the rank & file — in this case the sales people — were buying Salesforce.com with their credit cards and then expensing it.

The main reasons for bottom-up adoption becoming so dominant fall into two categories: motivation and opportunity. Enterprise users now have the motivation to buy IT services that just worked for them. As consumers we are getting used to technology that is simple, always available, on-demand and just easy and fun to use. So why is it that at work we use software that is not any of these things? That’s what folks are asking themselves creating the phenomenon that’s been called “the consumerization of IT.”

But of course motivation isn’t enough. They need an opportunity. And the opportunity is here now because cloud computing and SaaS services have removed so many of the barriers that existed in the past. Employees, low- and mid-level managers can simply make the decision on their own.

My (@rogerjenn) Microsoft’s Official Response to the Windows Azure Outage of 2/29/2012 post of 2/29/2012 reads as follows:

Bill Liang, Corporate VP, Server and Cloud, added the following post to the Windows Azure blog on 2/29/2012:

I lead the engineering organization responsible for the Windows Azure service and I want to update you on the service disruption we had over the past day. First let me apologize for any inconvenience this disruption has caused our customers. Our focus over the past day has been to resolve the Windows Azure Compute service disruption. As always we communicate the status of incidents through the Windows Azure Service Dashboard and update that status on an hourly basis or as the situation changes.

Yesterday, February 28th, 2012 at 5:45 PM PST Windows Azure operations became aware of an issue impacting the compute service in a number of regions. The issue was quickly triaged and it was determined to be caused by a software bug. While final root cause analysis is in progress, this issue appears to be due to a time calculation that was incorrect for the leap year. Once we discovered the issue we immediately took steps to protect customer services that were already up and running, and began creating a fix for the issue. The fix was successfully deployed to most of the Windows Azure sub-regions and we restored Windows Azure service availability to the majority of our customers and services by 2:57AM PST, Feb 29th.

However, some sub-regions and customers are still experiencing issues and as a result of these issues they may be experiencing a loss of application functionality. We are actively working to address these remaining issues. Customers should refer to the Windows Azure Service Dashboard for latest status. Windows Azure Storage was not impacted by this issue.

We will post an update on this situation, including details on the root cause analysis at the end of this incident. However, our current priority is to restore functionality for all of our customers, sub-regions and services.

We sincerely apologize for any inconvenience this has caused.

According to Pingdom, my OakLeaf Systems Azure Table Services Sample Project (Tools v1.4 with Azure Storage Analytics) was down for five minutes or less. The next OakLeaf Uptime Report, expected to post on 3/3/2012, will provide confirmation of downltime during the period in question. My SQL Azure Reporting Systems Preview Demo service indicated no problems during intermittent tests on 2/29/2012.

Simon Munro (@simonmunro) described Feature Shaping in a 2/27/2012 post:

One of the key concepts in scalability is the ability to allow for service degradation when an application is under load. But service degradation can be difficult to explain (an relate back to the term) and ‘degrade’ has negative connotations.

The networking people overcame the bad press of degradation by calling it ‘traffic shaping’ or ‘packet shaping’. Traffic shaping, as we see it on the edge of the network on our home broadband connections, allows some data packets to be of a lower priority (such online gaming) than others (such as web browsing). The idea is that a saturated network can handle the load by changing the profile or shape of priority traffic. Key to traffic shaping is that most users don’t notice that it is happening.

So along a similar vein I am starting to talk about feature shaping which is the ability for an application, when under load to shape the profile of features that get priority, or to shape the result to be one that is less costly (in terms of resources) to produce. This is best explained by examples.

A popular post on High Scalability talked about how Farmville degraded services when under load by dropping some of the in game features that required a lot of back end processing — shaping the richness of in-game functionality.

Email confirmations can be delayed to reduce load. The deferred load can either by the generation of the email itself, or the result of sending the email.

Encoding of videos on Facebook is not immediate and is shaped by the capacity that is available for encoding. During peak usage, the feature will take longer.

A different search index that produces less accurate results, but for a lower cost, may be used during heavy load — shaping the search result.

Real-time analytics for personalised in-page advertising can be switched off when under load — shaping the adverts to those that are more general.

So my quick definition of feature shaping is

Feature shaping allows some parts of an application degrade their normal performance or accuracy service levels in response to load.

Feature shaping is not fault tolerance — it is not a mechanism to cope when all hell breaks loose.

Feature shaping is for exceptional behaviour and features should not be shaped under normal conditions

Shaped features will be generally unnoticeable to most users. The application seems to behave as expected.

Feature shaping can be automated or manual.

Feature shaping can be applied differently to different sets of users at the same time (e.g. registered users don’t get features shaped).

So, does the terminology of feature shaping make sense to you?

Joab Jackson (@Joab_Jackson) asserted “The next version of Visual Studio will include a number of new tools to bring together developers and administrators” in a deck for his Microsoft Visual Studio 11 embraces DevOps of 2/23/2012 for ComputerWorld (missed when published):

Sensing the growing interest in DevOps, Microsoft will incorporate a number of new tools in its next edition of Visual Studio that will allow developers to work more closely with operations personnel.

Microsoft revealed these features in a Thursday webcast introducing the next version of the company’s IDE (Integrated Developer Environment), Visual Studio 2011. The company will release a beta of Visual Studio 11 on Feb. 29, along with a beta of version 4.5 of the .Net framework.

"If you are working in a team environment, we will provide you with a set of tools to work in a highly productive and agile way," said Soma Somasegar, Microsoft corporate vice president of the development division, in an interview following the presentation.

Historically, developers have had problems getting "actionable feedback" from operations staff about the programs that the developers created, said Jason Zander, Microsoft corporate vice president for Visual Studio, during the presentation. DevOps aims to solve this communication problem. It is an organizational philosophy and a set of customs for getting the two groups to work more closely together, so software can be rolled out more quickly and with fewer bugs.

An arbitrary wall "exists between the development community and the operations community in any enterprise," Somasegar said, noting that this divide has grown over the last 30 years.

"The development team takes the requirements and builds the application. Once they are finished, they throw it over the wall for the operations community to deploy. That’s fine if everything goes well, but if there is a problem [operations staff] is stuck," Somasegar said.

The developers often have to re-create the problem on their own, often using only vague information from the operations staff. Very few tools exist that allow operators to communicate issues to the developers in a way that the developers find useful. Microsoft has developed a number of new features and extensions for Visual Studio 2011 that will help the two parties work more closely together, Somasegar said.

One feature is a bridge to Microsoft System Center 2012, a management tool used by operators or system administrators. When a program crashes, System Center’s Operation Manager collects a set of diagnostic information, such as a stack trace, that can be used by the developer to pinpoint the bug. With a click of a mouse from within System Center, the administrator can pipe that information directly over to the developer.

"That allows the operations person to work in the tool they know, which is System Center, and allows the developer to work in the tool they know, which is Visual Studio. The ops person doesn’t have to figure out what a call stack is or how the software was written," Zander said. …

Read more: 2, Next page

Windows Azure Platform Appliance (WAPA), Hyper-V and Private/Hybrid Clouds

Kevin Remde (@KevinRemde) completed his series with Screencast: System Center 2012 Unified Installer (Part 3 of 3) on 3/2/2012:

Whoo boy! Finally! Part 3 of 3! The story so far…

In Part 1 we showed you how to download all that you need, and put it in its proper place.

In Part 2 we setup and pre-configured the destination servers properly.

And today in our final chapter, we actually run the Unified Installer. Oh joy!

NOTE: It is viewed best in full-screen, and 1280×768 if you can.

So.. that wasn’t so painful, was it?

“Actually, yes, it was. But I’m going to give it a try anyway. Thanks, Kevin!”

You’re quite welcome. Tell your friends.

—

Did you find this useful? Are you doing this install along with me? Let us know in the comments. Share your experiences here with the rest of the massive community that is “the people who read my blog”.

Kevin Remde (@KevinRemde) continued his series with Screencast: System Center 2012 Unified Installer (Part 2 of 3) on 3/1/2012:

Continuing on where we left off from our part 1 screencast yesterday, when we discussed the downloads and the prerequisites and what you needed in place for the installation; today in part 2 we discuss how to configure your servers.

“Is it really all that complicated, Kevin?”

Actually, yes. If you read the User Guide, you’ll see that there are several things you need to do to prepare the servers prior to using the Unified Installer to do the deployment to them. So in today’s screencast, I walk you through that process.

NOTE: It is viewed best in full-screen, and 1280×768 if you can.

Make sure you come back tomorrow for part 3, where we actually run the Unified Installer, and deploy the entire set of components in System Center into my test lab!

—

Did you find this useful? Are you doing this install along with me? Let us know in the comments. And I’ll see you back here tomorrow for Part 3.

Kevin Remde (@KevinRemde) started a series with Screencast: System Center 2012 Unified Installer (Part 1 of 3) on 2/29/2012:

Like many of you, I’ve been playing with the Release Candidate of System Center 2012. And also like many of you, I have heard that there is something called the “Unified Installer”, that supposedly will allow a person to install many – or even all – of the components of System Center 2012 on to multiple servers.

“Really? That’s cool.”

Yeah, that’s what I thought, too. But after looking at it for a bit, I came to realize quickly that it wasn’t really as straightforward as just launching a Setup.exe and clicking Next –> Next –> Next –> Install –> Finish. In fact, it requires no small amount of thought and pre-work to get things downloaded, extracted, and configured properly prior to ever launching the Unified Installer.

In other words: It was a set of screencasts just screaming to be created.

So that’s what I’ve done. Here is Part 1, where I describe how to download and extract the components, plus download and prepare the prerequisites that are, um.. pre-required.

NOTE: It is viewed best in full-screen, and 1280×768 if you can.

As promised, here are the links mentioned in the screencast:

Download the System Center 2012 Release Candidate: http://aka.ms/PvtCld

Check out the System Center 2012 Unified Installer Setup Guide: http ://technet.microsoft.com/en-us/library/hh751266.aspx

—

Did you find this useful? Are you going to do this install along with me? Let us know in the comments. And I’ll see you back here tomorrow for Part 2.

Cloud Security and Governance

Richard Santalesa reported NIST Releases Public Draft SP800-53 Addressing Cybersecurity Threats & Privacy Controls in a 2/29/2012 post to the Info Law Group blog:

Yesterday the National Institute of Standards and Technology (NIST) released the 4th iteration of what will ultimately be a mainstay document for federal agencies required to comply with provisions of the Federal Information Security Management Act (FISMA) and FIPS 200. As a result it should have a significant affect on federal cloud security practices that will ultimately also effect commercial non-governmental cloud usage.

Weighing in at 375 pages, NIST’s Special Publication 800-53, Rev. 4, entitled Security and Privacy Controls for Federal Information Systems and Organizations, is the first “public draft” of SP800-53. Previous iterations of parts of SP800-53 were released essentially piecemeal (i.e. Appendix J, Privacy Control Catalog, was earlier distributed separately, etc.). Given the breadth and scope of SP800-53 follow-up posts will examine specific notable sections of this important NIST SP. In addition, the public comment period for SP 800-53 runs until April 6, 2012. Comments may be sent via email to sec-cert@nist.gov.

This latest public draft includes major changes that include…

…according to NIST:

New security controls and control enhancements;

Clarification of security control requirements and specification language;

New tailoring guidance including the introduction of overlays;

Additional supplemental guidance for security controls and enhancements;

New privacy controls and implementation guidance;

Updated security control baselines;

New summary tables for security controls to facilitate ease-of-use; and

Revised minimum assurance requirements and designated assurance controls.

NIST notes that "[m]any of the changes were driven by particular cyber security issues and challenges requiring greater attention including, for example, insider threat, mobile and cloud computing, application security, firmware integrity, supply chain risk, and the advanced persistent threat (APT)."

Interestingly, despite the cloud-heavy focus of many recent NIST SP’s and reports, the release stresses that "in most instances, with the exception of the new privacy appendix, the new controls and enhancements are not labeled specifically as ‘cloud’ or ‘mobile computing’ controls or placed in one section of the catalog." In following posts I’ll explore the ramifications of this orientation and examine why NIST’s approach makes sense in light of the current infosec and threat landscape. We’ll also dig through the expected additional markup versions of Appendices D, F and G following the comment period and Appendices E and J, containing security and privacy controls. Stay tuned.

To discuss the latest SP800-53 public draft or expected implications of the recommended controls on your entity’s security and data infrastructure please feel free to contact me or any of the InfoLawGroup team of attorneys.

Integracon (@Integracon) reported Failure to Meet Requirements Hinders LA Cloud Implementation in a 2/29/2012 post:

After two years of trying and thousands of dollars investment, the City of Los Angeles withdrew their plans to migrate the police department to Google’s hosted email and office application.[1] Most of the city departments have already migrated, but the 13,000 member police force is forced to remain on Novell GroupWise because the cloud solution could not meet FBI requirements.

The city contracted with CSC to facilitate the systems integration on 2009, but both CSC and Google failed to meet commitments to implement security requirements again and again. This delay brought the migration of the LAPD to a standstill. Google and CSC both lost a percentage of service fees, and Google must pay for the Novell Groupwise service. The city is so frustrated that they are considering suing both organizations for failure to meet compliance requirements after assurances that the requirements would be met.

Google and CSC claim that the FBI requirements are not compatible with cloud implementations. They also claim that they did not realize the FBI requirements when they committed to the project. But Jeff Gould, CEO of IT consulting firm Peerstone Research, suggests that their excuses are disingenuous. Google and CSC should have known requirements clearly detailed in the CJIS policy document when they signed the contract.

This fiasco could have been prevented if the cloud providers had done the proper research to understand the exact requirements for the LAPD in advance. This raises a fundamental issue for a cloud services. When a municipality contracts with a cloud vender, they should make certain the provider fully understands the various requirements in advance. In fact, this is a lesson that any business should keep in mind when sourcing cloud services. Make sure that both the service that facilities your systems integration as well as cloud provider understand the full extent of requirements in advance.

Integracon is a leader in cloud implementations because our engineers focus on details and delivery. Proper preparation and planning is essential for successful implementation, and meeting all requirements is not optional but essential. To learn more about our depth of experience in cloud service implementation, contact Integracon today. We have the deep experience that is essential and can over a list of satisfied customers who continue to rely our service and support.

[1] Jaikumar Vijayan. “Los Angeles police drop plans to move to Google Apps.” December 27, 2011 <http://news.techworld.com/data-centre/3326744/los-angeles-police-drop-plans-to-move-to-google-apps/?cmpid=TD1N10&no1x1&olo=daily%20newsletter>.

Cloud Computing Events

Bruno Terkaly (@brunoterkaly) described Azure/Cloud, HTML5, Windows Phone Free Training Events Coming To Your City in a 2/29/2012 post:

Developer Camps (DevCamps for short) are free, fun, no-fluff events for developers, by developers. You learn from experts in a low-key, interactive way and then get hands-on time to apply what you’ve learned. Where else can you get so much good stuff all in a day?

Register today at the following links:

CityDate

Cloud RegLink

HTML5 Reglink

Windows Phone RegLink

Los Angeles – March 30

Click Here

Click Here

Click Here

Irvine – April 20

Click Here

Click Here

Click Here

Redmond – April 27

Click Here

Click Here

Click Here

Denver – May 18

Click Here

Click Here

Click Here

Phoenix – May 25

Click Here

Click Here

Click Here

CityDate	Cloud RegLink	HTML5 Reglink	Windows Phone RegLink
Los Angeles – March 30	Click Here	Click Here	Click Here
Irvine – April 20	Click Here	Click Here	Click Here
Redmond – April 27	Click Here	Click Here	Click Here
Denver – May 18	Click Here	Click Here	Click Here
Phoenix – May 25	Click Here	Click Here	Click Here

Come prepared

We want you to hit the ground running.

Please come prepared.

You don’t want to waste your time at an Azure DevCamp downloading and installing files. Setting up can take quite some time and will interfere with productivity. I will also provide some posts for those that could not attend.

This post includes:

Azure DevCamp – Hardware: Minimum

The following hardware is needed to install the needed software:

** Important Note **

Running Macintosh computers is not recommended. Storage and Compute emulators do not always work properly.

Azure DevCamp – Software: Supported Operating Systems

Don’t expect Windows XP to work. Here are the supported OS’s.

Visual Studio and SQL Server Express

Various versions of Visual Studio will work. Please be sure you have one of the following:

Free Trial Account / Azure SDK

Free Trial

Labs and PowerPoints

You are about to install Windows Azure Camps Kit.

The Windows Azure Camps Training Kit includes the hands-on labs and presentations that are used for the Windows Azure Camp events.

Installer for Azure Web Camps: http://www.contentinstaller.net/Install/ContentGroup/WAPCamps

Video – to prepare you are ready

The main thing to remember is that most of the labs can be done with emulators. This means you don’t necessarily need to deploy your app to the cloud (Microsoft Data Center). You can run most of these labs all on your local computer. This is important as there is not often enough network bandwidth at large events.

Currently, there is no audio. But it should be easy to follow.

Video to verify setup: http://brunoblogfiles.com/videos/PrepareForAzureDevCamp.wmv

Conclusion

Please verify you can run the sample demonstrated in the video above. This will give you a huge head start once you attend the boot camp.

Other Cloud Computing Platforms and Services

James Downey (@james_downey) asked Eventual Consistency: How Eventual? How Consistent? in a 3/2/2012 post:

In my last post on Riak, I discussed how an application developer could use the values of N (number of replication nodes), R (read quorum), and W (write quorum) to fine tune the availability, latency, and consistency trade-offs of a distributed database.

The idea of fine tuning consistency with the values of N, W, and R was first made popular in 2007 when Amazon engineers published an article explaining the design principles of Dynamo, a highly available and scalable distributed database used internally at Amazon. The principles popularized by the article were incorporated into three popular open-source NoSQL databases: Riak, Cassandra, and Project Voldemort.

Recall that W+R>N assures consistency. Such a system of overlapping nodes is referred to as a strict quorum, in that every read is guaranteed to return the latest write. Many developers, however, choose to configure W+R<N for the sake of greater availability or lower latency. Such a system, known as a weak or partial quorum, does not guarantee consistency. But these systems do use various mechanisms to guarantee eventual consistency, meaning that in the absence of additional writes to a key, the key’s value will eventually become consistent across all N nodes. (For a good summary of eventual consistency, see the post on the topic by Werner Vogel, Amazon’s CTO.)

But eventual consistency is an odd guarantee. How long might eventual take? What is the probability of an inconsistent read? Perhaps because the answers to these questions depend on many factors, NoSQL providers do not provide specifics.

Now a group of computer science graduate students at Berkeley are in pursuit of answers. Using data samples from Internet-scale companies and statistical methods (most notably Monte Carlo analysis), the team has put together mathematical models and a simulation tool to determine both average and upper bound answers to these key questions regarding eventual consistency. They refer to the model as Probabilistically Bounded Staleness (PBS). The model factors in the values of N, W, and R as well as sample latencies for read write operations. As of yet, the model does not account for nodes entering or leaving the cluster.

PBS is a brilliant application of statistics to a computer science problem. I enjoyed hearing Peter Bailis, a member of the research team, describe the research at a Basho-sponsored meet-up this Tuesday. To learn the details of PBS, visit the team’s web site. If your business depends on the tradeoffs of eventual consistency, this research is of tremendous importance.

Rod Mercado (@RodMatDell) offered Dell’s Virtual Network Architecture: Our Point of View in a 2/27/2012 post:

It’s time to re-think networking.

Today, companies are faced with an explosion of data growth. Applications have been migrated from laptops to “the cloud,” while an increasingly mobile workforce demands ubiquitous, secure access to all resources from any device. The proliferation of cloud, virtualization, mobility & traffic growth requires that businesses have to become more agile and flexible just to keep up.

This requires a new approach. Dell’s Virtual Network Architecture (VNA) is built for today’s dynamic IT environments and positions the network as an enabler for business; one that intelligently connects you to the workloads, applications and data you need to effectively grow your business.

An open framework for efficient IT infrastructure and workload intelligence, VNA allows customers to achieve more, get real results faster and maximize efficiency through…

Fabrics that fit so customers can scale up and out performance for all types of data centers and campus environments. Customers have the ability to use low power, small form factor systems with distributed core (video) or scale to larger chassis systems as their performance and density needs require.

Virtualized services that are the foundation of Dell FTOS and the Open Automation framework which provides for server-like plug and play networking.

Simplifying the complex through tighter solution integration by using end to end 10Gb Ethernet solutions with Dell servers (@Dell_Servers), storage (@Dell_Storage), and networking that enables customers to realize the full power of 10Gb Ethernet in their IT environments.

Mobilizing users by enabling networking for the campus, remote facilities, corporate issued devices or a personal device, like smartphones and tablets. (Dell Wireless Networking Video)

For Dell Networking, we’re happy to announce that we have fully integrated Force10 Networks and are accelerating our product, solutions and sales efforts. In the six months since acquiring Force10 Networks, Dell has dramatically extended the availability of Force10 products like the Dell Force10 Z9000 (video) worldwide. The Dell Force10 products, technology and engineering have and will continue to play a critical role in Dell VNA.

Throughout 2012, Dell Networking will continue to exhibit technology leadership through a number of product launches from Force10, PowerConnect and M-Series Blade IO. We will feature these at upcoming events including Interop and Dell Storage Forum. We’d like to invite you to come visit us at these events.

For detailed information on Dell Networking and VNA, visit Dell TechCenter where you can find these and other resources…

Dell Virtual Network Architecture Slideshare

Video of Arpit Joshipura session on Dell Virtual Network Architecture at Cloud Connect

Brad Hedlund blog "Construct a Leaf Spine design with 40G or 10G? An observation in scaling the fabric"

Networking Computing: This is the year of 10Gb

Continue the conversation on twitter by following @DellNetworking and hashtags #DoMoreIT and #DellVNA.