Cloud Computing, Database, Technology

SQL On-Premise to Azure SQL Migration

In this era of Cloud computing, developers are often tasked with migrating databases to Azure Cloud.  This blog highlights some of the steps required for a successful migration. In a recent project, we migrated  12 Sql Server databases with about 1TB data from on-premise to Azure Cloud.

sql azure migrate

Import Connect to the On-Premise SQL Server database and import database schema to a Database project in Visual Studio.
Migration Wizard This is SSDT compatibility checker. Set the target database to Microsoft Azure SQL Database V12. Build the project to start compatibly check.
Fix Errors Fix all errors and warning. Make sure to check for all warning and resolve them before publishing the database.
Publish Schema Create a new local database and publish the schema to the local database. This will create an Azure compatible database.
Validate Database Validate database, run some quick check to make sure all objects are migrated to the local database.
Compare Database Using Visual Studio SSDT data compare, do a compare between the source database and local database. This will create a list all source database with data and target (local database) with no data.
Copy Data Copy the data from source database to the local database.
Validate Database Run the Visual Studio SSDT data compare again. This time the source and target (local database) should match. You can also smoke test your application against this local database to validate the database.
Backup Database Create a storage account on Windows Azure and backup the local database to Windows Azure Storage.
Restore Database Create a database on Windows Azure SQL and restore the backup from the Azure storage.
Validate Database Run the Visual Studio SSDT schema/data compare between the local database and Azure SQL database. This time the source and target should match pretty close. It might not match 100% percent, because of the compatibility issues. Smoke test your application against this Azure SQL Database to validate the database.
Add to Source Control Add Database project to Source Control (Visual Studio online or any other on-premise /cloud based source control)
Advertisements
Architecture, Cloud Computing, Technology

Microservices

A successful application grows more complex over time, and that complexity creates challenges in development. Microservices provides a strategy for decomposing a large project into smaller, more manageable pieces. Microservices encapsulate many of the best practices of some of the industry’s biggest software development organisations and are at the core of the development practice of many organisations today, not least among them Netflix and Amazon.

While a monolithic app is One Big Program with many responsibilities, microservice-based apps are composed of several small programs, each with a single responsibility. When new technologies become available that makes more sense for a particular service, it’s feasible to rewrite just that service.

Microservice Characteristics

Microservices community and Martin Fowler describe the common characteristics of microservice as follows:

  • Componentization via services
  • Organized around business capabilities
  • Products not Projects
  • Smart End Point and dumb pipelines
  • Decentralized Governance
  • Decentralized Data Management
  • Infrastructure Automation
  • Design for Failure
  • Evolutionary Design

 

Componentization via services

The basic idea behind microservice is to split monolithic services to several small manageable components. A component is something that can be independently replaceable and independently upgradable. For example, replace or upgrade breaks on a car without replacing the entire car for a new break.

component_1

Components can be a library or a Service. A library is a piece of code, which is custom or third party, that we reference in our project. For example, an XML processing library we use in our project,  we should be able to upgrade this library without having many problems as long as we have all the dependencies figured out.

Service is a different kind of component, run on its own process. If you have an application with components running.NET 2.0 framework, and if you want to use the component in .NET 4.5  framework, you are out of luck. But with services, it is possible to move the.NET 4.5 component to a service and access via its endpoints.

component_2

Orgaserviceound business capabilities

A lot of the development organisation are organized around technologies. There is a team for UI, a team for Database etc and team for Business Logic etc.  The key thing about the microservices world is to organize around business capabilities rather than technologies. Each team should have some elements that directly impact end user experience.

organization_1

 

Smart End-Point and dumb Pipeline

When you talk about SOA (Service oriented Architecture), we often talk about getting a power piece of middleware which does a lot of the heavy lifting or processing logic automatically. This called the ESB, enterprise Service Bus.

endpoint_1

 

Microservice community rejects this idea, and they suggest moving all the processing logic to the end point themselves. What we want is connect everything together and process message effectively but it should be up to the end point to how and when to process the messages.All we want is a dumb set of pipes to route the messages.

endpoint_2

Decentralized Data Management

Microservices strongly argues against the way the technology landscape is governed today, particularly the way data is managed. When you look at the monolithic application, all of the data is sitting in one big data store, often time in a company standard relational databases. Even in SOA architecture, multiple services pulling data in and out of the same database.

decentralize_1

 

Microservices way of managing data is to have each service responsible for its own data. A service should never talk to another services data store directly, but only through its APIs. This approach has some advantages; first, it removes the mess of integrating through the database, second, the service developer is free to choose a datastore best fit for the service.

decentralize_2

 

Infrastructure Automation

To make all these works, we need infrastructure automation. Things like continuous delivery and blue/green deployment where you can deploy with zero downtime are mandatory for microservice to be successful. With Microservices, we are talking about one application with a dozen or so services, so we need an automated way to doing things. You should be able to spin new instances of VM relative quickly.

Also in order to make Microservices successful, we need effective monitoring capabilities. So when things go wrong, we can quickly spot them and fix them.

 Design for failure

Traditional approaches often equate reliability with preventing failure.  When you have several components/services deployed on multiple nodes, chances are some of them are going to fail. So we need to design the services with failure in mind. That’s why monitoring should be an integral part of the microservice architecture.

 

Conclusion

A microservice application separates functionality into separate smaller services and scales out by deploying each service independently. This preserve modularity, allows us to choose multiple platform/technologies and work well with the cloud. In order for this to work, we need to adopt a better DevOps culture, better infrastructure and monitoring capabilities. But overall, microservies provide a better development experience and agility of the technology landscape.

 

 

BigData, Technology

4 Vs of Big Data

    • Volume: We currently see the exponential growth in the data storage as the data is now more than text data. There are videos, music and large images on our social media channels. It is very common to have Terabytes and Petabytes of the storage system for enterprises.
    • Velocity: Velocity describes the frequency at which data is generated, captured and shared. Recent developments mean that not only consumers but also businesses generate more data in much shorter cycles.
    • Variety:
      Today’s data no longer fits into neat, easy to consume structures. New types include content, geo-spatial, hardware data points, location based, log data, machine data, metrics, mobile, physical data points, process, RFID etc.
    • Veracity: This refers to the uncertainty of the data available. Veracity isn’t just about data quality, it’s about data understandability. Veracity has an impact on the confidence data.

Big Data

Technology, Web

Responsive Web

Over the last few years, website traffic from mobiles and tablets has increased dramatically as smartphones become more prevalent. Unfortunately, many websites were designed to be viewed on large screens and, as a result, the experience on phones is pretty bad – having to zoom in and out of sites to read text and be able to navigate around. Responsive web design is an approach which suggests that, instead of users having to adapt to be able to read sites on phones, the website itself will change so that an optimized version is always shown no matter which device is used.

Responsive Web Design is a Web design approach aimed at crafting sites to provide an optimal viewing experience—easy reading and navigation with a minimum of resizing, panning, and scrolling—across a wide range of devices (from mobile phones to desktop computer monitors) – Wikipedia

While changing the design of a site is a key part of responsive design, there’s a lot more to it than simply changing the layout.

responsive_design

Core Experience
With smaller sizes, not everything can fit onto the screen. Similarly, slower connections shouldn’t be made to download extra ‘nice-to-have’ features such as web fonts or large images – you should consider what the core features of your site are (i.e. the content!) and make sure this is loaded first so that users aren’t negatively affected by the size of their device or speed of connection.

Screen Size is not the defining factor
Take into account the speed of the connection before serving large images or web fonts – just because a device has a large screen, it doesn’t mean that it will have a fast internet connection. Also, consider whether browsers can handle certain features; if browsers can handle things natively, use them! Forcing users to download polyfills to fix issues in browsers they aren’t even using is bad and slows the site down for everyone.

Multiple Experience
It’s OK for the same site to look different on different browsers. You can’t make a site look identical on desktop and on mobile so focus on creating the optimum experience for each one rather than having one all-round average version. Since sites don’t look the same on mobile and desktop; it’s an easier conversation to have with the client to explain that the site doesn’t have to be pixel perfect in every browser. As above, it’s possible to make the site look exactly the same in ancient browsers but this comes at the cost of making the site larger and therefore slower.

Mobile First Responsive Web

Mobile-First Responsive Web Design is a combination of philosophies/strategies, and ultimately boils down to a broader application of good old web best practices. Mobile Web is a philosophy that highlights the need to prioritize the mobile context when creating user experiences. Starting with mobile first:

  1. 1. Allows websites to reach more people (77% of the world’s population has a mobile device, 85% of phones sold in 2011 equipped with browser)
  2. 2. Forces designers to focus on core content and functionality (What do you do when you lose 80% of your screen real estate?)
  3. 3. Let the designers innovate and take advantage of new technologies (geolocation, camera, touch events and more)

Creating a responsive web design utilizes:

  1. 1. Fluid grids that ebb and flow with a devices’ screen size
  2. 2. Flexible images and media that keep content intact on any resolution
  3. 3. Media queries allowing designs to adapt by establishing dimension breakpoints

Ultimately, mobile-first responsive web design comes down to using Progressive Enhancement as a foundation for web strategy and design. Designing with progressive enhancement involves smartly adding layers of enhancements to a strong foundation in order to deliver an accessible (and hopefully optimized) experience to all.

Scaling up from the mobile context versus scaling down from the desktop context ensures that your message, content and functionality remain intact as the screen real estate and connection speed increase. Scaling down runs the risk of your core message and functionality getting lost by the time you squish it all the way down to the mobile context.

The picture below summarize the Responsive Web in one slide:

ResponsiveWeb

Cloud Computing, Technology, Uncategorized

Public, Private and Hybrid Cloud

What is Cloud Computing?

Cloud computing, often referred to as simply “the cloud,” is the delivery of on-demand computing resources—everything from applications to data centers—over the Internet and on a pay-for-use basis. Cloud computing comes into focus only when you think about what IT always needs: a way to increase capacity or add capabilities on the fly without investing in new infrastructure, training new personnel, or licensing new software. Cloud computing encompasses any subscription-based or pay-per-use service that, in real time over the Internet, extends IT’s existing capabilities. Running applications on machines in an Internet-accessible data center can bring plenty of advantages.

 

No matter which provider you choose, you’ll find that almost every cloud has these core characteristics:

  • It is Virtual, Unlimited Compute power, Unlimited Storage
  • It is Flexible and Scalable
  • It is Open
  • It can be Secure
  • It can be affordable (pay per use)

     


Private Cloud

A private cloud is owned and operated by a single company that controls the way virtualized resources and automated services are customized and used by various lines of business and constituent groups. Private clouds exist to take advantage of many of cloud’s efficiencies, while providing more control of resources and steering clear of multi-tenancy.

Key characteristics of private clouds include:

  • A self-service interface that controls common services, allowing IT staff to quickly provision, allocate and deliver on-demand IT resources
  • Highly automated management of resource pools for everything from compute capability to storage, analytics, and middleware
  • Sophisticated security and governance designed for a company’s specific requirements

 

Public Cloud

Public clouds are owned and operated by companies that use them to offer rapid access to affordable computing resources to other organizations or individuals. With public cloud services, users don’t need to purchase hardware, software or supporting infrastructure, which is owned and managed by providers.

Many businesses are using software-as-a-service (SaaS) delivered from the public cloud for applications ranging from customer resource management (CRM)—like Salesforce.com—to transaction management and data analytics.

Beyond SaaS applications, companies are using other public cloud services, including infrastructure as a service (IaaS) to add more storage or compute services on a moment’s notice, and platform as a service (PaaS) for cloud-based application development and deployment environments.

 

Some Public Cloud Service Providers are:

Hybrid Cloud

A hybrid cloud uses a private cloud foundation combined with the strategic use of public cloud services. The reality is a private cloud can’t exist in isolation from the rest of a company’s IT resources and the public cloud. Most companies with private clouds will evolve to manage workloads across data centers, private clouds and public clouds—thereby creating hybrid clouds.

Evolving to a hybrid cloud strategy will allow companies to keep critical line of business applications and sensitive data in a traditional data center environment or private cloud, while also taking advantage of public cloud resources like SaaS for the latest applications and IaaS for elastic, economical virtual resources to scale. The ability to efficiently and securely manage the combination of public and private cloud services as a single unified computing environment is the key capability to fully taking advantage of the cloud.

Technology, Uncategorized

Garbage Collector Enhancements in .NET 4 and .NET 4.5

Microsoft has made several enhancements to the .NET Garbage Collection in CLR 4.0 and CRL 4.5 releases.

Concurrent GC Mode

Before version 4, the .NET Framework provided a concurrent GC mode which was available only for workstation GC. In the .NET Framework 4, an improved version called the background workstation garbage collection, which reduced latency but only benefited client apps. .NET Framework 4.5, introduced a  background server garbage collection, which is typically used for server apps. As a result, all apps now have background GC available to them, regardless of which GC they use. The new background server GC in the .NET Framework 4.5 offloads much of the GC work associated with a full blocking collection to dedicated background GC threads that can run concurrently with user code, resulting in much shorter (less noticeable) pauses. One customer reported a 70% decrease in GC pause times.

Large Object Heap (LOH)

If an app triggers many GCs and spends a lot of time in GC, throughput drops noticeably. One of the common cases is imbalanced GC heaps. GC has a “heap balancing” mechanism to help these apps. The CLR manages two different heaps for allocation, the small object heap (SOH) and the large object heap (LOH). Any allocation greater than or equal to 85,000 bytes goes on the LOH. LOH is not compacted unlike the SOH and collected only during Gen 2 collection.  In the .NET Framework 4 and earlier versions, the small object heap (SOH) was balanced, but the large object heap (LOH) was not.  In the .NET Framework 4.5, the server GC allocator balances the allocations across the heaps when it finds that GC heaps, including both SOH and LOH, are not balanced. In apps that begin to develop imbalanced heaps, this mechanism eliminates unnecessary GCs. This reduces the total time spent in GC and improves the app throughput.

Application Scaling

In Framework 4.5, the GC has been updated to perform optimally with > 64 processor machine architecture and processors with NUMA (non-uniform memory access) model. Starting with Windows 7 and Windows Server 2008 R2, Windows supports more than 64 processors on a single computer. For machines that have very large numbers of processors (more than 64), the operating system splits the processors into multiple processor groups. In the .NET Framework 4.5, the GC can view CPUs across processor groups and takes all the cores into account when creating and balancing heaps. You’ll need to enable the <GCCpuGroup> element in your app’s configuration file to turn this behavior on. GC automatically detects and enables NUMA support. You can also use the processor group feature along with the NUMA feature enabled.

Latency Mode

In the .NET Framework 4.5,    SustainedLowLatency mode is available for both workstation and server GC. To turn it on, set the GCSettings.LatencyMode property to GCLatencyMode.SustainedLowLatency. The .NET Framework 4 includes a LowLatency mode for workstation GC; however, this setting is only intended to be used for short periods of time, whereas SustainedLowLatency mode is intended to be used for much longer.

Large datasets (uses objects > 2GB)

By default, the .NET Framework does not support arrays that are greater than 2 GB, but in the .NET Framework 4.5, you can use the <gcAllowVeryLargeObjects> element in your application configuration file to enable arrays that are greater than this size.

Background GC

The background workstation GC was introduced in the .NET Framework 4. Both workstation and server background GC share some characteristics when a background GC is in progress: They apply only to generation 2 GCs, they don’t compact the heap. Generation 0 and 1 GCs do not happen concurrently with background GC, but they suspend all user and background GC threads. Background GC is on by default for both workstation and server GC. If you’d like to turn off background collections and revert to the old GC behavior, you can do so by setting the <gcConcurrent> element in your application configuration file to false. Also a new overload is added for the GC.Collect method to allow requests for background GCs. You can set the blocking parameter to false to request a background GC. The garbage collector will decide whether to do a background GC or a blocking GC.

void Collect(int generation, GCCollectionMode mode, bool blocking)

 

For more info  refer:

Garbage Collection:  http://msdn.microsoft.com/en-us/library/ee787088%28v=VS.100%29.aspx

Runtime Configurations: http://msdn.microsoft.com/en-us/library/6bs4szyc.aspx

Technology

Tips for MVC Performance

Run in Release mode

You should always make sure that your application is compiled in Release mode and that your web.config file is configured with . That second part is super-important since MVC will not do any view lookup caching if you are running your application in debug mode. This helps when you are developing your application and frequently adding/deleting view files, but it will kill our performance in production.

Use only the View Engines that you need

When you use MVC, ASP.NET still loads the View Engines for both Razor and Web Forms by default. This can cause performance issues because MVC normally look for Web Forms view first, before switching over to Razor views if it can’t find them. You can eliminate this issue by adding the following lines to your Global.asax.cs in Application_Start().

protected void Application_Start()
{
   ViewEngines.Engines.Clear();
   ViewEngines.Engines.Add(new RazorViewEngine());        
}

Customize the view lookup caching

By default (when running in Release mode, of course) MVC will cache the results of the lookups in the application cache available via HttpContext.Cache. While this cache works great and helps us avoid having to check for view files on disk there is also a cost associated with using it (this includes the cost of a thread-safe lookup as well as all the additional cache management such as updating entry expiration policies and performance counters).

To speed things up you could introduce a faster cache in front of the application cache. Fortunately all view engines deriving from VirtualPathProviderViewEngine (that includes WebForms and Razor) have an extensibility point via the settable ViewLocationCache property.

	
	public class TwoLevelViewCache : IViewLocationCache   
        {          
              private readonly static object s_key = new object();          
              private readonly IViewLocationCache _cache;          
              public TwoLevelViewCache(IViewLocationCache cache)          
              {               
                   _cache = cache;           
              }           

              private static IDictionary<string, string> GetRequestCache(HttpContextBase httpContext)          
              {                  
                   var d = httpContext.Items[s_key] as IDictionary<string, string>;                   
                   if (d == null)                  
                   {                         
                        d = new Dictionary<string, string>();
                        httpContext.Items[s_key] = d;
                   }                   
                   return d;

             }          

             public string GetViewLocation(HttpContextBase httpContext, string key)          
             {
                 var d = GetRequestCache(httpContext);
                 string location;
                 if (!d.TryGetValue(key, out location))
                 {
                     location = _cache.GetViewLocation(httpContext, key);                          
                     d[key] = location;
                 }
                 return location;
            }

            public void InsertViewLocation(HttpContextBase httpContext, string key, string virtualPath)
            {
                _cache.InsertViewLocation(httpContext, key, virtualPath);
            }
      }

and augment our view engine registration in the following way:

      protected void Application_Start()
      { 
            ViewEngines.Engines.Clear();
            var ve = new RazorViewEngine();
            ve.ViewLocationCache = new TwoLevelViewCache(ve.ViewLocationCache);
            ViewEngines.Engines.Add(ve);
     }

This TwoLevelViewCache will work best in views that call the same partial multiple times in a single request (and should hopefully have minimum impact on simpler pages).

Reference Views by full path if possible

There is a of performance when referencing a view by name (i.e. calling something like@Html.Partial(“_ProductInfo”)) compared to referencing a view by path (@Html.Partial(“~/Views/Shared/_ProductInfo.cshtml”)). When a page is composed from many views and partial views (for example, when you are rendering a partial view for each item in a list) it’s more efficient to reference your views using the full path instead of just the view name.

Cache Often

Make use of the OutputCache annotation on MVC controllers. You can improve your application’s performance to a considerable extent by caching relatively stale data. That way the network bandwidth between the client and the server is also reduced. It is great if you can also cache the rendered action of web pages that are relatively stale, i.e., don’t change much over time.

Profile your Website and Database

Get MVC Mini Profiler and configure it to always run when you develop and test your website. You will get detailed execution times and big red warnings if the same database query is running multiple times or other performance hits.

Always profile your ORM database hits with SQL Profiler during development. ORMS get away from you quickly before you know it, you ran a query 1000 times in a loop, when you could have retrieved all data in one database call.

When In Production, carefully consider what you need to log

Too much and too less logging affect your application significantly. It is always advisable to check what you are logging, which targets you are outputting to, log archival strategy etc. If possible implement async logging. If you use Enterprise Library for logging , add some method like this to enable Async Logging.

    static public void LogMessage(LogEntry logEntry)    {         LogWriter logWriter = EnterpriseLibraryContainer.Current.GetInstance<LogWriter>();         if (logWriter.IsLoggingEnable)         {             if (logWriter.ShouldLog(logEntry))             {                 logWriter.Write(logEntry);             }         }    }    static public void LogMessageAsync(LogEntry logEntry)    {        Task.Factory.StartNew(() => LogMessage(logEntry));    }

JavaScript and Images

  • Add Script references at the bottom of the page, because asynchronous downloads halt when a script reference is reached. Style sheets and images can be downloaded asynchronously.
  • Using external JavaScript and CSS files generally produces faster pages because the files are cached by the browser. JavaScript and CSS that are inlined in HTML documents get downloaded each time the HTML document is requested. This reduces the number of HTTP requests but increases the HTML document size. On the other hand, if the JavaScript and CSS are in external files cached by the browser, the HTML document size is reduced without increasing the number of HTTP requests.
  • JavaScript minification removes unnecessary characters from a file to reduce its size, thereby improving load times. When a file is minified, comments and unneeded white space characters (space, newline, and tab) are removed. This improves response time since the size of the download files is reduced.
  • Use AJAX to retrieve components asynchronously that may not be needed immediately, such as the contents of a collapsed panel or the contents behind a tab etc.
  • Include Height and Width in tag will allow your page to render more quickly, because space can be allocated for the image before it is downloaded.
  • Use Image Sprites to retrieve smaller images in one download

Uninstall URL Rewrite if you don’t use it

If your IIS server has the URL Rewrite module installed but none of the applications on the server are using it and you are running ASP.NET MVC 3. When performing URL generation (for example via a method like Html.ActionLink) in some cases MVC checks to see if the currently requested URL has been rewritten by the URL Rewrite module. If that is the case the result is processed so that it correctly matches the URL requested by the client. The act of checking if a URL has been rewritten has a non-trivial cost (because it involves checking server variables). ASP.NET MVC 3 checks to see if URL Rewrite is turned off and can cache that fact thus avoiding the need to inspect server variables for each request. If URL Rewrite is turned on MVC will have to check server variables even if no rewriting happened for a particular request so if you are not using URL Rewrite you should turn it off (Note that MVC 2 always performs this check so turning URL Rewrite off will not make a difference).