Cloud Computing, Database, Technology

SQL On-Premise to Azure SQL Migration

In this era of Cloud computing, developers are often tasked with migrating databases to Azure Cloud.  This blog highlights some of the steps required for a successful migration. In a recent project, we migrated  12 Sql Server databases with about 1TB data from on-premise to Azure Cloud.

sql azure migrate

Import Connect to the On-Premise SQL Server database and import database schema to a Database project in Visual Studio.
Migration Wizard This is SSDT compatibility checker. Set the target database to Microsoft Azure SQL Database V12. Build the project to start compatibly check.
Fix Errors Fix all errors and warning. Make sure to check for all warning and resolve them before publishing the database.
Publish Schema Create a new local database and publish the schema to the local database. This will create an Azure compatible database.
Validate Database Validate database, run some quick check to make sure all objects are migrated to the local database.
Compare Database Using Visual Studio SSDT data compare, do a compare between the source database and local database. This will create a list all source database with data and target (local database) with no data.
Copy Data Copy the data from source database to the local database.
Validate Database Run the Visual Studio SSDT data compare again. This time the source and target (local database) should match. You can also smoke test your application against this local database to validate the database.
Backup Database Create a storage account on Windows Azure and backup the local database to Windows Azure Storage.
Restore Database Create a database on Windows Azure SQL and restore the backup from the Azure storage.
Validate Database Run the Visual Studio SSDT schema/data compare between the local database and Azure SQL database. This time the source and target should match pretty close. It might not match 100% percent, because of the compatibility issues. Smoke test your application against this Azure SQL Database to validate the database.
Add to Source Control Add Database project to Source Control (Visual Studio online or any other on-premise /cloud based source control)
Advertisements
Architecture, Cloud Computing, Technology

Microservices

A successful application grows more complex over time, and that complexity creates challenges in development. Microservices provides a strategy for decomposing a large project into smaller, more manageable pieces. Microservices encapsulate many of the best practices of some of the industry’s biggest software development organisations and are at the core of the development practice of many organisations today, not least among them Netflix and Amazon.

While a monolithic app is One Big Program with many responsibilities, microservice-based apps are composed of several small programs, each with a single responsibility. When new technologies become available that makes more sense for a particular service, it’s feasible to rewrite just that service.

Microservice Characteristics

Microservices community and Martin Fowler describe the common characteristics of microservice as follows:

  • Componentization via services
  • Organized around business capabilities
  • Products not Projects
  • Smart End Point and dumb pipelines
  • Decentralized Governance
  • Decentralized Data Management
  • Infrastructure Automation
  • Design for Failure
  • Evolutionary Design

 

Componentization via services

The basic idea behind microservice is to split monolithic services to several small manageable components. A component is something that can be independently replaceable and independently upgradable. For example, replace or upgrade breaks on a car without replacing the entire car for a new break.

component_1

Components can be a library or a Service. A library is a piece of code, which is custom or third party, that we reference in our project. For example, an XML processing library we use in our project,  we should be able to upgrade this library without having many problems as long as we have all the dependencies figured out.

Service is a different kind of component, run on its own process. If you have an application with components running.NET 2.0 framework, and if you want to use the component in .NET 4.5  framework, you are out of luck. But with services, it is possible to move the.NET 4.5 component to a service and access via its endpoints.

component_2

Orgaserviceound business capabilities

A lot of the development organisation are organized around technologies. There is a team for UI, a team for Database etc and team for Business Logic etc.  The key thing about the microservices world is to organize around business capabilities rather than technologies. Each team should have some elements that directly impact end user experience.

organization_1

 

Smart End-Point and dumb Pipeline

When you talk about SOA (Service oriented Architecture), we often talk about getting a power piece of middleware which does a lot of the heavy lifting or processing logic automatically. This called the ESB, enterprise Service Bus.

endpoint_1

 

Microservice community rejects this idea, and they suggest moving all the processing logic to the end point themselves. What we want is connect everything together and process message effectively but it should be up to the end point to how and when to process the messages.All we want is a dumb set of pipes to route the messages.

endpoint_2

Decentralized Data Management

Microservices strongly argues against the way the technology landscape is governed today, particularly the way data is managed. When you look at the monolithic application, all of the data is sitting in one big data store, often time in a company standard relational databases. Even in SOA architecture, multiple services pulling data in and out of the same database.

decentralize_1

 

Microservices way of managing data is to have each service responsible for its own data. A service should never talk to another services data store directly, but only through its APIs. This approach has some advantages; first, it removes the mess of integrating through the database, second, the service developer is free to choose a datastore best fit for the service.

decentralize_2

 

Infrastructure Automation

To make all these works, we need infrastructure automation. Things like continuous delivery and blue/green deployment where you can deploy with zero downtime are mandatory for microservice to be successful. With Microservices, we are talking about one application with a dozen or so services, so we need an automated way to doing things. You should be able to spin new instances of VM relative quickly.

Also in order to make Microservices successful, we need effective monitoring capabilities. So when things go wrong, we can quickly spot them and fix them.

 Design for failure

Traditional approaches often equate reliability with preventing failure.  When you have several components/services deployed on multiple nodes, chances are some of them are going to fail. So we need to design the services with failure in mind. That’s why monitoring should be an integral part of the microservice architecture.

 

Conclusion

A microservice application separates functionality into separate smaller services and scales out by deploying each service independently. This preserve modularity, allows us to choose multiple platform/technologies and work well with the cloud. In order for this to work, we need to adopt a better DevOps culture, better infrastructure and monitoring capabilities. But overall, microservies provide a better development experience and agility of the technology landscape.

 

 

BigData, Technology

4 Vs of Big Data

    • Volume: We currently see the exponential growth in the data storage as the data is now more than text data. There are videos, music and large images on our social media channels. It is very common to have Terabytes and Petabytes of the storage system for enterprises.
    • Velocity: Velocity describes the frequency at which data is generated, captured and shared. Recent developments mean that not only consumers but also businesses generate more data in much shorter cycles.
    • Variety:
      Today’s data no longer fits into neat, easy to consume structures. New types include content, geo-spatial, hardware data points, location based, log data, machine data, metrics, mobile, physical data points, process, RFID etc.
    • Veracity: This refers to the uncertainty of the data available. Veracity isn’t just about data quality, it’s about data understandability. Veracity has an impact on the confidence data.

Big Data

Uncategorized

RFP Workflow

What is a Proposal

  • Project proposals are documents designed to present a plan of action, outline the reasons why the action is necessary, and convince the reader to agree with and approve the implementation of the actions recommended in the body of the document.
  • Proposals are usually drafted as a response to a Request for Proposal (RFP) that is issued by a current or prospective client.
  • A project proposal will be clearly arranged so that readers can follow a logical progression of thought to the conclusion.
  • Often, the RFP will lay out the basic structure of the proposal, provide invaluable clues as to specific information that is of interest to the potential client, and define the order in which data is presented.
  • When an RFP is provided, it is essential to follow the specifications of the document to the letter. Otherwise, the proposal will be set aside and one of the other vendors who did follow the provisions closely will be awarded the business.

12 tips to a write a killer proposal

  1. Create a powerful, but concise executive summary     Decision-makers start with and focus on the executive summary, so create this section with that fact in mind. When writing the executive summary, assume that the reader knows little or nothing about the proposed project.
  2. Quantify the results that the client can expect from engaging you     Some consultants create proposals that overemphasize their consulting process and methodologies. Clients buy results, not tools or methodologies.
  3. Be generous with your ideas     You may fear that revealing your ideas about how to solve a problem during the proposal process could result in clients taking those ideas and completing the project themselves. In rare cases, that may happen. But you’ll have more success if you don’t hoard your ideas. Use them to show clients that your team thinks and approaches problems in creative and innovative ways.
  4. Size does matter     Keep your proposals as short as possible, while meeting the client’s request. Think quality, not quantity.
  5. Focus on the client     Many proposals begin with a long discussion of the consulting firm, describing its qualifications and history. Focus your proposal on the client’s needs first, and then describe your firm’s capabilities. Remember, clients care only about how you’ll address their issues, so show them how you’ll do that.
  6. Beware of best practices     The client may view your liberal use of “best practices” as a convenient crutch. Instead of relying on answers that worked for a previous client, find a blend of outstanding practices and innovative solutions that fit your client’s particular needs.
  7. Be accurate     If you are using client data to support aspects of your proposal, double-check and triple-check that information. It’s easy for facts to be misunderstood and misused in a proposal. You’ll risk turning a winning proposal into a loser if you present inaccurate data to the client.
  8. Sweat every detail     Watch for typos, use high-quality materials, and make sure that the right people receive the proposal on time.
  9. Rewrite your resume for every proposal     Highlight the skills in your resume that demonstrate your qualifications for the project at hand. A boilerplate resume is rarely up to the task.
  10. Finish early     Let your proposal sit for a day after you’ve completed the final draft, and then reread it completely before sending it to the client. You’re likely to come up with some new ideas that enhance your work, and you may find errors that you missed earlier.
  11. Let your personality shine through     Give clients a sense of your firm’s culture and its style of working. The traditional, stilted language of many consulting proposals doesn’t help clients answer the all-important question: What will it be like to work with these consultants?
  12. Don’t let your claims outdistance your true capabilities     Some proposals tout the expertise of the consulting firm by referring to past successes with similar projects. These testaments to past achievements are important, but be sure that the capabilities of the proposed consulting team can live up to your firm’s claims.

 

The proposals definitely a team effort. The following diagrams shows the workflow I normally follow to develop Proposal for an RFP.

 

Slide5

  
Slide6
  
Slide7
  
Slide8
  
Slide9
  
Slide10

    
 Final Thoughts
  
Project proposals are documents designed to present a plan of action, outline the reasons why the action is necessary, and convince the reader to agree with and approve the implementation of the actions recommended in the body of the document.

Proposals are usually drafted as a response to a Request for Proposal (RFP) that is issued by a current or prospective client.

A project proposal will be clearly arranged so that readers can follow a logical progression of thought to the conclusion.

Often, the RFP will lay out the basic structure of the proposal, provide invaluable clues as to specific information that is of interest to the potential client, and define the order in which data is presented.

When an RFP is provided, it is essential to follow the specifications of the document to the letter. Otherwise, the proposal will be set aside and one of the other vendors who did follow the provisions closely will be awarded the business.

Technology, Web

Responsive Web

Over the last few years, website traffic from mobiles and tablets has increased dramatically as smartphones become more prevalent. Unfortunately, many websites were designed to be viewed on large screens and, as a result, the experience on phones is pretty bad – having to zoom in and out of sites to read text and be able to navigate around. Responsive web design is an approach which suggests that, instead of users having to adapt to be able to read sites on phones, the website itself will change so that an optimized version is always shown no matter which device is used.

Responsive Web Design is a Web design approach aimed at crafting sites to provide an optimal viewing experience—easy reading and navigation with a minimum of resizing, panning, and scrolling—across a wide range of devices (from mobile phones to desktop computer monitors) – Wikipedia

While changing the design of a site is a key part of responsive design, there’s a lot more to it than simply changing the layout.

responsive_design

Core Experience
With smaller sizes, not everything can fit onto the screen. Similarly, slower connections shouldn’t be made to download extra ‘nice-to-have’ features such as web fonts or large images – you should consider what the core features of your site are (i.e. the content!) and make sure this is loaded first so that users aren’t negatively affected by the size of their device or speed of connection.

Screen Size is not the defining factor
Take into account the speed of the connection before serving large images or web fonts – just because a device has a large screen, it doesn’t mean that it will have a fast internet connection. Also, consider whether browsers can handle certain features; if browsers can handle things natively, use them! Forcing users to download polyfills to fix issues in browsers they aren’t even using is bad and slows the site down for everyone.

Multiple Experience
It’s OK for the same site to look different on different browsers. You can’t make a site look identical on desktop and on mobile so focus on creating the optimum experience for each one rather than having one all-round average version. Since sites don’t look the same on mobile and desktop; it’s an easier conversation to have with the client to explain that the site doesn’t have to be pixel perfect in every browser. As above, it’s possible to make the site look exactly the same in ancient browsers but this comes at the cost of making the site larger and therefore slower.

Mobile First Responsive Web

Mobile-First Responsive Web Design is a combination of philosophies/strategies, and ultimately boils down to a broader application of good old web best practices. Mobile Web is a philosophy that highlights the need to prioritize the mobile context when creating user experiences. Starting with mobile first:

  1. 1. Allows websites to reach more people (77% of the world’s population has a mobile device, 85% of phones sold in 2011 equipped with browser)
  2. 2. Forces designers to focus on core content and functionality (What do you do when you lose 80% of your screen real estate?)
  3. 3. Let the designers innovate and take advantage of new technologies (geolocation, camera, touch events and more)

Creating a responsive web design utilizes:

  1. 1. Fluid grids that ebb and flow with a devices’ screen size
  2. 2. Flexible images and media that keep content intact on any resolution
  3. 3. Media queries allowing designs to adapt by establishing dimension breakpoints

Ultimately, mobile-first responsive web design comes down to using Progressive Enhancement as a foundation for web strategy and design. Designing with progressive enhancement involves smartly adding layers of enhancements to a strong foundation in order to deliver an accessible (and hopefully optimized) experience to all.

Scaling up from the mobile context versus scaling down from the desktop context ensures that your message, content and functionality remain intact as the screen real estate and connection speed increase. Scaling down runs the risk of your core message and functionality getting lost by the time you squish it all the way down to the mobile context.

The picture below summarize the Responsive Web in one slide:

ResponsiveWeb