Timeless Software Architecture & Development

Kerberos SPNs and Double Hops in a Nuthshell

2012-02-04T22:44:00.000-07:00

By Saleh Najar

You are implementing an enterprise distributed application and everything is working nicely on your development machine. All layers are communicating properly. Your web UI is talking nicely to your WCF service and your WCF service is talking nicely to the database. Suddenly, when you deploy to a multi-tier environment, very close to your production release deadline, all hell breaks loose. You start getting weird non-descriptive errors that you suspect have to do with permissions. You could swear that everything was working fine in your development environment. What happened?

Welcome to Kerberos authentication double-hop SPN issues! Desperate to find out and after a thorough analysis, you find out that the only difference between your development environment and the UAT or test environment is the deployment architecture and topology. In your development environment, all your layers were running on the same tier or machine. In the test environment your layers were distributed over three tiers where you have the web server on one machine, the WCF service on another machine and the database server yet on a third machine.

When running on the same machine, Kerberos authentication is not engaged however, when two machine are communicating, Kerberos is engaged and when three machines are communicating then Kerberos SPNs are involved in the infamous double-hop scenario.

In the two-tier or two-machine scenario, Kerberos authentication should not cause a problem as the authentication is straight forward. The problem starts when a third machine is involved and the second machine has to delegate to that third machine. In other words, you have a web server that needs to impersonate the user to the service machine where the business logic resides and in turn the second machine or tier needs to communicate with the third machine: the data tier.

In this scenario, if SPNs (Service Primary Name) are not set up properly for the customer network account you are using, you will run into security issues. The best place to check for these issues is the Events Viewer as they are marked nicely as Kerberos errors.

Before Kerberos SPNs, authentication was only one way, the server authenticating the client and user. With the rise of server spoofing, the need came to also have the client authenticate the server to make sure that it wasn't spoofed. This is what Kerberos SPNs were created to solve. Service Primary Names (SPNs) are a mechanism that enables the client to authenticate the server to make sure it is talking to the authenticated server on the network.

For a client to authenticate a server's service, it needs some kind of an identifier to send to the authentication authority (KDC). This identifier is what is called an SPN or Service Primary Name. The format of this SPN is predetermined by convention. For example, the browser knows how to put together that SPN from available information.

Once a client forms the SPN string, it sends a request to the authentication authority to authenticate this SPN. The authentication service in return checks its store (Active Directory) to see which network account is this SPN associated with, then uses the password of this account to encrypt messages and communicate with that service host to make sure it is the right server. If that server service is able to decrypt the message (using its password) and reply to the authentication service, then the authentication service or KDC knows that this is the right service with the right SPN so it authenticates the service to the client who requested it and communications goes on. That's Kerberos SPNs in a nutshell.

So back to our three-tier scenario. To have your distributed system work in a three tier scenario with custom accounts (service accounts or user accounts) make sure to create an SPN for your first and second tiers and make sure to associate them with the account that is running your web application and your WCF service. You will need domain admin permission in order to do that using the Microsoft spn tool. You also need to allow delegation in Active Directory for these machines. You will need a domain admin to work with you on this and make sure to allow enough time in your project schedule because in a big enterprise, it typically takes around a couple of weeks to get your request for a domain admin resource to work with.

If your services run using system accounts such as NETWORK SERVICE then you don't need to worry about creating new SPN as the default ones will be used.

Architectural Pattern: Caller Dependency Controller

2011-10-16T08:41:00.004-06:00

By Saleh Najar

If you are dealing with multi-layer architecture, you will frequently run into having to make the decision of where to control a dependency. For example, if you're calling the Data Layer should you let the data layer decide which data layer object family to instantiate or should you let the Workflow inject that? Should the Data Layer decide when to submit a transaction or should this be decided at the higher layer?

As a best practice, the Caller Dependency Controller (CDC) dictates that the caller should always act as the controller of the dependency. In our example above, it should be the Workflow layer who gets to decide 1) the type of Data Layer objects to create and 2) when the Data Layer gets to submit or commit the changes. If this pattern is not followed, you'll pay dearly later by having to re-write a lot of code crossing multiple layers right before the release date. Not good.

For instance, submitting from the Data Gateway layer is fine and dandy when you're dealing with simple transactions but what if you're dealing with multi-step transactions? If the transaction involves more than one step such as making a withdrawal and updating a customer balance, the Data Layer (the dependency) must be controlled by the Workflow Layer (the caller) to maintain atomicity, to only submit when both steps are successful.

If we let the Data Layer decide and commit every time it's called then what happens when making the withdrawal succeeds and commits but updating the balance fails? As you can see in this simple scenario the callee has no high level visibility of the full use case or transaction so it shouldn't make that decision.

This pattern is not just limited to layers, it applies to any situation where you have a caller and a callee, a function calling another function, even when a client calling a server (cross-boundary transactions). We know this when we write simple micro functions and it also applies to macro structures. In micro functions, it always makes sense and without much thinking to maintain the result of the calculation in the calling function. By the same token this applies to bigger structures such as layers and systems.

Architects should set the guidelines for developers straight on this one and look for it in their design reviews.

Significance of Interfaces in Enterprise Integration Architecture

2011-07-27T20:38:00.000-06:00

By Saleh Najar

The trend in large enterprises nowadays seems to be in creating enterprise data integration architectures. Business leaders are becoming more aware of the huge costs associated with having duplicate, erroneous and inconsistent data scattered all over the enterprise in spread sheets and little Access databases, more referred to as the 'data swamp'.

In this short article, I would like to shed the light on the role of interfaces and what they really mean when creating the foundation of integration efforts. We hear interfaces to mean flat files, views, web services, database layer, xml files. These are all valid implementations or artifacts of interfaces, however, these are not all what an interface is or should be.

Just because a department exports their data to a delimited flat file for example so that other systems can pick up in an ETL process, this doesn't constitute an interface in the architectural sense

What happens if this flat file is in a different environment then its clients? In that case, the clients would need to know the access details and protocols to access that file. Multiply this by ten different client systems trying to integrate or ETL this data and you have a fragile system. Why fragile? for one thing, this breaks the abstraction rule in good architecture. Now all ten clients must know the internal details of the source environment in order to access it.

What happens if the access details of the source environment change? Now we have a cascading problem. All client systems would have to update their access methods to ETL data out of this source.

Another problem will be encountered if the source system decided to change the schema of data. Now suddenly, all ten client systems will break and all data warehouse reports and front ends will break too.

So from the above examples, it becomes obvious that an interface is not simply equal to its implementation format. It is more than that. At least an interface must include the following attributes to serve as a true interface that provides stability in an integration architecture:

1. The interface should be understood by teams on both sides to mean a real contract. This means that the source system must promise the target client system that the schema will not change without a proper change management process and notification. Guaranteeing backward compatibility or versioning the interface is a big plus (web services are ideal for this).

2. There must be loose coupling and dependency inversion. This means that the client systems must not depend on knowing the details of the source systems. This dependency should be inverted and both system should depend on a third system accessible by both to retrieve data from. In our example, the source system should place the delimited file on a universally accessible drop location using standard protocols, instead of requiring client systems to depend on its internal details.

3. The format of the interface must be standard and not internal to either the source or the destination systems.

The above list is not comprehensive, but using those quick core rules for understanding what an interface means will help create a stable and reusable enterprise integration architecture implementation. Above all, it will create a common understanding among team members as to what an interface means when they use that word.

Setting up a project for success with a client

2011-06-04T09:45:00.001-06:00

When setting up a new project with a client it is critical to analyze what kind of a project it is from the customer involvement perspective. Understanding the degree of involvement of the customer in the Form and Function elements of a project is foundation for its success.

Form and Function exist in every project. Every solution has an aesthetic element and also a functionality element. So what I mean by form is the aesthetic angle including style, shape, theme, color, usability etc... and what I mean by function is the actual value that the system delivers to the client.

To make it measurable and therefore controllable, let's give an index value to the degree of customer involvement with each element using a range from 0 to 10. For example, if the customer's interaction with Form is high we give it a 10 and if the interaction with the Function element is low, we give it a 0. Then we can say that the Customer Involvement iNdex (CIN) for this project is: 10-0.

A project's CIN value can serve as the foundation for defining the rest of the project parameters and resources from estimation, development methodology, type of resources to use to the tools used.

For instance, what is the CIN that you would give a project that involves building a standard security or database engine? On the Form factor, I would give it zero as there's little involvement or say of the customer on how it looks and feels, after all it is an engine.

On the Function factor, I would also give it a one as beyond defining the objective of the engine, there's not much customer involvement. So for this project the CIN is 0-1, very low. Now we can make our selections on how to set it up (including the expectations of the client) in accordance to a low CIN rating.

For example, for a estimation/billing model a low CIN rating leans itself towards giving the customer a fixed price. Also, there wouldn't be a need to setup a system of project tracking with the customer. We can also use a waterfall development approach vs. iterative, also we can use outsourced resources vs in-house etc... All based on the CIN value of the project.

On the other hand, a custom software development project has a high CIN rating close to a 10-10. For instance, there will be a lot of customer involvement and say in the Form factor and a lot of customer involvement in the Function factor. The customer's change requests will be endless because it would be an exploratory project.

So for a high CIN project, we can choose an iterative development methodology, a project tracking system, a change request process, T & M billing if we can get away with it or just include a 50% buffer, use in-house resources etc...

One last example, let's suppose the project is a simple website development project, what is the CIN for this kind of project? Websites usually involve a lot of aesthetics so let's give it an 8 on the Form factor. For function, a simple website's function is well defined so we can give it a 5 on Function. So the CIN for this simple website project is 8-5. This can help us setup this project for success by either limiting how many mockups a customer can have or charging T & M for that aesthetic part and give fixed pricing for the functionality part. A hybrid approach.

Paying attention to the Customer Involvement iNdex (CIN) can make or break a project.

Enterprise Architecture: IT Investments & Growing Systems Complexity

2011-03-27T17:08:00.000-06:00

Throughout my fifteen years of professional experience in designing, implementing, deploying and maintaining IT systems, I’ve observed a pattern: Out of the business need to stay competitive, IT systems are getting more complex; however, these systems aren’t aligning with the business objectives of their organizations.

Organizations are spending more money but realizing less value from the unmanaged growing complexity of their IT systems.

I have helped organizations facing the following challenges:

1. Mapping business strategies and objectives to a well defined set of streamlined and integrated systems.

2. Systems growing in complexity and cost to maintain and manage, to the extent that some of those systems were crumbling under their own weights.

3. Organizations that lost many market opportunities because their IT systems did not allow them the flexibility to respond to current and near-future competitive demands. The weakness or non-existence of enterprise architecture effectively held back their growth.

4. Organizations that ran at higher-than-normal costs because of lack of integration policies between their existing systems.

5. Critical business data that was out-of-date or housed in duplicate repositories.

6. Business processes, although well defined, were ignored because of the lack of proper IT governance policies and commitment from top management.

I’ve come to understand that as systems become more complex they require more planning. Enterprise Architecture helps maximize the return on every IT dollar invested delivering real business value.

Also, I’ve seen that every organization is different; no one enterprise architecture approach fits all. Whether it is Zachman, TOGAF, or Gartner/Meta, each of the prominent enterprise architecture frameworks and processes has its strengths and weaknesses. In my experience, none of the top enterprise architecture frameworks is complete by itself. The winning enterprise architecture approach is often a hybrid.

It’s important to note what I have witnessed to be a major reason why enterprise architecture fails to deliver maximum business value: For an organization to reap the competitive benefits that enterprise architecture affords, it requires the commitment of the organization’s top management. Top management’s commitment is crucial when it comes to success of enterprise architecture because enterprise architecture is, at its roots, a set of processes, policies and frameworks.

At the end of the day, what enterprise architecture must deliver is business value and agility - reducing IT costs and complexity and increasing efficiency and effectiveness.

What is the Software Architect's Role in Development?

2011-01-09T16:55:00.000-07:00

The outdated view that a Software Architect is someone who sits on a pedestal and draws diagrams is an old and defunct paradigm that failed to provide business value in reality. Reality proves day after day that this portrait of an SA doesn't fit with the nature of business and the nature of software development as an exploratory process.

I have seen first hand how paper solution architect having flaws in their thinking when they do the top level diagrams. Because they lack the low level implementation detail, I've seen them often over-engineer and therefore, increase the cost of systems in orders of magnitude because they don't know the low level implementation details.

I've witnessed those "paper" architects being replaced so fast because the smart business people realize soon enough that these architects are designing systems specifications that are not in touch with reality. Plus, for the sake of argument, even if the business folks keep those "paper" architects, their involvement in the project is minimal. After they draw the diagrams, the "paper" architect's involvement is diminished significantly.

And I'm sure you know how fast the gap between the paper diagram and the real implementation grows if the SA's role is just diagramming.

In reality, and what I've observed in all small, medium and large corporate environments, that Software Architect role has evolved to be providing the following business values:

1- The software architect is the one to maintain and enforce *ongoing* conceptual integrity of the system to the development team.

2- The software architect is looked up to when developers on the team need to understand a design pattern or have a question on how to proceed.

3- The software architect MUST have come from a *guru* level development background to solve complex implementation problems for the development team and this will also gain him tremendous respect and buy in.

4- The software architect is the *factory* that implements/codes the components for cross-cutting concerns that every system needs.

5- The software architect is the one who can and must do the integration of all all features and code together.

6- The software architect is the one to inspect code to make sure it implements his/her top level design.

7- The software architect is the one to do design reviews of code.

8- The software architect is the one who represents the full system view to all stakeholders

For all the above value to be delivered the software or solution architect must have impressive and always-current coding, design patterns, architecture and high-level view skills.

He or she must have the mental flexibility to go high and low, to be an evangelist, politician and the engineer. He or she must be able to role up their sleeves and dig into the code and shine anytime anywhere.

That's the reality. That's what businesses are looking for. That's the solution or software architect who provides true value and is worth paying the big bucks.

Notes on Git and the difference between the git revert and the git reset commands

2010-09-17T02:15:00.000-06:00

It was love at first sight with git. After having been using git as the source control management tool (SCM) on a few projects, my appreciation of its architecture and ease of use grew stronger.

Git's Main Features:
I don't imagine working on a serious project without git. It's my secret weapon (not anymore after this article). What I love about git are the following main features:

1- Distributed and peer-to-peer SCM vs. the traditional central repository style

2- Extremley powerful and flexible branching that's built into its bones

3- Works great as a compliment to central repository SCM. In my opinion, git was the missing piece in the overall source control management discipline. Of course, Git can also act as the central repository as one of the nodes

4- Intuitive command line usage

5- Small foot print and simplicity

6- Introduces the concept of a third staging state (index) vs. just working set and committed states

Git's Benefits to the Development Team
1- Increases innovation. Git encourages experimentation which is the corner stone of innovation. Let's face it, a lot of developers shy away from experimenting with new ideas because they don't want to check in their experimental code into the main tree and have everyone look at it or the check in policies are strict. Having a local distributed SCM encourages the developer to play for as long as he or she wants locally with multiple versions and branches and only check in once they found their EUREKA.

2- Developers have their own local versions and branches seperated from the central repository. How many times as a developer you were asked to drop everything you're doing and work on some other code or do a quick build?

More often than not, you wouldn't want to check in your work-in-progress into the main tree. With git, you can simply switch to any build or version (commit) or branch, do a build or what have you, then easily go back to what you were doing without touching the main central repository, under the radar while your manager is wondering how could you pull this off so quickly and without waiting for the official build date when the stars are all aligned.

Implemenation Scenario of Git
I don't recommend that companies replace their central repository approach with a distributed one, to the contrary, the git approach and the central repository approach compliment one another.

What I do recommend is that each developer should have a copy of git locally to unleash their creativity, provide agility and only check in the "good" code into the company's central repository for safe keeping. The best of both worlds.

Difference Between Git Revert and Git Reset
A quick google search will introduce you to the basics of git, however, what you will not find (rarely with git), is a clear explanation of the confusing usage of the reset and the revert commands. So, I will attempt to clarify for those of you who are still struggling with them, like I was, to know when to use each.

Think of git reset [commit], especially with the --hard argument as the command that moves the HEAD "pointer" and working set to any version or "commit/checkin", disregarding all subsequent ones. It is like random access to any commit with the addition that it "resets" the HEAD pointer at that location. It is simply your HEAD mover.

On the other hand, git revert [commit] only reverts the [commit] and your current working set to the previous commit. It simply "reverts" the commit you point to the previous one. This last statement is very important to get and is mostly the source of confusion.

You can conclude from this that git revert only works on work that you have committed and not if you have code that is "not checked in" yet.

Usage Scenarios of Git Reset and Revert
For git revert, a typical usage scenario is when you get a new idea or you think you are going to refactor some code, and after you commit your supposedly "better" code, you discover that oops, it broke other areas and the refactoring was more involved that you had thought.

In that case, the proper git command to use is git revert HEAD which simply tells git to move the HEAD and index and working state to the previous state and record this revert as a new commit for records keeping.

For git reset --hard, a typical usage is the same as the above scenario but this time, you haven't commited your supposed improvements yet. Also, you just want to trash your changes. As I explained, this command will move the implied HEAD pointer (you don't have to type it at the command line) to the last commit. The --hard option will align your working directory and index with it too. So you will lose your changes and bring back the last commited one.

Another scenario is when you want to move the pointer HEAD to some old commit in the past where you want to start fresh. This is not a likely scenario as you should've used the powerful git branch features, never the less, this demonstrates the "random" access to your commits of the reset command.

Resolving Hard-to-Catch Multi-threading and Concurrency Issues

2010-06-27T20:56:00.006-06:00

Locking, Memory Barriers and Atomicity
When talking about multithreading, developers usually focus on one part of the equation which is locking or pre-emption of multiple threads. What they most of the time overlook is memory fencing and atomicity.

Know when to think and apply thread safety?
Basically, anytime the same variable or resource can be read or written to from different threads. In that case, you should review the code for three areas: Pre-emption, Memory Fencing and Atomcity. Then, apply the correct mechanism in a minimalistic way instead of wrapping a lock around everything and increasing the chances of deadlocks.

For example, if a 32 bit int is read in one place and written in another place by different threads, then concurrency is not an issue on today's processors because the int is is 32 bits and can be read by the processor in one pass. No lock statement is necessary, it is already atomic.

However, the issue that is mostly overlooked is that you still have to deal with the fact that the other thread might not see the up-to-date last value of that variable. This is because some processors try to optimize and cache values for threads from the same processor. This int must be "flushed" before the other threads can see its latest value.

To fix this, create a full memory fence by declaring the variable as volatile (in c#)to prevent the processor from optimizing and caching.

As far as pre-emption is concerned, don't bother about it here as it doesn't apply. This is because the variable is not read and written by two threads in the same place.

In summary, whenever you have a variable that can be read and written by multiple threads, analyze it for all three things: pre-emption, memory fencing and atomicity.

Memory Fence
For multithreading to be robust, a developer must handle pre-emption and memory fencing. Pre-emption deals with race conditions and deadlocks. Memory fencing deals with flushing memory from the cache to the heap. This is necessary because many processors optimize and cache variables in the cpu registers and making sure this optimizations is transparent to the current thread, but not necessarily the other threads.

For example, if you store int x = 5, the processor might not write it right away to the main memory, making it invisible to other threads. What is needed is flushing or forcing the writing of x to the main memory. In the .NET platform, you can accomplish flushing using the Threading.MemoryBarrier() or the volatile key word.

The volatile keyword basically tells the processor to not cache or re-order the variable. It will be current all the time for OTHER threads. Remember that for the current thread, this doesn't matter because the processor guarantees that no matter what caching or optimization happens, the current thread doesn't get affected.

Some Types Are Treated Atomically
It is important to also know that some variables are read and written atomically. That depends on the CPU bus size, or how many bytes can the CPU read/write in one pass. For example, on 32bit machines, any variable size of 32 bytes is read in one pass. No need to lock it. But if it is a 64 bit variable, then it is read in two passes and a lock is needed to avoid the possibility that another thread might read or write the other 32 bit seqment of that 64 bit variable and getting weird values. A very hard to catch bug. On the other side, on a 64 bit machine, a 64 bit variable is atomic because the processor reads and writes it one pass.

Locking Ways
On the .NET platform, different locking mechanisms exist for every level. The highest ones do all three areas I mentioned above of locking (concurrency (preemption), memory fencing and atomicity). Others handle one area only.

1. The lock, monitor, AutoResetEvent and ManualResetEvent, Mutex and Semophore all implement all three areas of multithreading.

2. The Interlock handles preemption and memory fencing for quick operation like increment, add, assignment

3. The volatile key words and MemoryBarrier() handle fencing only making sure that memory is flushed from the processor to the main memory so the other threads see the updates. No locking or waiting.

How to use MemoryBarrier() in Microsoft .NET?
Basically, MemoryBarrier() is more granular than volatile. To create a read memory fence, then call it BEFORE the read. To flush memory to main when writing to a variable, call MemoryBarrier() AFTER the assignment.

Creating an Extensible UI on Rich Clients

2010-05-02T09:22:00.012-06:00

So you were a very happy software developer when you rapidly created the two-tier rich client application for your customer. You dragged and dropped away the forms and user controls, hooked up their events and voila!, you released it to your customer.

Trouble starts a few months later when your customer asks for a few 'simple' modifications that touch the User Interface (UI). For example, he wants a drop down control to be populated based on the invoice type or a data grid to have certain cells editable depending on the account type. At first glance, these requested features seem simple but not after you look into your code and think about how to 'squeeze' or 'hack' in these new features.

If you have not implemented the UI with extensibility in mind, you're about to start diminishing the return of investment (ROI) of your app. Your code will start smelling by the process of patching in new code. This will gradually bring your app development-abiliy to a halt where you won't be able to modify it anymore. A few modifications later, it will crumble under its own weight. You wouldn't be able to respond to your new customer's features and your code becomes what's called a BBOM (Big Ball Of Mud) or start to resemble a famous Italian dish.

However, if you did implement your UI with an extensibility design in mind, you wouldn't be 'hacking' in those new features. They would fit right in like a glove.

By now, you're thinking Model-View-Controller (MVC), but if there's a pattern so widely mis-applied MVC would be the one. This pattern is rarely clarified or implemented properly for rich clients. Usually, I've seen developers implement one side of it but not the other and therefore not reaping the full benefit of using it in the first place.

Since the program is started by a Main Form, it is very convenient (and error prone) for the developer to start the control from the Main Form (UI) and consequently have all forms do a lot of business logic work while creating a 'cosmetic' controller to handle data manupulation.

An extensible MVC implementation is to have the controller implement an interface (IController), initialize and create the view (forms). Then, injecting itself in that view. After that, the have the controller subscribe to the view's events to handle communication. Now the view is totally decoupled from the UI controller.

This design will allow for swapping the controller with another one to implement different variations. So when your customer asks for a variation on UI behavior you can easily subclass the controller or compose it with a different UI strategy.

To go one step further, you can make the view (forms) reusable by having them implement an IView interface of their own. That way you can swap a different view anytime with minimal coding and least errors.

To keep the view interactive have it subscribe directly to the model data events or the more groomed controller data events. That way, the view is decoupled from the model and controller.

One issue remains is the data transfer objects (DTOs) or entities to carry data back and forth from the view. Should they be leaked all the way to the view or make the view totally data generic and use dictionaries and name/value pairs to display data? In other words, should the data in the view be strongly typed or generic?

Although from a purely design standpoint it would be better to have a totally data-generic view for the purpose of reusability of the view, my experience on this with complex/large data entities has been towards using strongly typed entity data in the view whenever you can. In reality, forms are coupled to their projects and are not widely shared, so it is not worth the pain of mapping back and forth from generic data containers to the domain/model data entities.

This is unless of course it is a general utility form. In that case, it makes sense to keep its data generic like dictionary of strings from drop down controls and name/value pairs.

It is simpler and easier to use the same model data entities that are used in the domain and it doesn't hurt extensibility. This is because by definition, these model data entities or DTO's are standard, versioned and also change with the domain data entities anyways. If you change your domain/model data entities, chances are you will need to make those changes to the view data entities, so mind as well reuse the same data entities.

This does mean that as a good practice you should have your domain/model data entities seperately designed and maintained for reuse in all your application layers.

So in a nutshell, to have an easilty extensible UI, do the following:

1. Implement the full MVC pattern by having the controller implement an IController interface, instantiate and initialize the views (forms and user controls).

2. Use events (Observer pattern) and abstraction (interfaces) for both the views and the controller to interact with each other.

3. Use independent DTO's or domain/model entities to communicate all the way to the view in most cases.

I hope now that you have less pain and less bugs adding more UI features to your app. Long live well designed apps!

Open Source Software: Exploitation At Its Finest

2010-03-04T14:19:00.004-07:00

Open Source is bad for small developers, bad for business users and amazing for big software vendors. As far as innovation for humanity is concerned, the jury is still out. As far as I can tell, so far, most innovations were produced by proprietary/closed innovation organizations. Examples are the computer mouse, graphical user interface, iPhone, all NASA and government agencies innovations etc…

Open Source seems to expedite development in the domain of generic, commodity-type software projects such as databases, operating systems etc... This is expected. I mean upon stumbling on a great innovation, why would a rational human being developer give it away for free when they can benefit from it either by licensing it to big companies or starting their own? Would you?

Now, why would a corporation whose sole existence is to maximize shareholder profit by selling software such as Microsoft and Google adopt Open Source? If you notice, in the beginning, Microsoft opposed Open Source, but later, when they "saw the light", they changed their position.

Of course, it is naïve to think that these corporations are doing it from the goodness of their heart and their community spirit. We know that a corporation's mandate is to maximize profit. This can be done either by increasing revenue or lowing costs. Obviously, they see something in Open Source that dramatically enhances both approaches to maximizing profit.

But before I get into the principle of it, let's take care of the business user first and see the effects of Open Source.

Open Source Software and Business User's Perspective
From a business user's perspective, although it seems to be cheaper to use Open Source at the start, what is often overlooked is the total cost of ownership (TCO). Despite the utopian belief that underlies Open Source that somehow developers work for free, even Open Source developers have to make money to live. One way they do it is by charging for support later. So for the business user, the ride is not free. There's no free lunch. They will have to pay for support later, thus increasing the total cost of ownership (TCO).

In addition, the TCO of Open Source is further increased by the need to customize further and resolve the widespread integration and versioning issues that plague Open Source software.

Now, if the business user requires security, it doesn't look too good either with Open Source. What worse security can someone have when the source code is open for any hacker to study and exploit? Yes, one might argue that this is neutralized by the illusion that anyone also can read the code and figure out the exploitation, but this remains theory as it is rare that a developer would go through every line of code looking for hack-ability.

So if a hacker knows that your company is using an Open Source software, he or she would go study the Open Source of that project that is available on the web and figure out a way to hack it, then launch my attack on your server. Usually, big software vendors who use Open Source software projects on their servers add their own proprietary code to seal off any vulnerabilities. But does the average business user have the resources to research and do this?

On the legal side and business disrupt-ability, the other big issue for users is the risk involved in indemnification. Pardon me as I could not find a better example, Open Source software is like a woman who have slept with many men and no one knows who the real father of the baby is. It is touched by many and no one is responsible. There's a big litigation risk as we saw in the MySQL court case.

Here's how it would go. Some developer somewhere would copy someone else's code and paste code into an Open Source project where he or she infringes on that patent holder. Trouble comes when this patent holder realizes the infringement and goes to court. All copies of the software that contain the infringement code will be ordered to cease and desist bringing the business to a halt. It is like opening a can of worms.

A business customer needs a single point of accountability, they want to focus on their own business. These business customers don't have time to fuddle with the code. Open Source is created by many where no one is responsible. This is unacceptable in the business world from a user's perspective who are not software vendors themselves and don't know anything about opening the source code and tweaking the code. They would still have to pay someone to do it, thus, increasing the TCO again.

Looking at the above points, one can conclude that from the user's view, Open Source seems like a gimmick where they're enticed by the free software upfront, but later suckered into paying for support, integration fixing, customizations and a possible huge lawsuit that could pop its head at any time. Even worse yet, desisting from using the software until a court case is decided. The effect of which is worst if the software happens to be a Line of Business (LOB) software for the business.

Contrast the above with proprietary software where there's a single point of accountability, protection against indemnification and litigation, support included, minimal integration problems, reasonable security and always maintained, organized and up-to-date all in one upfront price.

Back to Big Software Corporations and Open Source Software
With the tweaking of the GPL license (copyleft) into the BSD and Apache licenses, and naturally so, to accommodate business and their natural purpose of making money, Open Source became the best and the most clever exploitation gimmick for big software corporations.

That could explain why Microsoft shifted to using Open Source on a few projects. I say the best exploitation gimmick because it serves big software players the following major purposes:

1. Open Source software expedites software development to a much faster rate than closed software development. This is because it uses the power of the crowd, the many developers to slave for free and use their creative genius so the software company can eventually charge/benefit from the product as is allowed by the modified BSD and Apache license.

2. The best of both worlds. Software companies can now use the fast development speed by the slaving of the many and also include their own proprietary important pieces closed in the same package. What more can a software company ask for?

3. Free or cheap labor, the dream of every corporation. Before, exploitation was limited to physical goods, now, thanks to Open Source software, it is extended to software. Big software companies used to value and pay premiums for a lot of talent. Now, they get a lot of project work done for free by the many slaving developers in the community and hire and pay for fewer talent. Brilliant!

Open Source Gives Monopoly to Big Software Vendors and Eliminates Small Ones
As we notice, the major Open Source projects find their way to the laps of major software corporations. Examples are Red Hat Linux, MySQL. Corporations, rightfully so, always find a way to make money. That's their natural purpose. But in this case, they're collecting the fruits of the hard work of the many poor developers who contributed to the innovation of the project under the guise of Open Source and "freedom" etc...

This puts big companies light years ahead of the small software companies who don't have the same marketing power to get their projects publicized. They will lag in innovation because they won't attract the huge developer base to work on their little Open Source projects for free.

Open Source Software Undermines the Software Development Profession
Before, what was sought after was local talent. Now, with Open Source software, it has become a commodity. This happened because of the availability of Open Source software offered for free by some poor developer somewhere in the world. Therefore, the value of a software engineer in general is diminishing. Standards and certifications have to be put in place. For example, software engineers should be licensed for different classification levels of software development engagements. I see that as the next natural path in the software industry to neutralize this downgrading effect of Open Source software.

Open Source Software Puts Local Economies At an Unfair Advantage With Countries of Large Idle Populations
Winning by sheer numbers. A professional developer is a developer who gains an income and a living out of his or her skill. However, if you are a developer with no job with another million like you in a poor country then your profile perfectly matches that of an Open Source developer who has a lot of spare time to undercut other paid developers and offer free "open" code.

Therefore, countries with large populations of idle developers will gain the advantage and put all the local talent out of work by offering free code for the taking. It is simple economics. Of course, large corporations don't mind free products and labor as globalization and Free-Trade got them used to.

As a matter of fact, the need arose to morph an Open Source license out of the original restrictive GPL license just for exploitation. It is called the BSD or the Apache license. These licenses allow for including proprietary software side by side with Open Source software.

By promoting and using Open Source, developers are shooting themselves in the foot and are the biggest losers in the Open Source game.

Software is becoming more important in our lives and taking on more critical role. It is time for a standard Software Developer License to develop software to stop the madness.

Software Bureaucracy: Layering Optimizes Downward, Not Upward Collaboration

2009-10-11T14:38:00.012-06:00

The Layering architecture style is one way to structure an application and separate concerns into a hierarchy of layers. The question is then: are layers efficient? aren't we adding more obstacles and making things more complex? What gives?

It All Started With Delegation
Layering is just organized delegation. Delegation is good. Before Layering, there was delegation. To enable a component to do more, it delegated the task to another component to be executed. That's one layer created already. Everything was fine with one layer (flat structure) until the tasks in that first layer became more in number or more complex and it became more efficient to delegate to yet another layer.

It is also important to note that a delegatee should be usable by any delegator. It should not be bound to one delegator. This promotes reusability, scalability and maintainability.

As you can see, each layer is created out of a need to optimize for the layer on top. There's nothing wrong with that. If we don't layer, and go with a flat structure instead, the top layer will have to communicate with all, decreasing its efficiency and defocusing it from its main responsibility, as a result, leading to ineffectiveness. Total failure.

Lower Layers Should Not Call Upward Ones (Software Bureaucracy)
Just like a delegatee should not be bound to one delegator, a layer should not know or be bound to one specific layer above it, or depend on the layers above to do its job.

As you have observed above, the whole strategy behind the layering architecture style it so focus the top layers on their main responsibility to achieve more and better results. The key to remember here is that layering was not fabricated to help the bottom layers.

What happens when the lower layers initiate the communication going upwards to each layer? Inefficiency, decreased performance and low maintainability. Because now, lower layers have to keep track and know each top layer in order to initiate communication with them and to do their job. This adds cyclic dependencies and chaos in general. Usually, these kind of software applications are not extensible and provide a very low return on investment.

When this mis-application of layering is done in the business world in corporate hierarchies, it is called bureaucracy, and that's not a good thing. I chose the term Software Bureaucracy as a term for this condition.

Because of this misunderstanding of the layering architecture some people dismiss the whole style altogether.

Then How to Communicate Upwards?
Communicating upwards is best done without dependency. Dependency on upper layers is at the core of the problem. The best way to implement that is through the publish-subscribe pattern or the Observer pattern. If any component is interested in the lower layers' work, then just subscribe to their events. This achieves the best of both worlds. Top layers are now in-the-know and lower layers do not have to depend on communication with the top layers. They just fire and forget. No obstacles in their way.

This architecture can be extended to the business world to benefit from. Keep the corporate hierarchy optimized but empower the lower levels and employees to make their own decisions without the need to depend on top managers. To keep top managers informed, employees can update one central corporate system about their current issues and status.

Who said that business cannot benefit for software architecture :)

Executable Architecture: Design Must Be Incremental

2009-10-06T08:47:00.014-06:00

Incremental Design vs Complete Upfront Design
No matter how genius humans are, in my view, it is impossible to have a complete design that works perfectly from the first time when innovating a new software or inventing anything for that matter.

We hear the emphasis in real development situations being on implementation to be incremental but we rarely see incremental design being stressed out.

Increments must always be created twice: once in design and once in implementation. Why design? Design simply means think. Think, then execute. It is as simple as that. On one extreme, some developers got it backwards. They jump straight into coding first, then they think about it, afterwards, if they even do that. On the other extreme end, you got the ones who think about the total design first, bordering analysis-paralysis, then totally execute. The correct way is usually in the middle, neither extremes. Think (a.k.a. design) a little, execute a little, then think a little, then execute a little. From my personal experience and observation, this seems to be the optimal approach.

An analogy would be the journey of discovery. It is like entering a dark cave with a flashlight. You can only see (design) a few feet ahead. You don't know all the possible shortcuts or dangers that lie ahead. Every time you walk forward (implement) you might discover new crevices or nooks that you weren't aware of or dangers lurking to avoid.

Similarly, for new product ideas to succeed, the development approach is always a cycle or an iteration: design a little, implement a little, get feedback. Then, take that feedback, add on top of the first design another little incremental design, implement, feedback. Keep repeating until you're satisfied with the results and you achieve the complete work flow or requirement.

Once we have a complete product or we're in post-production, now we can manufacture or "assemble" as many pieces of this design as we want. As you can notice, unlike design, assembly can be planned in one big shot. This is because the design has already been proven to work and we are past the discovery phase.

What is executable architecture?
Having established the above, it follows that since architecture is a design activity, there's no such thing as an initial complete architecture on paper, then complete implementation. This is old style. It doesn't work. Avoid it like the plague. This was where the architect sat on a high pedestal and all he or she did was draw diagrams and hand them down for implementation. This perception of an architect is archaic.

The realistic and most effective way of creating a robust software product is to start with an architecture increment then implement that increment and then repeat as mentioned above. Start with a core architecture that addresses the significant use case scenario. Stop. Implement it. Get the feedback, then add on top of it. To distinguish this architecture from the paper-architecture I like to give it the term Executable Architecture. It can be executed and the results can be observed.

Software Architects: Keep your development skills sharp. The architecture is the actual code not the paper. The main benefit of paper or UML diagrams is to communicate, document and understand the architecture.

What Is The Difference Between A Layer & A Tier

2009-09-06T15:20:00.001-06:00

When talking about software architecture, sometimes we hear the use of software layers and tiers used interchangeably. Can these terms really be used interchangeably? In other words, is a tier the same as a layer?

Architectural Layers
Layering, a structural architectural pattern, is the act of creating layers where each layer contains components that are all on the same level of abstraction. It is a core universal and logical design activity. The bottom layers are more concrete then the top layers. The higher you go in the layer stack, the higher the level of abstraction.

The higher in the layer stack, the less components are usually present. So if we were to visualize this layering logical structure, it will look like a pyramid or a triangle more than a vertical pipe.

Another important principle in the layering architectural pattern is that the lower layers are agnostic to who is calling them. They are highly reusable.

The Layers architectural pattern permeates everything in nature and anything that is well-built by humans. You always have the concrete workers at the bottom who do the detailed work, then you have a layer of controllers above them, then another layer of high level controllers and they repeat until you reach the top of the pyramid where a single responsibility or purpose is started. Optimally, this top purpose should be a use case or a business process that rolls down the layers for execution.

Architectural Tiers
Organizing a system into tiers is an architectural deployment pattern. This means that it has to do with how the architect physically organizes the system by assigning components to physical machines.

Why do we organize a system into separate physical tiers? Well, for many reasons. The main reason is scalability where we can scale out by adding more machines to get better performance for heavy loads. Another reason is security where sometimes we want to keep a certain layer on the perimeter and keep other layers secure within the internal network. Yet another reason is reusability where we might want to reuse the functionality of a certain layer from many different places to save on cost and keep the system consistent and easier to maintain/control.

Usually, layers are mapped to tiers. For example, we can assign the presentation layer to a perimeter machine or server that runs a web server while keeping the business layer secure inside and also usable by internal rich clients.

As you can see, tiers and layers are totally different. Layers are logical and a structural architectural pattern while Tiers are a physical and a deployment architectural pattern. However, they work nicely together.

Difference Between Abstraction & Aggregation

2009-08-22T18:56:00.006-06:00

It is easy to confuse abstraction and aggregation. Briefly, I will shed some light on the difference between the two in this article because they are core concepts that must be grasped by any serious architect.

Abstraction
In one of my previous articles I highlighted that the art of abstraction is the bread-and-butter for an architect. I also mentioned that the application of abstraction is behind most, if not all the progress that humanity has made and the enabler of any progress that the humanity will make.

As a recap, abstraction is about generalizing entities, especially complex ones and the ones involving many processes or parts, into one single entity. It is very important to note that this entity must have laser-focused, single responsibility or else this is not abstraction. This generalization accomplishes two major benefits and enablers of human progress. They are paramount for innovation:

1- Simplification. When things are made simpler, we can do more and create more complex innovations. I attributed this mainly to the limitation of how many entities can the human brains grasp at the same time. We must condense in order to progress.

2- Reuse. When something is generalized, it can apply to many more situations and that in turn, will boost progress.

As examples, chances are, I can bet you that anything you think about is an abstraction. A programming function is an abstraction. A class type is an abstraction. A component is an abstraction. Pure Fabrication in GRASP (the activity of creating a new class that doesn't map to a real-life object) is an abstraction activity. The SRP (Single Responsibility Principle) is about abstraction. A SOA service is an abstraction. A framework is an abstraction. An operating system is an abstraction. A car is an abstraction. A post office is an abstraction etc... I hope you catch the drift.

The main thing to note about these abstraction is that they all perform a single responsibility that they accomplish. They are not simply a group of items.

Aggregation
Aggregation on the other hand is one type of abstraction. A subclass of abstraction in that it shares the benefits or purpose of abstraction but with a specific way.

Aggregation is a method of grouping the individual entities under one access point. It doesn't involve responsibility of performing the actual action. Like a catalyst or a middle man. So one major characteristic of this single access point is that it doesn't do any of the functions of the individual parts it is aggregating. Otherwise, it is not aggregation, it is abstraction. Its pure function is to group. That is its meta responsibility. Not to get involved in any of the details (please don't confuse that with delegation).

Examples of aggregation would be a the Facade pattern, a service layer, and a Message Bus. In business, one can see Unions aggregating employees, iTunes as an aggregator of music (iTunes simply groups all the individual artists but is not in music production itself), Google is an aggregator (when you search for electronics, Google doesn't do electronics but they group them for you). Aggregation seems to be the big opportunity in business ventures these days. The name of the game.

With this definition, one can see that aggregation shares the same benefits of abstraction, i.e., simplification and reuse but not its broadness. Aggregation has an already pre-defined single responsibility and doesn't get involved in the function of that which is aggregating, while abstraction is generic, a supertype if you will that most of the time is the function and carries the responsibility itself.

Standarizing Architectural Core Quality Attributes

2009-08-14T00:03:00.010-06:00

To differentiate a "good" architecture from a "not so good" one there are a few common quality attributes across the board, regardless of the specific business application. Once we have a common standard, it will be easy to have a basic rating of any software architecture.

When we analyze the quality attributes such a performance, availability, scalability, testability, supportability, usability etc... we notice that we can classify any of them in two broad groups. The dividing line between those two groups is runtime. To simplify, there are pre-runtime quality attributes and post-runtime quality attributes. We can call the pre-runtime ones design-time qualities and the post-runtime simply run-time qualities. For example, scalability is a run-time quality attribute but reusability is a design-time quality attribute. Here are the two groups:

Architectural Run-Time Quality Attributes

1. Availability
2. Interoperability
3. Monitorability
4. Performance
5. Reliability
6. Scalability
7. Security

Architectural Design-Time Quality Attributes

1. Conceptual Integrity
2. Configurability
3. Extensibility
4. Reusability
5. Usability
6. Supportability a.k.a. Readability and Fixability
7. Testability

Upon closer examination we can observe that the run-time qualities vary more often depending on the business application whereas the design-time qualities are standard across the board. For example, in the run-time qualities group, not every system needs to be scalable or highly secure. It depends on the business application. Also, run-time qualities can compete with each other such as performance and security.

On the other hand, all design-time qualities for nearly all systems regardless of their business application can be optimized without negative implications. For example, it is highly beneficial for any system to have excellent usability, conceptual integrity, configurability, extensibility, supportability... the more the merrier. Also, we notice that none of the attributes in the design-time group compete with each other. On the contrary, they support each other.

In conclusion, software architects in general should strive to optimize design-time quality attributes regardless of the business application. Optimizing design-time quality attributes must be a best-practice for software architecture. Design-time quality attributes can be called the architectural "core" attributes.

The TAO of Software Development: Implement Horizontally & Deliver Vertically

2009-08-06T17:40:00.006-06:00

Many development teams struggle with using a consistent approach in delivering quality software. By delivery, I mean the iterative build drop to the customer up to the final release.

Should the task assignment to a developer be use case based or component/feature based? Should we focus our build drops around use cases or components? Should we just let it be in a lassais-faire environment where delivery is haphazard and to each his own?

To know the best delivery method we have to go back to the basics. To a user, the internal components of the software are a black box. A software is developed to satisfy certain human goals and actions, not just for the bits of it.

So a characteristic of the correct method to follow is to be centered around the elementary business process that the user is trying to accomplish. Components do not exist in vacuum.

Vertical Delivery
A user goal or use case is a vertical execution path that spans multiple components. Therefore, to be more client-centric, our deliverable must be organized, tested and delivered to execute use cases. The benefits of this vertical delivery approach include:

1- Early delivery of functional and observable code to the client
2- Early delivery of value for the customer's investment. The "get-hit-by-a-bus" factor is always their concern
3- Effective testability where it is in line with user goals
4- Early discovery of integration issues, since we're always integrating
5- Forcing the creation of a base-line architecture early
6- Forcing the early discovery and resolution of architecturally significant issues

As you can see, the vertical or use case based delivery approach enhances software quality dramatically, mitigates risks and increases the sponsor's return on investment.

Horizontal Implementation
When it comes to assigning development tasks to team members, we should also apply the same reality criteria we used above to arrive at the answer. To start with, software is an ocean of specializations where each one hosts a deep well of knowledge and issues.

Except for small, experimental and throw-away code (.com era comes to mind) projects, it is counter-productive and detriment to quality software to have a team of "do-it-all" developers. Those "do-it-all" developers should be on the way up their career ladders to become Development Leads and Architects.

It is impossible to be an expert at everything. A developer cannot be highly knowledgeable in the intricacies of development technologies such as WCF, Multi-Threading, BizTalk, GUI design, security, compression, web etc... all at the same time.

Plus, the professional maturity level of developers differ. You have the programmer who is a routine coder, then you have the component coder then the application coder etc...

A development dream team is the one that is made up of specialists. A database specialist, a performance specialist, a work flow specialist, a communication specialist, a security specialist etc...

Based on the above reality, the best implementation method that aligns with it is horizontal, not vertical implementation of software. The architect usually determines the needed components and assigns the specialist developer that task (maybe with the help of the development lead). In long standing teams, it is critical to the productivity of a team to have that same developer own that component.

Typically, at the beginning of the software project, the architect builds a matrix of use cases (vertical) vs components (horizontal) needed and adjust in every iteration.

Applying MVC Correctly: Where does it fit?

2009-04-06T17:15:00.015-06:00

When someone talks about Presentation layer patterns, the first pattern that comes to mind is MVC (Model-View-Controller). Although MVC might be well known in its name, however, I found out that it is a little vague in applicability and understanding to a lot of developers.

Who Am I? Where do I belong?
Is MVC and architectural style or just a composite design pattern? MVC is misapplied and misplaced in the application stack. MVC is a full application architecture style and not confined to the Presentation layer as I often see. If you think about it, MVC seems to be an early evolution of the ubiquitous three-layered architecture. The Model is the "data access layer", the Controller is the "business logic layer" and the View is the "presentation layer".

The cleanest application for MVC (Passive) that I've seen thus far and agree with is in Web Applications (ISAPI filters, HTTP handlers, ASP MVC) and not Rich Internet Applications or Rich Clients.

This is because the gist of MVC that makes it different from the Layered Architecture style is the emphasis that the Controller is the interceptor of requests. I think MVC is the perfect fit as a Web Application architctural style because the controller intercepts the call and creates the appropriate web page to be sent back to the client. This is because in Web Applications, and I don't mean here rich internet applications, the request is sent to the server to send back the UI with the data.

Other than that, especially in Rich Clients, I don't recommend the MVC architecture style. It is a misfit. In almost every rich client implementation of MVC that I've seen, it was always misunderstood and confined to the presentaion layer. I could not find one rich client implementation that was the canonical MVC. When I ask the designers about their implementation they would always say this is MVC but with this modification or that twist...

It is In the Interceptor
A lot of ambiguity has surrounded MVC on rich clients that many variants had to evolve such as Active MVC, MVP, Supervising Controller MVP etc... In my opinion, the main reason for that is because in rich clients there's a paradigm shift. Whereas in the canonical MVC, the Controller is the interceptor of user requests, here, in rich clients, it is the view that is the interceptor. A major architectural driver shift which calls for a different style.

In addition, with the advent and stabilization of the more mature Layered and SOA Architecture style, MVC seems redundant, a misnomer or out of place. It only serves the purpose to emphasize that there's an interceptor here and it is the controller. In the absence of a Service layer, some people find this useful.

Presentation Layer Architectural Pattern
Instead of trying to force MVC on rich client application and rich internet applications, in my next blog, I will put together an architectural pattern for the Presentation layer that is optimized for the following quality attributes in this order:

1- Maintainability
2- Reusability
3- Testability
4- Usability

ABA Principle of Architecture: Always Be Abstracting

2009-03-12T09:35:00.003-06:00

Every discipline has a dominating activity or a guiding principle that dominates its practice. For example, if we take a snapshot of a Test Engineer's activity, we'll find that he or she is always trying to "break" the software. So we can say that a Test Engineer is always trying to break things as a general principle. Another example is a software programmer who's guiding principle is to always be looking for efficiency; how to use less memory, how to speed things up etc...

An architect's guiding principle or modus operandi is to Always Be Abstracting. To remember this, I use the three letter words ABA to package this principle in a portable way.

Why Abstraction?
The main benefit of abstraction is that it simplifies complex systems and as a result of this simplification, the system can do more. I can safely say that all our advances in technology throughout history are a simple act of adding more layers of abstraction. Thus, we're able to do more and more. Take for example the OS, it abstracts the underlying hardware system so applications can do more. Then came the software libraries which added another layer of abstraction and allowed us to develop more powerful software faster. Then came the Microsoft .NET platform which added yet another layer of abstraction so we can deliver even more sophisticated systems.

What is driving abstraction then?
Two physical constraints: The first one and most important one is that the human brains can only handle a few concepts or entities at a time (they say seven). The second driving force is the embedded quality of the human being to keep "building" things, "improving" things and making thing more efficient and effective. That's a human's never ending quest since the cave man. So on one hand we have a limited capacity to hold in our brains a few concepts and on the other hand we have an unlimited flow of "needs". Something has to give: Abstraction is the only solution to allow for progress.

This is true throughout life and other disciplines other than software. For example, when we put a manager we're actually adding a layer of abstraction so we can simplify the functions of an organization and as a result, we can do more complex tasks. In the financial world, an investor is at the top of the food chain and probably in the highest levels of abstraction. On the other hand, probably an engineer or a technician are at the lower levels of abstraction. In the automotive world, a car brake pedal abstracts the details of stopping a car... Even the human brains abstracts functions by filing them in the short-term memory. That way it can do more complex tasks. Examples are riding a bicycle, driving etc... These are all functions that the brains "caches" and abstracts them in order to allow us to do more. Abstractions are all around us. I am sure you can think of a few.

Going back to the software world, we can safely say that a good software system is one where abstractions are well defined throughout. The well known Single Responsibility Principle (SRP) ties very well into that.

If we are always aware of abstraction as a principle, we can always make the software simpler to build and easier to implement more complex functionality and delivery more high quality capabilities. This applies to the full layer stack of a system down to the functions level. From the low granularity entry point service operations that abstract a use case or user goal to the workflow layer, to each of the components, it is a trickling down effect through layers of abstraction. This flow resembles more a pyramid structure then rectangles on top of each other. If you analyse closely, you have few components at the top "controllers" or "orchestrators" of the high level concepts and delegating to the relatively many "primitive" components that don't know much about the big picture upstairs but good at accomplishing a certain relatively lower level task.

So the key to human advancement in general as I demonstrated briefly is continuous abstraction. In the software world, which is becoming the foundation of most other worlds, abstraction is the job of a software architect. No pressure though.

Longevity As The Main Architectural Goal

2009-02-27T21:30:00.003-07:00

Architecture focuses on quality for the major reason of having the software or
building or system last for a very long time. I can argue that if a software doesn't need to last, then architecture is not needed. So for throw-away software projects that are only planned to last few months to a year, an elaborate architecture is not needed and is a waste of time. But it is rare to find software written intentionally to be thrown away.

Quality makes software last for a long time, maximizing the ROI. What gives quality to software design is a good architecture.

One can argue that some systems are meant for one usage or to self-destruct so where's the goal of "longevity" in that? When we say that the purpose of a system is longevity, we don't mean the actual physical atoms of the system to last forever, what's meant is the system design, static and runtime structure to have longevity and stand the test of time. If this design of this one-use system is re-used over and over then longevity was attained. For an example, the car air bag design is used over and over in so many instances.

Software architects should always strive for longevity of a system to maximize ROI for the stakeholders. This is done by optimizing the archictural qualities.

Why Software Projects Seem to Be Late?

2009-01-15T09:23:00.007-07:00

Development Models
To answer this question, we have to get a basic understanding of the "what is it" of the software development process. What kind of a beast is it? The easiest way to understand is to model it. If we look at any development process, we find that roughly it is made up of three phases: the envisioning phase, the design phase and the implementation or construction phase. To simplify our discussion, let us merge the envisioning and design phases into one phase and call it the Inventing Phase. Some people call it discovery, others call it exploration, but let's just call it Inventing Phase. After we invent something, then we send it to the factory or what have you to be constructed, assembled or implemented etc... So we can break any development effort whether it is buildings, cars, spaceships etc... into two clearly distinct phases in development: Inventing Phase and Construction Phase.

The Nature of Software Development
Unlike any other domain, in the world of the software, the Construction Phase is highly efficient and optimized because of the fact that software can be easily "copied" and constructed in no time. Other domains, they cannot simply copy/construct their entities in such a fast way. Therefore, in other industries, the construction phase is highly visible. One doesn't just copy a car or a house. It takes time to construct a house or assemble a car. In software, you just copy or xcopy it, install it to deploy it. So we can safely conclude that Software development leans itself more heavily towards the first phase which is the Inventing Phase.

Comparing Apples To Oranges
The mistake that we make is that we compare Software Development which is mostly an "Inventing Phase" process to the "Construction Phase" of other domains and then we wonder, why do most software projects miss their deadlines. Of course there are other factors involved in the delay and it can definitely be remedied, but we won't be able to do that unless we understand the nature of and classify it in the right category. Only then we'll be able to deliver on time.

We've seen the ubiquitous analogy comparing software development to constructing building and now we can clearly see that this is the wrong analogy. This is because houses are already invented, explored and discovered. All what's left is to construct it in a deterministic way. On the other hand, every software project is unique in its kind and it involves a lot of discovery, exploration and back and forth with the customer to get it right. In short, it is safe to say that every software project is an invention. After it's been invented, then it is "constructed" or deployed many times over or copied to CDs which is a very fast and simple process compared to other domains such as constructing airplanes or buildings.

In summary, software projects seem to miss their deadlines because we are setting the expectation of the customer that developing software is like developing a house. Therefore, we're setting up the wrong analogy for the customer and in turn the wrong expectation because now the customer will compare that to the deterministic process of building houses. In other words, compare the Inventing Phase of software development to the Construction Phase of developing a house or a building.You can see how this could lead to trouble.

Helping Customers and Correcting the "Misunderstanding":
Software Development is a Inventing Process
Maybe we can help our customers understand the nature of software development by using the correct name for the process. Instead of calling it only the abstract "Software Development", we can probably also call it Software Invention or Solution Invention... something like that to make it clear that they are about to embark on an invention project type. Now how to accurately estimate an Invention project type will be the topic of another post. But at least we know what we're dealing with.

Software Development is Architecture-Centric by Design
As a last note and a future candidate for another article, since software projects are design and envisioning intensive, they lean heavily and by nature on a role to keep the conceptual integrity together, to keep the master design coherent, to make sure that the solution meets the high-level requirements (architectural qualities) and stand the test of time. This role is the Architect. The nature of software is architecture-centric by nature. Any other approach as we've witnessed, has the potential to bring trouble to the project, especially for complex, enterprise level projects. As Grady Booch said: "You don't build a sky rise like you build a dog house".

Architectural Analysis: How to Determine Architectural Drivers/Qualities

2009-01-07T12:32:00.002-07:00

The Sea of Architectural Qualities
The main goal of an architecture is to satisfy architectural qualities. Architectural qualities are not limited to a specific set such as scalability, usability, maintainability, reliability etc... They are dictated by the business strategic objectives. Each system has its own set. For example, "Jam-ability" is a quality that the famous Kalashnikov rifle design was optimized for. An architectural quality or driver for a building in Japan where there are lots of earthquakes is a building architecture in Japan could be "Shake-ability" for instance. Software Architects should be open about determining the qualities to design for getting the inspiration for them from the business objectives.

Something Has to Give
An architect cannot simply optimize the architecture to satisfy all system qualities. He or she has to realize that architectural qualities could compete with on another. The choices of which qualities to optimize for should be done carefully to make sure they don't contradict each other. For example, Security and Usability are competing qualities.

Determining Architectural Qualities or Drivers
A starting point for an architect to find the architectural qualities or drivers is by analysing areas of change: external and internal, present and future. Ask yourself, what are the current areas that need flexibility or have variability in the system? What realistic changes the system will endure in the near future? Select those qualities and build your architecture to resolve them and guard against them. This approach will lower the total cost of ownership of the system and increase its longevity.

Elements of a Simple Plan

2009-01-07T12:08:00.001-07:00

Saleh Najar
The content of this article might be changed without notice.

We here about plans everywhere. Project plan, business plan and it seems like it is scary or vague to most people and they shy away from it. If you don't plan, you don't deserve to succeed. Planning and writing your plan shows that you've matured professionally and you are playing on the same level to stakeholders. It enchances communication and your own understanding of the project at hand.

I see the elements of a good plan to be:

1- Purpose: Quick summary of the end goal of the project. This should be based on the vision.
2- Management: Who's in charge and accountable, there qualifications.
3- Scope: A list of tasks starting with a verb that would accomplish the purpose.
4- Specs: The parts that make up the project and their expected qualities.
5- Schedule: When will things happen?
6- Phases: Grouping of the tasks together to simplify management and assignment.
7- Providers: Who will do the what? Humans, machines, companies, specialists etc...
8- Money: How much does it cost and what will it return? At a minimum, this should include an itemized budget, sales and profit forecast.

This can be tabularized and almost each item becomes a column header.

What is Architecture?

2007-07-01T01:01:00.001-06:00

Have you been in an airplane where you looked down and observed the landscape beneath you? From a bird's eye-view, you can tell instantly its order, you can tell right away the quality of a city; of whether it is organized in blocks or whether houses are randomly sprinkled all over. You can even foretell the future growth of the city and its evolution points. You can tell the surface quality whether it is smooth or rough. If you take a closer look at the motion of cars and the synchronization of traffic flow, you can see the harmony (or the lack of) in flow, color, in shape, in space and in time.

All of these major components are defined by what we call: Architecture. Therefore, architects in all disciplines are defining the "look and feel" of the world, therefore, shaping the quality of human experiences. For example, if the achitects in a certain country don't implement traffic lights, they negatively impact the daily experience of its citizens. Another example is the healthcare system is some countries, if it is not well architected, millions of people "suffer" adding to their negative human experience.

Based on the fact that architecture shapes the human life experience and it plays on the "macro" level, the architect's responsibility is very serious indeed.

So in short, architecture has major implications. Architecture defines the order, quality, organization, interaction, relationships and future evolution, general "feel", of the landscape, be it software, cities, buildings or any other system.

In software, architecture is the science of analyzing the system's quality requirements and the art of making the right decisions to satisfy those requirements. It is the first step in the quest for a solution after Problem/Goal/Vision Definition and before an implementation is designed and executed.

Architecture is based on the following four pillars:

1- Analysing alternatives. Having collected the quality requirements, no architectural decision is complete without first analyzing a few alternatives on major factors of the system. The first solution should not be jumped on without at least considering a second solution. Considering alternatives is usually a sign of a senior, well versed architect.

2- Decisions to resolve system-wide quality requirements. The key words here are "system-wide" and "quality requirements". Architecture deals with the big picture of the system and its overall quality. By quality I mean factors like usability, reliability, performance, supportability, scalability and the other "-ilities".

3- Seperation of concerns. Architecture is about breaking down a system into independent modules and defining the interactions between them. Each module deals with one concern only. Examples of concernds are persistence, security etc...

4- Variability and Protected Variation. Architecture is mainly about protecting the system from variability. An architect should strive for having the least trauma on the system from the most probable variability or "change". Variability has two parts: existing flexibility and future evolution. Therefore, the architect must be highly concerned with points of current flexibility and future (realistic) evolution points.

Notes On ClickOnce Deployment

2007-06-22T23:22:00.002-06:00

Overview
Microsoft's ClickOnce is a major leap in deploying client software applications. Unlike previous attempts that only dealt with one part of deployment, ClickOnce was designed to abstract the full installation life cycle; from the API and integrating with .NET in the System.Deployment.Application namespace and Visual Studio, to the development environment (Visual Studio), to the administrator, to handling automatic application updates, to rolling back to previous versions, and finally to repairing and removing the application. ClickOnce is truly the first step cutting-edge technology in client software installation, good job Microsoft.

ClickOnce: A deployment best practice
ClickOnce was well architected to handle client apps deployment in a best-practices way. So if you find yourself in some convoluted situations, it's time to review your deployment strategy instead of trying to do "hacks" in ClickOnce. Chances are, it will fit with ClickOnce best practices. For example, ClickOnce doesn't support setting up Desktop shortcuts because this habit proved to be a cause of more nuisance than help to users. Even research shows that users don't like apps cluttering their desktops with shortcuts. Plus many more disadvantages to this practice and this post is not the place to flesh them out.

About this post
This post is not for for the ones who are already familiar with ClickOne. If you're not familiar with the basics of ClickOnce, then do a quick search on Google and you'll get plenty of info. This post is for the ones who are stuck on some sticky issue with ClickOnce that hasn't been well documented or commented on at the time of this writing. All the answers I present here come from first hand experience with ClickOnce where I've personally tested every statement I made. The below statements are valid for .NET 2.0 framework and Visual Studio 2005.

Once an application is deployed, you must not touch any deployed application files. The deployment files are usually signed. So any attempt to modify them will break the signature and produce an error in installation. If data must be modified such as connection strings then ClickOnce allows for this too. Read more about this point further down.

The "secret weapon" of ClickOnce is its DataDirectory property. The cool thing about local Data directory is that it is maintained across different versions. So user specific settings are preserved when they upgrade to a new version. It is only preserved if the user makes local changes to the data file. If not, then the latest data file for the new version is downloaded. All versions read from the central same local data source. Even if you rollback to a previous version, the most recent local data for the latest version will be used. The way ClickOnce does this is by simply replacing the old DataDirectory path with the new one. In a nutshell, the latest user modified settings are guaranteed to be preserved across versions. No more worrying about how to preserve the local settings of previous versions. How cool is that?

Rolling back to older versions is a breeze. An admin can switch on the fly to any version by simply renaming the _.application to .application in the deployment directory.

How to get the application deployment directory path. One should not need to programmatically know where the application was installed from. If you find yourself needing that, then most probably, you’re approaching your problem from the wrong angle. However, if you still think you do need this URI, the only way to get the location of the deployment share is by using the CurrentDeployment.UpdateLocation member. The others are always null for some reason in this version. Most of the CurrentDeployment members are null. The only ones I found to be working were the DataDirectory and the UpdateLocation members.

Admins can easily modify the deployment settings. An admin can edit any of the deployment settings (all that’s available for the developer from VS 2005) using the MageUI.exe (GUI) or Mage.exe (Command prompt) tool that comes with the SDKs. However, the admin would need to sign the changes with a certificate (any certificate installed on the machine will do) before saving, for the new settings to take effect. For example, the admin can install two editions of the same app to different share names side by side. The cool thing is that a new shortcut will be added pointing to the new edition and grouped under the same company name! This is ideal if one app needs to be in a different environment that the other such as Test and Production and they need to be run side by side.

DataDirectory is the "entry" point to data sources settings. The DataDirectory member gives the location where a data file such as a database or a data file is stored in the local application store (temp cache directory). Use the DataDirectory location to get to the “entry point” required data such as connection strings or paths etc… So for example, if the client app needs to access central data stored in a database, then the natural place to add the connection string is from the application GUI. This is so because usually, the connection string passwords need to be encrypted, so functionality is required and mind as well add this layer of functionality to the existing GUI (most simple cases). Another example (although not recommended except for very simple apps) is if your application needs to update a central data file in some standard location, then by the same token, have this functionality to add the path to this central data file in the application GUI where the user is prompted and can edit it any time from a friendly interface. The app will write this path to the local data file in the app storate data directory and it will be preserved across different versions.

I hope you find the above brief explanations helpful. If you have any comments, corrections, questions or need further explanation, please feel free to leave a comment and I would love to try and help out.