GitHub's Electron platform is pretty sweet! Electron allows developers to write cross platform desktop applications using the web stack of HTML, CSS and JavaScript. If you have experience building web apps, then building desktop apps with Electron is simply an extension of what you already do on the web. You can bring in all your favorite libraries and frameworks like jQuery, Angular, Meteor, etc. without much work. And the learning curve for using Electron's Node APIs to interact with native OS resources is very small. Save for a few places, like menu creation, the APIs  provide a unified interface across the 3 supported platforms (Mac, Windows and Linux). The underlying node.js system handles the platform differences for you.

All in all, the experience of developing Electron apps is sublime. That is until you get to packaging and deployment. While development is easy, deployment is hard.

During development, to run your app you simply run the command npm start or electron app in your project folder. However, in production you cannot expect your users to clone your git / bitbucket repo, install the dependencies and launch the app via shell commands. You need to provide executables native to each platform.

The manual way of doing packaging and deployment as prescribed in the Application Distribution tutorial on GitHub is to:

• For Windows/Linux, 
• create a directory for the target platform (I will call this $ELECTRON_HOME)
• get the electron executable for each target platform and place it in $ELECTRON_HOME
• create the following directory structure under $ELECTRON_HOME:
  

  resources/app/

• place your source code (e.g. package.json, main.js, index.html, etc) inside the app directory
• Rename the electron executable to your application's name
• On Windows, you can edit the app icon using a tool like rcedit or ResEdit
• On Linux, if you are targeting Debian derivatives, you'll need to create a .desktop file to set the icon and then deploy the app and .desktop file using a deploy script.
• For Mac OS X
• get the electron executable Electron.app 
• open up the directory by selecting "Show Package Contents" from the right-click menu and the navigate to Electron.app/Contents/Resources/
• create a directory called app/ inside of Resources/
• place your source code (e.g. package.json, main.js, index.html, etc) inside the app directory
• rename Electron.app to your application's name
• rename the CFBundleDisplayName, CFBundleIdentifier and CFBundleName fields in following files:
• Electron.app/Contents/Info.plist
• Electron.app/Contents/Frameworks/Electron Helper.app/Contents/Info.plist
• place your custom icon (icns) file in the Electron.app/Contents/Resources/app directory and change the value of the CFBundleIconFile field in the Electron.app/Contents/Info.plist file
• optionally: rename the Electron helper app's executable file's name and then the Electron Helper.app itself to your application's name

These steps may not seem like much, but having to do them manually every time you want to do a build for three different platforms can get pretty tedious. And in a continuous delivery environment, this is a none starter. It would be nice if there was a way to automate this process. And there is, it's called gulp-electron.

Gulp-electron is a gulp plugin that helps to automate the creation of electron-based application executables. Once a gulp task is defined, building executables with gulp-electron is as simple as running the command gulp electron. The gulp electron command automates the process of downloading the Electron distributable for each platform specified, packaging your application within the correct directory structure, modifying all plist files as needed, setting up your application's icon, and packaging the whole shebang into zip files per platform.

Here's an example of the definition of a gulp-electron task. This code should be placed in a file called gulpfile.js in the root of your project folder.

'use strict';
/* jshint node:true */

var electron = require('gulp-electron');
var gulp = require('gulp');

var packageJson = require('./src/package.json');

process.NODE_ENV = 'test';

gulp.task('electron', function() {

    gulp.src("")
    .pipe(electron({
        src: './src',
        packageJson: packageJson,
        release: './release',
        cache: './cache',
        version: 'v0.35.2',
        rebuild: false,
        packaging: true,
        asar: false,
        platforms: ['linux-ia32', 'linux-x64', 'win32-ia32', 'darwin-x64'],
        platformResources: {
            darwin: {
                CFBundleDisplayName: packageJson.name,
                CFBundleIdentifier: packageJson.name,
                CFBundleName: packageJson.name,
                CFBundleVersion: packageJson.version,
                icon: './resources/cheatsheet.icns'
            },
            win: {
                "version-string": packageJson.version,
                "file-version": packageJson.version,
                "product-version": packageJson.version,
                "icon": './resources/cheatsheet.ico'
            }
        }
    }))
    .pipe(gulp.dest(""));
});

gulp.task('default', ['electron']);

For a detailed explanation of what each property means; checkout the gulp-electron README.md.

A couple of things to note about gulp-electron as of the time of this writing are:

1. Your application code needs to be in source folder under the project root. The source folder can be called anything (typically 'src'). It lets gulp-electron know what needs to be distributed to
   

   Electron/resources/app/

2.  The cache folder is where the platform specific distributables are downloaded to and the release folder is where your packaged application is created in. These folders should not be checked into version control. If you are using git you'll want to add them to your .gitignore file.
3. Creation of the custom application icon for windows did not work for me. I'm investigating this and I'll update this post when I have some new information. If anyone has had success with it please let me know in the comments.
4. At this time, there isn't support for signing an application for distribution in the Mac App Store. Check out the Mac App Store Submission Guide for details on how to do this.

For further reference on packaging and deployment with gulp-electron, check out my Cheat Sheet project on GitHub.

I needed a way to decipher the delimiter character for a csv file I was processing in Ruby.

In order to move massive amounts of data around between organizations we (at my current job) have found file transfer to be significantly more performant than web service calls. Therefore we often share data with our clients through Secure FTP.  A particular project I was working on required parsing csv files that are provided by clients which could have different formats. So I needed an automated way to tell the csv parser whether to use commas, tabs or pipes in parsing without having to manually examine the files or setup some per-client configuration. That's why I wrote the csv_sniffer ruby gem. The word "sniffer" is meant to indicate that the determination of the properties of the csv file is done heuristically. Plus Python has a csv.Sniffer class that does many of the same things my Ruby csv_sniffer does. So the terminology is already familiar to developers in another dynamic language.

The design goals of csv_sniffer were:

1. To answer the question the fastest, simplest and resource-conscious way possible at the same time maintaining as high of an accuracy standard as more complex methods
2. To be easy to use

To that end, all csv_sniffer methods are static and simply require a single argument which is the file name. As little of the file is read as possible and file handles are always proactively closed the instant they are not needed. The detection algorithms themselves are also designed to be efficient and use pre-determined knowledge to be simple and fast.

Initially csv_sniffer started out with delimiter detection, quote enclosed values detection and quote enclosing character detection. I later added in header detection by porting the Python algorithm for header detection to Ruby.

All the source code, tests and usage examples are available on GitHub under the MIT license so it's free use, modify and redistribute. The gem itself is is published to rubygems.org and can be installed with a simple gem install csv_sniffer.

Clean Code by Robert C. Martin (Uncle Bob) is one of those books on most lists of "books every developer should read". It offers guidance on how to write good code. Good (or clean) code is defined as code that:

• does one thing well
• is simple and direct
• has meaningful names
• has straightforward logic, which makes it hard for bugs to hide
• is full of crisp abstractions and straight forward lines of control
• contains no duplication
• has unit and acceptance tests
• is maintainable
• reads like what you would expect
• looks like its written by someone who cares

The book lays out several principles for writing clean code according to the Uncle Bob school of thought; which he readily admits is not the only right school of thought, but many, myself included, consider it to be an excellent one.

It stands to reason that if all of the Clean Code principles were followed in a green-field application it would leave you with a system that has no technical debt (by Uncle Bob's standards). The following question could then be asked; is it possible to build and maintain a large system that has no debt? My answer to that would be: yes, but at a very high cost. Such a system would probably need to have a heavy dose of both upfront design and agile/successive refinement. In addition, every change would require careful thought and a certain amount of refactoring of the application. In a brown-field application (which is what 90+% of developers work on 90+% of the time), the cost is even higher.

Quick Sidebar: According to a quick Google search there are only 5 countries in the world with no debt; Brunei, Liechtenstein, Palau, Nieu, And Macau Of China. Most other countries have to carry some debt to achieve their goals.

The thought behind this review is to perform a cost benefit analysis of some of the clean code principles: What are the costs of implementing a principle and is the benefit worth the cost to the developer making the decision? This process of cost benefit analysis is done (either consciously or subconsciously) by every developer who is sitting at a keyboard and making a design decision. This post is my version of that thought process, done from my perspective. I've done some grouping of the Clean Code principles based on my perceived cost of implementing a principle and the benefits gained from that implementation.

Low Cost, Low/Medium Benefit

These principles are so low cost in my opinion that even if they provided just a little benefit, everyone should almost always adhere to them. Following them would make your code easier to read and be understood which has its own set of benefits.

• Use meaningful names - Meaningful names should reveal intent, be human pronounceable, be searchable, use one word per concept consistently throughout the code base, and should draw from the solution and problem domain. Classes should have noun or noun phrase names and methods should have verb or verb phrase names. Meaningful names improve the readability of the code for cheap.

• Formatting and spacing - The top parts of a source file should provide the high-level concepts and algorithms. Detail should increase as we move downward, until at the end we find the lowest level functions and details in the source file. In a block of code, blank lines should separate concepts. 

Low Cost, High Benefit

Not only are these principles low cost but they also provide great benefit in terms of the maintainability and robustness of your code. Unsurprisingly, these principles are very familiar to modern developers.

• Don't Repeat Yourself - As Uncle Bob says "duplication may be the root of all evil in software". The good news is that duplication is often easy to identify and refactor out into its own method.
• Have no side effects - Side effects break the promise made by the function definition; the function promises to do one thing but it also does other hidden things. In this sense, side effects break the Single Responsibility Principle. Avoiding side effects affects not only improves the maintainability of your functions but also makes them easy to parallelize.

Medium Cost, High Benefits

These are the hallmarks of a clean coder. They illustrate how much love and thought you put into your code, and pay off in readable, extensible, maintainable code with clear lines of control. These principles may take some time to implement but produce big time savings when maintaining and troubleshooting the code in the future.

• Keep functions small, very small. Functions should be transparently obvious and tell a story. They should do only one thing and contain only one level of abstraction. They should be meaningfully named and should have as few arguments as possible. Large numbers of arguments take up conceptual power and make testing more difficult.
• Use meaningful exceptions with context - Error handling should be separated from the business logic so as not to obscure the business logic. In addition, exceptions thrown should provide enough context to determine the source, location and type of failure. Nulls should never be passed into a function or returned from a function.

High Cost, High Benefit

These are principles that significantly alter the way you do development. When done right these principles can be highly beneficial to the reliability and maintainability of a code base. However, implementing these in a greenfield application can be tasking, and is even more difficult in a brownfield application. Developers that aren't convinced of the benefits may sometimes choose to carry some technical debt in this area. Although I'd advise that in projects that value the benefits these principles bring, you should try to have as little debt in this area as possible.

• Test Driven Development - The 3 laws of TDD are: 1. You may not write production code until you have written a failing unit test. 2. You may not write more of a unit test than is sufficient to fail, and not compiling is failing. 3. You may not write more production code than is sufficient to pass the currently failing test. Tests should be fast, independent, and repeatable in any environment.

Medium/High Cost, Low/Medium Benefit

In my experience I'd classify these as nice to have. They provide a slight to moderate benefit but require good amounts of refactoring to achieve so technical debt in these areas isn't highly detrimental.

• Clean Tests - In my experience, unit tests for internal projects rarely gets read, they just get run, so readability isn't a big factor. However, if you are writing clean tests the same rules that make all code readable apply; clarity, simplicity, and density of expression. There should be no duplication and only one concept per test.
  
• Small Classes - In some respects, small classes are another manifestation of the single responsibility principle. The idea is that a class should have only one reason to change and should be highly cohesive. In other respects, small classes are about the organization of code to make things neater and easier to find. In practice however, it may be challenging to consistently have small classes throughout a code base because it requires a lot of refactoring to achieve.

Conclusion

I concur with probably every other person who has read Clean Code; its one of the best books ever written on the craft of software development. All that's left now is finding ways to incorporate these principles into your daily development life cycle, and get to writing clean code. Or at least cleanish code.

I gave an Introduction to Graph Databases presentation at the Chattanooga Developer Meetup on July 30th, 2015.

Warning: I'm not a very good speaker so watch at your own risk! ;-)


The slides can be accessed here.

The majority of devs work in teams regardless of whether its a small shop, big shop, open source project or even single developer shops. Teams are typically more productive than a single developer working alone, and there are some legendary teams out there like the WhatsApp team that supported 450 million users, on multiple platforms, with only 32 engineers. This is why a fundamental job requirement in most job postings is the ability to work well in teams.

If you think back to your experience working in teams you may have come across one or two of these types of teammates? The first guy is the teammate that the other team members cringe when they have to interact with him/her. The teammate that causes mild anxiety whenever they check-in/commit some code. The one that the others feel is dragging the team down or cannot fully be trusted. On the other hand is the great teammate. The one other team members to bounce ideas of and brainstorm with. The one whom everyone respects and who has a great reputation. The guy that raises the productivity and harmony of the entire team.

Here are a couple of things to get you on your way to being that great team mate.

You write good unit tests and run them often!

Bad dev teammates don't test their code! They push out broken code to the central repository or into production and wait for someone else to find the bugs.
Unit tests help to verify at the most basic level that the code you wrote is consistent with your intent and behaves appropriately under varying inputs. Good unit tests don't only test the happy path but also exceptional and edge cases. But its not good enough to write unit tests, they also have to be run. Unit tests that don't get run are just as useful as not having unit tests at all. In teams that value unit testing, bad dev teammates don't write/run the tests and reduce the code coverage.

Side bar: In writing unit tests, care should be taken to ensure that the tests are fast. When tests run slow, you won't want to run them frequently.

You communicate any and all information that could be beneficial to your teammates.

Make a breaking change to an API? Found a new trick/tool that could be useful? Found a bug that affects others? Don't think you can make that deadline? Changed the database password? Created a new web service or shared library? Communicate that. Send out a message on the dev channel of whatever collaboration tool you use or to the team email inbox or wiki. Make a habit of broadcasting information you think may be remotely useful to someone else on the team. Information exchange is the currency by which a team functions. Individuals who frequently communicate relevant information capture the ear and attention of audiences.
Another important form of communication is documentation. Users of your library, class, app, script or service need to know how to use your product. And your fellow developers need to understand the problem you are trying to solve in order to be able to support the product when you are unavailable. Documentation is part of being a good citizen.

You know your strengths and weaknesses and be honest with your team about what they are

This helps with expectations. If you pass yourself up as great at javascript but end up writing crappy javascript code you break the trust of your teammates and invite greater criticism of your work. However, if you are honest with your javascript skills and ask for help or a peer review, your teammates will respect your effort and will jump in to help. You'll get some learning out of the experience and would have fostered an atmosphere of cooperation.
Transparency about your strengths and weaknesses also aid the team in delegating tasks to everyone's strengths and allows the team to function at peak efficiency.

You write readable / clean code

We spend 10 times more time reading code than writing it; therefore code should be written with readability in mind. How many times have you wanted to punch the developer who wrote the massive mess of spaghetti code you are now having to maintain / refactor. How about the dev who wanted to show how smart he/she was so they implemented a really convoluted architecture or distilled a 20 line block of code into 1 unreadable line.
Clean code is simple and direct^[2]. It's expressive, doesn't obscure the designer's intent but rather is full of crisp abstractions and straightforward lines of control^[2]. It has meaningful names, minimal dependencies, which are explicitly defined, and provides a clear and minimal API^[3]. The logic should be straightforward to make it hard for bugs to hide, and performance close to optimal so as not to tempt people to make the code messy with unprincipled optimizations^[1]. Clean code can be read, and enhanced by a developer other than its original author^[3].

You listen to the opinions of your teammates (even the ones you think you are smarter than)

I've had team leaders before who didn't listen to my opinions on architectures, designs or ideas. Do you know what happened with that relationship? I stopped giving them my opinion! There aren't too many things that kill communication faster than deaf ears. There's a difference between hearing and listening. Hearing is passive and takes no effort. Listening is active and implies leaning in. Even if the other person's pitch may be flawed, listen to it without deriding them. If necessary, show them where you think the shortcomings in their arguments are. Think about the factors behind their proposed solution. Take time out to prove things. Don't be in-flexible or unwavering.
In the same vein, be humble, give others credit when its due, admit it when you are wrong or made a mistake. No one likes an arrogant prick. 

You take time out to help your teammates with their questions, issues, or bugs

There is a tendency for some of us devs to get really focused on our tasks and not be willing to help others out with their questions, ideas or bugs. We often do this because distraction is a top killer of productivity. However, we need to understand that while your productivity is important, of equal importance is the productivity of the team! And if a teammate cannot move forward because of an issue, question or bug that they need your help with, you may get your stuff done but the team suffers as a whole. Stopping what you are doing to help out not only unclogs the gears so that the engine that is the team can start functioning efficiently again, it also earns you appreciation, respect, beers and IOUs.
When you concern yourself with what others are working on, it weaves you deeper into the fabric of the operation of the team and makes you more indispensable.

You solve pain points on the team

Is there a compiler warning that bugs the crap out of the team but no one makes the little change to make it go away? A small improvement to the painful deployment process to make it a little less painful? A small tool or library that needs to be written or introduced in order to solve a common problem?  A script that can automate an oft repeated manual process? An office printer that needs to be taken out into a field and beaten to death with a baseball bat?
What's great about stepping up and making the little extra effort it takes to alleviate these pain points is that you are not only solving a problem for the team, you are also solving the problem for yourself.
Sometimes the solution is not something you have to do, but something you have to research and find out. When you find that "greatest regex cheat-sheet" or "git reset tutorial", be sure to share it with the team if its a common pain point.

Lastly, Don't be lazy

Laziness leads to that unclean, untested, untestable, unmodularized code that causes maintenance nightmares for your teammates. Laziness also sends the message that you are unreliable and untrustworthy as a teammate. It damages your reputation. Hardwork on the other hand is infectious. It causes the rest of your teammates to raise their game. It earns people's respect


Have more advice for how to be a great developer teammate from your experience? Leave them in the comments.


[1] Bjarne Stroustrup. Clean Code (page 7) [2] Grady Booch. Clean Code (page 8) [3] Dave Thomas. Clean Code (page 9)

Run the following commands to download and import neotechnology's public gpg key into the rpm keyring

cd /tmp
wget http://debian.neo4j.org/neotechnology.gpg.key
sudo rpm --import neotechnology.gpg.key

Create the neo4j repo file using you editor of choice and add it to the yum repos definition directory. Add the following lines to the neo4j.repo file

[neo4j]
name=Neo4j Yum Repo
baseurl=http://yum.neo4j.org
enabled=1
gpgcheck=1

nano neo4j.repo
sudo cp neo4j.repo /etc/yum.repos.d/

Finally run the command to install neo4j

sudo yum install neo4j

Here are some things to note with the install:

• - An java jdk (openjdk) is installed and configured for you even if you already have one on the server.
• - The neo4j binaries are installed to /usr/share/neo4j
• - The neo4j properties files are installed to /etc/neo4j
• - The default/initial database location is /var/lib/neo4j
• - Neo4j is installed as a service in the stopped state. You can start it by running sudo service neo4j start. By default, the neo4j instance is only accessible via http://localhost:7474/. If you want to enable outside access, you need to uncomment the line in the neo4j-server.properties file: #org.neo4j.server.webserver.address=0.0.0.0 and restart the server.

To most developers, the word "concurrency" automatically implies "multithreading" or "parallel programming". And while multithreading is a form of concurrency, it is not the only form of concurrency. Concurrency means doing more than one thing at a time. While multithreading is a form of concurrency that uses multiple threads of execution, its not the only form of concurrency. Other forms of concurrency are asynchronous programming and reactive programming. This article is intended to highlight the different forms of concurrency and when/how to introduce them in the right places in your application.

Parallel Programming

Parallel programming takes a computation task and breaks it up into smaller tasks/chunks that get executed in parallel using multiple threads. Parallel programming aims to take advantage of multiple CPU cores in modern machines. For computations that make use of parallel programming, CPU throughput is increased which should translate to increased performance. Take the example of a news aggregator application which pulls down articles as HTML pages and is then tasked with parsing each article. To improve the performance of the app, it would be prudent to perform the parsing of the collection of articles using parallel programming. Here are a couple of factors that indicate that a task would benefit from being parallelized;

  - The task to be parallelized should be a CPU intensive/bound task. If the computation task is I/O bound such as making a long running web service call or database query, then you should be using a different form of concurrency instead; asynchronous programming. Even if you have multiple I/O bound operations that you want to execute in parallel, async programming is still a better fit.

  - The chunked tasks should be independent of each other. There should be minimal (preferrably zero) shared state between the tasks and the execution of each task should produce no side effects. This model of computation is characteristic of functional programming and is why the functional style is often used for high performance applications/servers. The chunked task code should be as functional as possible. Shared state would require synchronization, which reduces the parallelism and efficiency of the application. Side effects are also problematic as tasks can be run in different threads causing unpredictability.

  - The chunked tasks should be relatively long-running enough to mitigate the performance overhead of task division and thread provisioning or scheduling.

Sometimes the parent thread is required to wait for the completion of the sub-tasks, most modern languages these days have higher level abstractions (than Thread.join) to make this easier. In Java, the CountDownLatch class is great for this. In C#, this functionality is built into the Task class via the Wait* methods. Results from sub-tasks, if any, should be added to a thread-safe collection. These collections are optimized for multithreaded access and are often more efficient than synchronizing access to a non-thread-safe collection using a mutex / lock.

Asynchronous Programming

Asynchronous Programming is a form of concurrency in which "callbacks" or "futures" are used to respond to some operation that will complete in the future thereby freeing up an application to do other work. A "future" is a type/interface that represents the operation that will complete in the future. The model for asynchronous programming typically follows the following pattern, some operation is started that will complete at some later time. While waiting for the operation to complete, the current process is not blocked from doing other work. When the operation is completed, it notifies its "future" or invokes its callback method to let the process know that the operation has been completed.

Async programming avoids creating unnecessary threads and therefore avoids the cost of thread provisioning and context switching. Instead of creating a separate thread to execute the "future operation", async programming allows developers to both continue doing other work and respond to the "future operation" when it completes using a single thread. Java, C#, Python, Javascript and many other high level programming languages provide abstractions to make it easier to do async programming. These obviously won't be covered here but we should note what makes an operation a good candidate for async programming as opposed to parallel programming.

  - Async tasks should be I/O bound. Because CPU rates are orders of magnitude faster than disk or network rates, processes that synchronously perform I/O bound tasks (such as data transfers from a database or a remote server in the form of database or web calls) typically spend a lot of time in a blocked state, waiting for the transfer to complete. Performing such tasks asynchronously allows the CPU to continue executing other instructions for the process while waiting for the transfer to complete.

  - There should be some other work to do. If there are no other tasks/work that can be done while waiting for data transfers to complete then there is no point in running a task asynchronously. In some cases, the "other work to do" similarly involve data transfers and could also be run async.

  - Concurrent tasks that use async programming should be independent from one another. Any dependencies, such as one task using the results of another, leads to synchronous programming.

In the fictitious news aggregator app we introduced earlier, async programming should be used for the task of downloading the HTML articles concurrently. This job meets our criteria for async candidacy as each download task is I/O bound and independent of the other. This is an example of client-side async programming. Another very common client-side usage of async programming is in user interfaces. The non-blocking property of async programming allows the UI to stay responsive while some form of data transfer occurs in the background.

Server side async programming has gained popularity with the introduction of the Twisted and Node.js (or IO.js) frameworks. For certain applications async web servers scale better than multithreaded servers by foregoing the performance penalty of context switching and thread provisioning, in favor of using a single thread and servicing requests in the time in which other requests are asynchronously waiting on I/O. For an excellent explanation of server-side asynchronous programming, see the following link [http://cs.brown.edu/courses/cs168/f12/handouts/async.pdf].

Other forms of concurrency

I have no experience with Reactive Programming (Rx) so I won't spend time discussing it here. Reactive programming is programming with asynchronous data streams. It allows you to treat a stream of events like a stream of data that can be observed and subsequently manipulated, filtered and transformed. For a quick, Java-centric guide to Rx, check out this blog post (https://www.bignerdranch.com/blog/what-is-functional-reactive-programming/). For a more expressive (read: longer), C#-influenced guide to Rx, check out this other blog post (https://gist.github.com/staltz/868e7e9bc2a7b8c1f754).

It seems like every time I turn around I find someone else describing their product, API, architecture or service as beautiful or elegant. I believe the use of these terms in tech was pioneered by Apple Inc. or more specifically the late Steve Jobs. Jobs and Apple have been the defacto gold standard for design since the first iPod was released and have inspired generations to think about design as a first class concept in not only products and services but also APIs and architectures.

One word that is constantly associated with the perception of elegance, cleanliness and beauty is simplicity / simple. Simple design is not an excuse for poor design but rather a concerted effort to directly meet a need as efficiently and effectively as possible and without unnecessary elements. A simple product does exactly what it promises, no more, no less.

Developers and architects often try to over-engineer solutions for several reasons. Over-engineering is typically motivated by a need to;

• code for the future
• build a system that targets multiple uses / users
• try to take every possible solution into consideration
• impress management or other developers.

While none of these are necessarily bad goals, they often lead to products, interfaces and architectures that are bloated, convoluted, unmanageable, slow and ugly. On the other hand products that are singularly focused and complexity free are usually a joy to not only build but also to use. An example is Skype. It doesn't try to be a social network, a file sync service or a general purpose collaboration tool. It simply provides a list of contacts and the option to call, video chat or IM that contact. And while it has other features beyond that, they don't get in the way of the core product.

Some benefits of simple design are:

Simple is easier to debug and fix. Because there are fewer moving parts, there is less that can be broken and when something does break, its easier to find it and fix it. A consequence of this ease is a system that is much cheaper to build and maintian when compared to a complex project. The simplicity of the design or architecture also results in stability. There is less to refactor. As the French writer Antoine de Saint-Exupery said "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away".

Simple is more performant. As a result of booting unnecessary features, abstractions, tasks, frills, etc. from a design the product is made more performant. The less an application does, the fewer CPU cycles, memory and I/O it needs. Applications should get a user from A to B as quickly as possible without putting roadblocks in the user's way.

Simple is more communicable and approachable. This results in a design or product that is easier to pitch to stakeholders and easier to get buy-in on. This enhanced communicability promotes dialogue and feedback and makes it easier for others to champion your product. Approachability makes a product more appealing to users.

To be balanced it should be noted that simplicity sometimes comes at the cost of flexibility. For example, a GUI interface may provide more simplicity than a command-line interface but it may do so at the cost of flexibility. Another instance in which this shows up is in the offering of myriads of options and configurations versus offering only one or two. Flexibility is often a response to complexity. In an effort to handle the messiness of the real world we often turn to creating products that can adapt, accept multiple inputs, have multiple configurations, etc. If a need for flexibility is present, it should always be balanced with the need for simplicity. Here are some thoughts on how to combat complexity and simplify your design:

1. Strive to do only one thing. If you have to do more than one thing, break your product into modules. Microservices architectures and other design patterns are prime examples of this philosophy in action.
2. Communicate clearly. Your application should clearly and succinctly state its purpose and deliver that. Implicit understandings and side effects should be eliminated.
3. Eliminate unnecessary things; steps, frills, options. You Aint Gonna Need It.
4. Design from the user's perspective. This is probably the most important item on the list. Take off your designer / architect hat and think like a user. What would I like to see? How would I like to interact with this system?
5. Always hide complexity unless it is required. A simple design does not have to pretend that complexity does not exist, but it can hide it and make it available upon request by the user via a "more options" button or flag.

Simple designs are clean, elegant and beautiful. They just work.

I'm currently employed at AudiencePoint Inc., the industry leader in Email Send Time Optimization. AudiencePoint Send Time Optimization allows clients determine the optimal time to send a piece of correspondence to each individual on their subscriber list. To do this we need to crunch through A LOT of data. This massive amount of data is stored today in a SQL Server 2012 database running on powerful dedicated hardware. As the data and volume of operations on that data has continued to grow, we have learnt some lessons on how to scale SQL Server to keep up with the heavy demand.

One thing to note is that database operations at AudiencePoint (AP) look different from DB operations at a web based companies like StackOverflow or Amazon.com. Operations at those companies are typically transactional (OLTP) in nature; e.g. every page view requires a certain number of DB reads, new records are inserted into the DB when a question or answer is posted, etc. At AP our DB operations are typically in batches; e.g. 10 million records need to be inserted, queried, sorted, updated in batch fashion. The question becomes how do you speed up and/or scale these operations? Especially when you don't have available to you all the traditional scaling techniques used by OLTP DB applications such as caching and partitioning. Here are some of the things we learnt:

Joins are bad
And by bad, I mean slow. Especially when we are talking about large tables and more than two tables are involved. Joins combine scattered data fragments to form a single table. Therefore they are sensitive to disk seek latencies. They also generate a lot of data which may not fit into memory and have to involve the use of a swap file. Additionally, a join operation can only involve two tables at a time so when more than 2 tables are involved, it must first join the first 2 tables to create an intermediate dataset and then join that data set to the next table. This is actually a denormalization process. Therefore it's worth examining your tables and queries to see if you are overnormalized and denormalize your tables or adjust your queries like we had to do. Indexes can help improve join performance on SELECT statements but comes at a cost to INSERT operations. More on that later.


Don't put it in the DB
The fastest database operation possible is the one you don't do. No matter how fast your query operation is, not having to query for the data at all is faster. Same goes for inserts, updates or deletes. Based on the information we had at the time, we used to stuff a lot of unnecessary data into the DB and then have to query back out those millions of rows of unnecessary data. This is obviously not good for performance. In the OLTP world, unnecessary data operations are avoided by various layers of caching. If that's not available to you, then you need to think critically and work hard to avoid unnecessary DB operations. Does it need to be in the database? Why?

SQL Server performs better on subqueries than joins
Questions about the performance of joins versus the IN clause have been asked several times on StackOverflow. Two such instances can be found here: SQL Join vs. In clause and SQL Join vs. Sub-query (first link has an example). We found that SQL server performs better with sub-queries / the IN clause. This is because the IN clause precludes the necessity of a join and as we now know; Joins are bad!
On StackOverflow the common answer to the debate of joins vs. subqueries is usually "it depends on your DB engine". In our experience SQL Server (and apparently Oracle according to an SO poster) has picked a side.

Multithread your DB operations
Horizontal scaling is the holy grail of DB scaling. This is why data partitioning is so popular. In batch processing though, the tendency is to have a process repeat the same operations over and over again (hence batch). The problem with that is that if you have a SQL Server database sitting on a quad-core or octa-core server and you fire up the profiler to look at what's going on, you'll find that only one core on your SQL Server database is active while the others are just hanging around working on their tans. When we divvied up our work into smaller batches and had multiple threads doing our DB operations we saw all the other cores come to life and our performance increase by orders of magnitude. We had just achieved horizontal scaling but within one box. Snow cones for everyone!

Watch your indexes
Markus Winand explains it better; while indexes really speed up SELECT queries they have the opposite effect on INSERT (as well as UPDATE and DELETE) statements. For an insert operation on a table;

adding a single index is enough to increase the execute time by a factor of a hundred. Each additional index slows the execution down further

That's why indexes should be used deliberately, sparingly and with database usage patterns in mind. At AP we had (and probably still have) some tables where the index size is greater than the actual table size.

and finally...

Don't use SQL Server
... for things its not good at. Gotcha there?! The tendency is to try to shoehorn one solution for every problem to be solved. We've been there, done that and have learned not to do that anymore. For areas where SQL Server isn't a good fit, evaluate other solutions to pair with it. NoSQL, Graph databases, Time series databases, etc. all have their strong points and would perform better at those functions than an RDBMS. As the popular saying goes, "use the right tool for the job".

In today's short-attention-span world its important to make sure that your software product meets certain minimum speed requirements. Not doing so causes your target user to go somewhere else looking for someone who can provide the same service you offer faster and/or more conveniently. If they can find one, you've lost a potential user and your product might as well not exist when it comes to that user. Let's use web browsers as a case study. In 2009, one year after the Chrome browser launched, it had 7.1% usage compared to Firefox's 46.6% performance. Fast forward to today where Chrome's usage is 61.6% compared to Firefox's 23.6%. How did Chrome get such a leg up on FF? While being advertised on the main Google website certainly helped, their calling card was speed. Bottom line; performance matters!

If performance matters so much, we should certainly pay attention to it in the design phase of our software projects and not just leave it as an after thought. This goes against the popular notion that you should only worry about the functionality of your application and leave performance tuning until when it is needed. While it is true that premature optimization is the root of many evils, no amount of post-design performance optimization can make up for poorly designed software. Your performance requirement will differ based on user expectation, business goals and of course the work that is actually getting done. The performance expectations for pulling and presenting a record from a data store is different from performance requirements for crunching through lots of data for a machine learning problem.

Ultimately performance comes down to CPU cycles. One core can only perform one computation within one cycle. Which then implies that to speed up your software task you can; use faster hardware that can perform more cycles per second, parallelize your job so that multiple cores are working on it at the same time, or reduce the number of cycles it takes to complete your job. This post focuses on the software solutions, however, giving granular advice on how write performant software is not something that can be done in this article. What I can do is list some reasons why systems are slow and design considerations to keep in mind in order to write fast software.

Excessive memory allocation / usage:
This is especially important in managed languages like Java and C# where garbage collection is handled by the runtime. While the process of memory allocation is almost free compared to C/C++, the more memory that is allocated means that garbage collection runs longer and more frequently. In addition to that, programs that frequently need to access memory and secondary storage end up being I/O bound. CPU bound processes will always be faster than I/O bound processes because the rate at which data is transferred from main memory or secondary storage to the CPU is typically slower than the rate at which the CPU can execute instructions.

Inefficient algorithms:
Its fairly obvious that the efficiency / inefficiency of your algorithm is directly related to your software's performance. Algorithms with quadratic complexity or higher fail to scale and there's always a performance benefit in the reduction of the complexity of your algorithm. Some techniques for improving the performance of your algorithm are:
1. using a hybrid or adaptive algorithm (read links for more info),
2. avoiding unnecessary work by; caching the result of an expensive function call and returning the cached result when the same inputs occur again, providing a fast path for common cases, etc.
 It's also sometimes necessary to consider the performance of third party or framework libraries / functions and switch to a library or function with a faster algorithm if available.

Too many layers of abstraction:
While layers of abstraction are great for encapsulation, information and complexity hiding, re-use, etc. Abstractions typically add overhead. Sometimes the overhead is trivial and other times it is significant. Therefore it becomes a question of trade-offs. To gain increased performance it may be necessary to lose a layer of abstraction and along with it, both the abstraction layer's overhead as well as whatever benefits it may provide.

Bring the data closer to the computation (or vice versa):
This is closely related to my first point but takes more of a macro view. The closer the data that is being processed is, the faster the process can run. So if the data is on disk and can be brought and managed in memory effectively (maybe in chunks) we can increase performance. Another example is bulk downloading remote data and operating on it locally instead of making several individual remote calls. Hadoop solves the problem of scaling big data problems by distributing the data and sending the computation as close to the data as possible or a machine, rack or cluster level.

Task parallelization opportunities:
This one is an obvious one. While parallelization can be difficult to do properly, it can certainly yield great results in the right situations. In addition to running multiple tasks in parallel on multiple threads, look for opportunities in which instead of waiting for a long running request such as a web service or database call to return a response, your process can make the call, continue doing other things and resume its original work when the call returns. An implementation of this is the async and await keywords in C#.

Avoid / minimize bottlenecks:
The benefits of multiple threads or processes can be limited when there is a bottle neck. A bottle neck is typically a resource that cannot be accessed or modified in parallel in order to maintain its integrity. There are cases where bottlenecks can be avoided (by not sharing data / resources), but sometimes we can't get around them. In those situations the solution is to hold on to a resource lock for as little time as possible and keep the operations involving the lock as simple as possible. Bottlenecks are also prevalent in horizontal scaling situations, where the database cannot scale like the rest of the infrastructure. Database sharding is a common approach to solving that problem.

In conclusion, there are many more performance factors that this article could have covered and I only went skin deep. In future posts I'll be taking a deeper dive into these and other factors. Feel free to hit up the comments section with questions or comments.

Thanks,

Wikipedia has a good article on program optimization here: http://en.wikipedia.org/wiki/Program_optimization

I was recently asked in an interview what I would do if I didn't have to earn a living; and even though my answer was a bit cliche I believed it to be completely true. I love programming and would do it even if I wasn't paid for it (I do like getting paid for it though!). So since everyone loves lists, I thought I'd make one on why I love coding / programming.

1.  Programming is a form of creation and self expression
A quote from The New Yorker magazine states "The world is being rebuilt in code". Where we had encyclopedias and libraries we now have search engines. Mechanical devices are being replaced with electronic ones. Where we had phone calls we now have numerous forms of communication and social interaction that don't involve talking. And the amazing thing is that, given a certain baseline level of hardware, all these creations are made possible with just a text editor, a compiler and a brain.  I'm not a very handy guy, but put me in front of a computer with those three things and I can actually create something that is of value to myself and others. I can be a designer, an architect, an engineer, a project manager, an artist, a creator, all behind a keyboard. How cool is that!

2.  I get the opportunity to solve problems
Who doesn't like solving problems? Men are naturally egotistical creatures, so we jump at the opportunity to take on challenges and solve problems. Need a way to update several people at once when an event occurs? I can solve that for you. Hate having to mail in multiple bill payments? What if I built you an app that would electronically and seamlessly handle your bill payments? Is it too hard to read through the mountain of logs your program produces? I can write you a program to parse, filter, and even alert and perform intelligent actions when an event is detected.

3. It's dynamic
Another great thing about programming is its dynamicism. There's no final state. There's always more to do, more to learn, new takes on previous solutions, better techniques, more to learn. You don't reach the pinnacle or complete your knowledge of programming. It's fluid, so it stays engaging. It's dynamic so it stays interesting. Bored on the job with the client server architecture? Take a look at the microkernel architecture or the pipeline architecture or the peer to peer architecture or the pub-sub architecture or the layered architecture or the ...

4. It's liberating
Unlike working within the confines of published software like enterprise systems, web or desktop applications and other software products where you can only play with the cards you are dealt, software development gives you the freedom to do whatever you want with your product. Want your software product to greet you with a clip from "Gangam Style" when you log in? Code it in there. Think graphical user interfaces are for children? You have the power! Trying to fend off the NSA? Encrypt the crap out of stuff. You want it, you got it, so long as you are willing to work for it.

5. It's just fun!
Remember Lego?
Lego == fun. Programming == Grown up Lego. Therefore;  Programming == fun

A large portion of my development career has been devoted to interprocess communication; from building Java based SOAP web services to using JMS and Websphere MQ for reliable messaging. I recently started working with ServiceStack which introduced me to the idea of Message Based web services. But before I start talking about the advantages, let me provide a little context.

The SOAP protocol is a protocol for performing Remote Procedure Calls (RPC). It allows clients call methods on a server via service boundaries (network, processes, etc) as if they were calling local methods. This method of interprocess communication has dominated the web services and RPC space for years due to its solid tooling and low learning curve. However, the SOAP protocol is complex and heavyweight. While this has its advantages such as; being able to accommodate advanced security and reliability requirements often found within enterprises, and being able to model complex / application-specific objects, SOAP's complexity and addressing scheme does not work well in the world of the world wide web.

There is another option however. Starting from the mid to late 2000's, REST-ful web services emerged and have now dethroned SOAP on the web in part due to its simplicity and its seamless integration with HTTP. And now REST-ful web services provide the foundation for Message based web services. Please note the subtle but vital difference in the two terms I'm using; "Message based services" and "Message based web services". Interprocess communication using Messaging does not have to occur over the web or over HTTP. There are several other message based architectures such as Message Queues, JMS, Erlang processes, F# Mailboxes, etc that have little to do with the web.

Some of the advantages of Message based services over their counterparts

• Message based architectures tend to have coarse grained interfaces, therefore they encourage less back and forth compared to Remote Method Invocation (RMI) style interfaces. This helps further decouple components.
• Messages can be versioned and evolved easily without breaking method signature. This makes refactoring less stressful.
• Messaging is naturally "batch-ful". This is important in high volume scenarios where batching requests together can result in a significantly more performant architecture / system.
• Messages can be cached, routed, chained and decorated. These qualities are especially important in web systems. Messages can also easily be logged and replayed at a later time.
• Messages are ideal for concurrency as they are thread-safe and can be easily multi-plexed with their handlers over multiple threads

These advantages point to message based services as being the better choice for distributed systems. And therefore they should be considered for your next integration project. The one caveat to my recommendation is that message based services, in particular, implementations using REST over HTTP do not have as strong of vendor / tools support as SOAP services. This sometimes leads to having to write a lot of serialization and service protocol code by hand.

To get started with implementing message based web services on C# .Net you can take a look at ServiceStack which provides an entire framework around message based web services but comes with a one-time perpetual licensing fee or you can use Microsoft Web API 2 and follow messaging principles. In Java, you can use a combination of a suitable JAX-RS implementation with JAXB to provide the message serialization and deserialization. For MQ and service buses there are several options out there that can be found with a simple google/bing search. ActiveMQ is a popular open source option.

Thanks for reading