Thursday, December 11, 2008

F# : Programming Not Typing

From Miriam-Webster:
Concise - marked by brevity of expression or statement: free from all elaboration and superfluous detail

Over the 4 months I have been developing an F# data visualization project for one of my clients. I found F# development extremely enjoyable experience. I have been trying to fully quantify why it has been so much fun versus other experiences I have had programming. One of the key benefits of F# is conciseness. Robert Pickering, the author of the “Foundation of F#” has discussed this aspect of F# and I agree with his observation regarding F#’s conciseness.

Developing in F# reminds of Peter Siebel's Practical Common Lisp talk in which Peter Siebel describes Peter Naur's theory of the programming in the context of Lisp programming to explain the expressive power of Lisp.

I looked up Peter Naur’s original text and found the following passages:

"..programming should be regarded as an activity by which programmers form or achieve a certain kind of insight of the matters at hand: a theory (...and not only the production of a program text)...

"Programming shall denote the whole activity of both design and implementation of solutions. "

To paraphrase Naur and Siebel, programming conciseness is the extent to which your program expresses this theory with least elaboration. One of F#'s biggest strengths is the conciseness of expression. F# incorporates many language capabilities to create a very expressive language, which leads to this conciseness. Here is a short, incomplete list of F# features, which promote code conciseness.

1. Type Inference
The F# compiler deduces at compile-time what types you are using based on the eventual evaluation of the expressions in your program.

Example:
// C#

String myString = new String ("I like the clicking of keys");

Double myNumber = new Double(4.0);


// F#

let myString = "I crave silence"

let myNumber = 4.



The compiler types both myString and MyNumber as type String and Float (F# equivalent of Double) because the right side of the let expression has a type that can be inferred.

In C#/Java case you have to tell the compiler that you are declaring and creating an instance String assigned to the identifier myString and and you are declaring an identifier myNumber of type Double assigned a new instance of Double. If you take a step back, the code has enough information to determine the types, so why should you have to type the type names?

In F#, the compiler infers the type from the expression. As you are typing your program into the Visual Studio F# editor, the compiler is constantly running in the background compiling and type checking. It creates this really interesting dialog between you and the compiler as you are developing your program. It took me a bit to get used to but overtime the compilers feedback made me more aware of types in my program and the overall semantics those types were trying to capture.

At first type inference feels a bit like Python's duck-typing, however the key difference is that type inference is statically checked and as a result you have a lot fewer type-error surprises at run-time. Type-inference eliminates the over-elaboration of type information within a program. F# type inference is not perfect. However, type ambiguities are treated as the exception and identifiers can be type-annotated, when the compiler cannot make a type determination based on the your programs expressions.

2. Significant White Space
F# handling of significant white space is Pythonic. At the top of an F# code file, you can specify a #light syntax directive, which enables significant white space for subsequent F-Sharp module. The #light syntax directive in F# is much like Python. Programming blocks are defined by the indentation of the programming statements in the file.

Significant white space is definitely a taste issue. Some developers hate significant white space, because it dictates a certain code aesthetic. I personally, like significant white space, as it eliminates line-noise in code created by the tyranny of the curly-brace.

Example:


void Foo
{
   if (something == true)
   {
      // this is a block
   }
   else
   {
      // this is another block
   }
}


//F#
#light

let foo()=
   if something then
      // here is block
   else
      // here is another block

3. Everything does not need to be an object.

I started my career, while object-oriented development was starting to take the industry by storm. I was one of the early zealots who believed OOP was going to save the world. Moving to objects wasn’t easy for the industry. Developers were extremely wary of "object stuff". A common refrain was that it would slow programs down. In the early days it probably did. Presently, all the major languages (by popularity of use) have object support and developers don't think twice about using objects for everything.

In C# and Java, all functionality is encapsulated through objects. C# one-upped Java by making all types derive ultimately from Object. The industry’s embrace of OOP, seems to be an over-correct. More recently C# has introduced closures, which allow functions to be defined independent of objects. Java 7 very nearly introduced a notation for closures, but unfortunately it didn’t make the cut for the next release. There are many instances where closures provide a better notation than standard object encapsulation.

F# straddles object functional programming models, which provides great expressiveness and the opportunity to use object when you needed and functions where better suited.

Example:

class FooClassHolder
{
   public static void foo()
   {

      // I can do some stuff

   }
}
let foo() =

     // I do some stuff


// call some foo

FooClassHolder.foo()

foo()



This is just a trivial example, but it illustrates how you can define a function without having to define a wrapping object class.

Conciseness De-clutters the Programming Mind


Developers say “big-deal” less typing…I type really fast and my fancy IDE types-ahead for me! Even so, I find that concise code allows me to better hold the concepts in my head. IDE have become really smart in large part to the typing challenge created by large bodies of code text. F#’s focus on conciseness in many ways obviates these intelligent IDE features. Don’t get me wrong, I really love Intellisense. I couldn't live without it and IDE’s provide a ton of value to developers. However, I find working with less code freeing mentally. I find myself having more mental bandwidth because I can more easily scan a smaller amount of code. Smaller amounts of code allows me to focus more on the intent of program, rather than pushing text around.

Tuesday, October 21, 2008

Enterprise Software: The Land of Diminished Expectations

I have been developing software systems for the past 14 years in a variety of industries. Much of my experience has been working on large systems. I have had the opportunity (and luck) to work on some awesome projects with great people. The last company I founded developed enterprise network management software. Working on a project from whiteboard to over 2 million lines of code gave me a lot perspective on the people dynamics created systems architecture and the burden created by large complex systems.

All projects go through a natural evolutionary cycle. Architectures die, are rewritten or evolve. Sometimes software just hangs around. When software gets really large and complex a tipping point occurs where complexity, maintenance, management dynamics overtake the initial innovation and creativity. Software, teams and businesses can fall into a negative reinforcing feedback loops.

Organizational dynamics play a key role in this negative feedback loop. I have observed that businesses tend to operate within vary narrow comfort zones. Managers will plan projects that are well within their delivery comfort zones and tend not engage their developers in risk taking. Overtime complex systems narrow the options for development teams by increasing the expensive and risk of development. Throwing away working code is the hardest thing for a software business to do. Most organizations have little reward for risk taking. Development managers usually have no incentive to deliver more than expected and there is always serious downside to having a project delivered late. The innovation loop is crushed under the jackboot of paycheck-fear.

Development teams taking their cues from management, do the minimum work necessary to meet the requirements and get home by 7 PM. Crank out another web-form, check the box and move on. These dynamics reinforce one another. The net result is to cut off real problem solving and stifle engineering and ultimately drive technology and products teams to mediocrity.

What is really crazy is that end-users get mediocre products and lower their own expectations as well. This is a race to the bottom. The funny part is that good developers want to tackle really hard problems, management wants more productivity and marketing people want to sell more innovative and differentiated products. Yet when this feedback loop occurs, no one is happy. To paraphrase Thoreau, most developers on large software projects toil away on mundane problems, living lives of quiet desperation. This is the land of diminished expectation.

A good friend of mine once described a database migration project as "soul crushing." It wasn't because the database work was boring, it was because the system he was working on became so complex that he was found himself constantly having to work around issue after issue. He felt he was spending his time and mental energies cleaning up after bad design. This kind of work can totally break developers.

I don't believe it has to be this way. Much of this dynamic is controllable. Design and architecture play significant role. Design and architecture establish the basis for both the product technology and the long-term development. If complexity is not controlled, teams spiral off into these negative cycles. This not to say that there is no mundane development. There is always some development that is mucking-out-the-stalls. However, I believe that poorly designed systems create lots of unnecessary work, discourage teams and generate mediocre products.

Thursday, October 16, 2008

Outsourcing: The Myth of Elves





As a the VP of R&D at a network management software company I learned quite a bit about the pitfalls of outsourcing software development. My initial experience with outsourcing was terrible and subsequent experiences were mixed. Like many development executives, I felt tremendous pressure by my CEO to outsource . Our company was trying to expand development capacity and outsourcing seemed like a good way to keep development costs down, while offloading a bunch of "non-value" work from my very talented Austin team. This all seemed reasonable. My guys could focus on product innovation and architecture and the dirty work would go the offshore developers. This is what my friend calls big-leather-chair-smoking-a-pipe thinking. Brilliant! There was one problem. Good developers will not be happy or productive doing menial development, regardless of their location. Unfortunately there are no elves to come in at night, clean-up our rooms and finish up the code that we don't want to work on.

So where does this leave the outsourcing model? As I came to understand, outsourced development works best when you are working with a real partner who has an incentive for mutual success. If you outsource development, try and outsource a whole product component. This provides development teams with a "whole something" to work on. It is really hard for teams to work within the your curly braces on a narrow snippets of functionality. At the end of the day snippet development is not worth it and all you are doing is outsourcing code typing with a bunch of management overhead. If you outsource little pieces of functionality, you will spend more money retooling deliverables and that cheap $25/Hr. rate will turn into net $100/Hr. because you and your team effectively explained the problem 20 times and eventually rewrote all the code.

If you think about outsourcing at the product level, you then can make the following trade-off: Can outsourced team A develop my product X end-to-end? Additionally, outsourcing a whole product is highly dependent on the complexity of your product space and the maturity of your product management process. If you only have one product, then you have to decide if there are clear enough boundaries cleave a "whole" component for outsourcing. If you can't establish a clear boundary, then the case for outsourcing become very murky. Also, if your internal development team has trouble meeting your product management requirements, chances are very low that you will be successful outsourcing. In general, it is better to find development partners and not body shops. Partners share risk and are incented for product success. Body shops are looking to bill as many hours as possible and are not motivated by your product revenue. Remember, there are no elves.

Wednesday, October 15, 2008

Software Entropy

Software, if done "correctly", can be an awesome business model. The software business model generates money for the repeated sale of ideas. The last 30 years of wealth creation in the software business is a testimonial to the power of this business model. However, the execution of this business model is highly varied. There are many business reasons for this variation and the correspondingly broad spectrum of outcomes. Different approaches to sales, marketing and macroeconomic conditions immediately come to mind. However, I feel that an area that is often overlooked: The software itself (aka the code).

Second Law of Thermodynamics
The entropy of the universe tends to a maximum. - (Clausius)

Unfortunately the development of large software systems obeys this law. Overtime development costs do not grow linearly. Each new line of code represents a potential anchor that must be carried to implement the next line of code. Each new team member is a new link in the communications-chain that creates potential signal loss. Unbounded complexity vines into the development of software systems and can strangle their development team's creativity. Complexity creep slows systems down and destroys development culture. Overtime as with the end of the universe, most large software projects experience heat-death, where all work energy is dissipated.

Managing complexity is not an accident. Understanding the problem domain and having systems designed that effectively express terms of that domain create the exact opposite dynamics. Each new feature builds upon the previous feature, reusing the work completed previously. This efficiency of expression leads to smaller development teams, which significantly lessens communications and process burdens. Development teams not only are more productive in this positive dynamics, overtime develop better understanding of the domain, which creates the opportunity develop a self-reinforcing innovation loop.

Unfortunately, most developers have not experience this positive loop in team environments or if they have it has been fleeting. I think that many factors have to come together in order to create the right conditions for profitable software. One of the key aspects is software design. Software design is not a process approach, a tool or technology. Effective software design come from developing efficient representation for a problem domain that can easily be understood and communicated. Effective design is the bulwark against software complexity and software project heat death.

First Post...

Over last 14 years I have been working on large and varied software systems. In that time I feel like I learned some good and bad lessons about what it take to make software work in products and complex operational environments. This purpose of this blog is to share my insights about the software development process and the business of software. Hopefully my random rantings will keep others from making the same mistakes I made.

At the start of my career I was fortunate to have a great mentor. He had a saying that I think is appropriate to the mission of my blog:
"It is always better to learn on another person's dime".