?

Log in

No account? Create an account

Previous 10

Dec. 29th, 2007

beer

Autonomic Computing

I was clicking around on Slashdot today and came across some stuff about agent computing.  We've also done some looking at it in my Master's  program and I worked on it a little bit at my internship at IBM.  My problem with agents has always been the fuzziness of what actually constitutes an agent.  It really isn't very clear.  Then there is all of the different terminology, some of which overlaps depending on who you talk to.

You've got agent computing, organic computing, and autonomic computing.  All of these seem to be different words for the same thing to me, but I would bet if you talked to a researching in each one they would say that they are different from the others.  So to add to the confusion here is my understanding of the entire thing.

Agent, organic, and autonomic are all the same thing.  I'm going to choose autonomic because it seems to be the most descriptive of the idea.  The setting for the idea is not financial calculations or number crunching.  Instead we want to move to the areas of business rules, controlling systems, and so on.  Essentially we are looking at where humans have to take a large amount of data and making complex decisions to make a change to the system creating that data. For instance, a system administrator will take apache logs, his knowledge of traffic patterns on routers, queries being done against databases, etc. and tune the parameters on a few systems to keep the site running "well".

The idea in Autonomic computing is that humans have better things to do than sit and look at those logs and fiddle with the MaxClients setting in Apache.  Instead we should come up with a way of describing the system (the "model"), possible things that can be done to the system (the "actions"), and how we want the system to look (the "goals").  Once we have that we should be able to have the computer then watch things and using the model try out various things in an effort to keep the system in line with the goals by performing different actions.  This is an automation of the the system administrator does.  So now the system administrator needs to maintain the model, the actions, and the goals instead of the system itself.

The problems in this are how to make that model, choosing the right actions, and describing the goals.  Choosing the right actions seems like it should usually be fairly straightforward.  In fact most do it already.  Keeping with the system administrator, he just chooses actions as the scripts that he usually executes.  Models and goals, however, are not so simple.

The model needs to be done in such a way that we can make predictions about the real world.  That means that it has be remain computable.  In many situations simply computable isn't enough, it has to be efficiently computable.

The goals are also hard.  In many cases it isn't even clear to the human what the goals are.  How are we supposed to describe them?  It seems, though, that if we could get the model, then the goals are a statement about properties of the model.

So that is the idea behind autonomic computing.  Doesn't seem so hard until you try to work through it for a specific system, then you start running into all sorts of problems.  What aspects of the system are going to be in your model?  What aspects are not relevant? How is the system going to figure out what it can do?  How do you describe the changes to the system/model that the actions will do?  Is it even deterministic what they will do?  Who is responsible if the system doesn't do the right thing?  How are you going to represent the model?  And the list goes on.

Nov. 5th, 2007

beer

An interesting thing has happened lately...

in the world of perl.

I think it all started in ruby, but may be traced back even farther, and I'll try. Ruby is another scripting language that has a lot in common with perl (take a look at some of the methods available on String such as chop, chomp, etc.). But Matz, the creator of ruby, stated once that a lot of the inspiration came from lisp.

Now lisp (and its children) is interesting because it has these things called macros. No, nothing like the watered down, dangerous macros of C. These macros are an important part of the language because they let you essentially define new language. The macros have been used in lisp for many, many years to create libraries of essentially new *language* features (not just libraries) which has caused some to say that lisp programmers don't write programs, they write programs that write programs.

All of this language changing has been taking on a name lately and been getting called "Domain Specific Languages." These DSLs, as the name implies, are little languages put together to better express the domain you are working in than a general purpose programming language can.

Sometimes a DSL is external to the language (you see them quite often in Java land as XML) a good example of that is YAML (YAML Ain't Markup Language) which is a language for defining data in data-structures:

-
	name: Bob Marley
	age: 30
	children: [Mary-Sue, Moonbeam]
-
	name: Jay Leno
	age: 55 


Sometimes it is internal to the language (written in the syntax of the language itself). Ruby on Rails uses this quite often for things like dispatching responses based on what was requested:

	respond_to do |format|
		format.html
		format.xml { render :xml => @recipe.to_xml }
	end


These kinds of mini-languages are not really new (lisp did them, ATT Bell Labs had a philosophy for them along the lines of "Library design is Language Design"), but they are getting a lot more attention now. Mostly because as systems get bigger and bigger and become their own large worlds unto themselves the *general* purpose languages are just too general to express clearly what is happening. Think of it as the technical jargon of a specialty being formalized so that a computer can understand it (in fact I would say that that is a major part of what software engineering is).

What does this have to do with perl? We'll for a while perl has been a bit of a languishing language. Many malodorous features of five (perl version 5) have cause confusion and contempt among possible perl programmers. But lo! Language laxness is the perl prime! Raucous ruby wranglers poo-pooed perl's packages, obfuscated objects, and odious operators.

Perl has a new vigor in perl 5 now. Many of the techniques that formed in ruby for creating these languages are now crossing over into perl. Much more use of blocks, thick-commas, and objects is taking hold. The Moose system is giving perl a nicer syntax (and semantic) for objects. Interestingly it is happening in a very similar manner as it did in lisp (no objects in the language and then language features were used to create a system). I'm also starting to see several modules that work by creating mini-language declarative style interfaces.

We'll see how far this goes...and what it will do for perl 6.

Jul. 26th, 2007

beer

TDD is hard

I've told a lot of people that learning to do TDD was one of the hardest things that I've ever had to learn. Partly I say this because people don't like testing or consider testing something too hard and by saying this I admit that it is hard and so we can get past that point. I do really believe that it is hard too, but since it is hard maybe we should look at why.

There is a common thread in the TDD community that "TDD is about design! It is only minimally about testing!" I think that that statement is exactly why we find it hard. TDD you are trying to specify what the design is going to have to do. That means that you are going to have to think at the specification level. Most of us have almost no training at that level. We immediatly start thinking about how we want it to work instead of thinking about what it is supposed to do.

I still have problems when TDDing. My biggest problem right now is that I very quickly loose sight of the bigger picture. I miss that there are other things in the design that I'm going to have to tackle once I figured out this one thing that I am currently laser focused on. I get tunnel vision and then in the euphoria of getting it working I skip some things that should have been done as well.

And that is then where pair programming comes in.

Apr. 20th, 2007

beer

Design Patterns and DSLs

I just read an article by Mark Jason Dominus (author of the wonderful "Higher Order Perl" which complements "The Structure and Interpretation of Computer Programs" completely) about design patterns being deficiencies in the languages where they appear.

Ralph Johnson (author of the Design Patterns book) wrote a response where he seemed to miss the point. He finds it surprising that anyone finds it surprising what MJD said. And then goes on to imply that what MJD was talking about leads to bloated, unusable languages.

I have to agree with MJD on this one. What he had written wasn't the first time that I had heard it. But that is because I had already read the stuff by Peter Norvig about patterns in the dynamic languages, and because I had asked Damian Conway at one point his thoughts on design patterns. His answer was terse and abrupt, but that abruptness got me thinking (I had been trying to apply some of the design patterns from the book in Perl and had been finding many of them unwieldy).

MJD says:

What I imagine is that when pattern P applies to language L, then, to the extent that some programmer on some project finds themselves needing to use P in their project, the use of P indicates a deficiency in language L for that project.


To me this all points to the idea of DSLs (Domain Specific Languages). Rather than trying to do everything in these general purpose languages, which cannot and should not include a lot of the patterns, we should be doing a lot more creation and use of Domain Specific Languages.

Many languages that have the ability to create internal DSLs do not show the need for as many of these design patterns. They have little languages that cover them in the situations when they are needed. The power to create DSLs inside a general purpose language is something that a lot of the mainstream languages lack. The newer "hot" languages have this ability (for the most part), and I believe that this is the reason for a lot of the popularity.

Looking at it this way you can see that it is completely reasonable to implement pattern P in language L to address the deficiencies of L in the context of a project.

P.S. For some reason that I cannot fathom, if you try to follow the link from MJD's article to Ralph Johnson's article you will get a 410 Gone error. Just click into your address bar and hit enter and you will get the page. I don't know if that will happen from my post as well or not.

Mar. 26th, 2007

beer

Templating

I'm back in Seattle for a while and working for the company that I used to work for.  There are a couple of things that took me by surprise getting back into the codebase.  Most of these are about the saying/idea/adage/whatever that says that code is good when you can come back after 6 months and still understand it.

Well, I'm back after about 9 months and had little trouble understanding most of the code.  The largest hurdle was getting my system setup to run the software, but even that wasn't too bad.  I had to do a couple things that were not quite documented (and I updated the docs to reflect that), but for the most part I could just do what the things said.

The code itself is not as bad as everyone thought when I was here.  Looking at it now, it is actually pretty good.  The nice thing that I learned is that the unit tests that go with the code are some of the most useful things that there are.  And since the code here has such good coverage, I have really had very few problems understanding everything.  Even the poorly written tests are very useful!  I did notice that with the fresh perspective, I could more easily seperate the good from the bad and mediocre.

I even got a chance to try out some new techniques for Template Toolkit!  One problem that we had had, was the when modularizing a template and creating BLOCKs and seperating things out for reuse and using PROCESS to bring them in, you began to have problems with Javascript and CSS that those parts require. 

A simple solution is to place all of that CSS/JS into a main CSS/JS file for the site.  That will work, but then that CSS/JS file starts to get unwieldy and adds quite a bit of unneeded overhead to most pages.  Most of the older templates did it by simply ignoring the HTML standards and putting script, link and style tags all over the place.  That isn't ideal because it means that you are not really future proof for browsers.

I figured out (and this has probably already been done and documented somewhere) that you can use TT's crazy scoping to an advantage.  The part of the template that will print out the head section of the page is put into a WRAPPER.  It expects some variables to possibly contain  an array some extra CSS/JS/what-have-you.  When it prints the head it just loops over the array putting them into the style block with any other CSS/etc.  The other templates can then just push things onto the variable to get their own elements into the head.  Since the WRAPPER code runs after its contents this will work to get all of the content's code into head, and since variables end up being essentially global all of the time (except if you use INCLUDE, I think), they can communicate effectively this way.

Feb. 25th, 2007

beer

Specification

In my classes we have been working a lot with formal specifications of software systems. Modeling the system, moving toward the theories required for verifying the models, and composing systems using the models.

In all of this I kept seeing very familiar topics and techniques coming up. Again and again I had a sense that I had seen this before. After a while it dawned on me that it was another approach to what TDD had been trying to accomplish.

Both techniques are trying to get to a similar (I won't say the same) endpoint. TDD took the path from what people are doing right now and what they can improve on. Model Checking took the path from the theory and what should be possible and tried to get it to a point where people could use it in day to day work. I keep thinking about this, and I think it is becoming clearer and clearer to me.

In the beginning of software development we had Code & Fix. Someone said that the system needed to do something and the developer(s) would go away code something up, put some fixes in, and come back with the result. This kinda worked, but what the developers came back with didn't always match what had been asked for.

In response the practitioners added in a Test step. So now we had Code, Test & Fix. The researchers took a different approach. They said, "We have all of these ways of describing things very specifically with mathematics. Can we come up with a way to describe what the system should do and then prove that the system does that?" That was the birth of formal methods in computers. Djikstra, Hoare, and many many others worked on these problems.

The formal methods crowd made headway. They created algebras to describe what a system should do. They developed languages for annotating a program with what it does. They applied mathematical logic to show that you can then prove parts of the system. These algebras and calculuses kept evolving, but kept running into the problem of complexity of the systems.

The practitioners kept trying to refine the Code, Test & Fix approach. They added more guards into the process to catch errors earlier. They had there own models. They talked more about process models. What does who do when to make sure that we create the correct system in the end. The improvement here was a bit more haphazard. Much more was changed based on personal experience than hard numbers or provable results. Other problems were that it was never clear what needed to be tested, how you could tests some things, or where testing fit into the entire thing.

In response to this the idea of "Test Everything That Could Possibly Break" emerged. A little rule of thumb that gives you some guidance about what to look at. Sitting behind the idea, however, was something much more important. Quickly TETCPB transformed into "Test First". If those tests are so important, then maybe we should figure them out first. That also helps in knowing that the test is looking at what you think it is (it should be failing because you don't have the code it is testing yet). This idea of writing the test first continued to evolve. Names changed: "Test-First Development", "Test-Driven Design", etc. but the concepts remained essentially the same.

At this point the formal methods crowed was looking seriously into figuring out ways to automate the incredible complexity that comes with trying to prove that a piece of software matches the formal specification for it. Large model checking systems were created. Languages to describe the software and the specification came about. The systems were able to be successfully used to check real world systems. They even had several cases of major success such as showing that a protocol that had been believed to be secure was actually insecure.

Ideas started to cross the chasm between the two groups. One of these was the idea of Design by Contract that made it into the Eiffel programming language. Design by Contract based itself on many of the basic ideas of the formal methods crowd, but worked in a practical and widely used programming language. It achieved this by realizing that complete checking is not currently feasible, but simply specifying the software is of enormous use. Rather than exhaustively proving that the software conforms to the specification, one needs only make sure that during use the software conforms. When you pass your software through enough use cases and all the specifications are satisfied you can then be reasonably certain that you have correct software.

At the same time TDD continued to evolve (Eiffel and TDD started around the same time, I believe). Once again that theories started to come across. Specification is much better than test. Specification means that you know what the software is supposed to do. Test only says that the software does one thing in an expected manner. One of the fundamental ideas of TDD started to be pushed more. Many practitioners had not caught on to this, however: TDD is a method of specification, not of test. In response the current state of TDD was reevaluated and some practitioners said that a major problem came about because of the language (meaning the words, not a particular programming language). The language in most TDD literature and frameworks was that of test, not of specification.

Out of that realization the idea of Behavior Driven Development came about. TDD started to near Design by Contract in terminology. However, the disciplines remain distinct. This is where we find ourselves right now. Where will it go from here?

I predict that BDD will take on more formalisms and will hopefully be incorporated directly into more languages. This will marry the process practices of TDD/BDD and the implementation of DbC. The formal methods will continue to evolve and will hopefully one day be able to understand the specifications used for DbC and BDD. Then we will have the practices of BDD with the check by example that it uses for checking conformance, DbC with the runtime checks for conformance, and Model Checking for more exhaustive checking when needed.

Feb. 10th, 2007

beer

Specifications

In my classes here we talk a lot about formal specifications of programs and using the specifications to prove the correctness of implementations. It is a very interesting idea, that has unfortunately not been able to be applied much in practice because of the shear complexity involved in trying to prove things about programs.

I have been for a long time now very interested in TDD and testing of software. One of the things about TDD (Test-Driven Development) is that you are not supposed to be thinking of the practice of TDD as testing. Even though that is in the name, and everything uses the nomenclature of testing it is not about testing.

It is about design.

Formal specifications are about design as well.

The more I've seen these things the more that I have noticed that they are trying to do almost the same thing. The TDD crowd took the very pragmatic approach of using what we can do (write snippets of specifications as code to check some other code) and applying it. The formal methods crowd took the route of purity by taking pure math and using that to describe the systems.

Both sides have downsides. In TDD you can't really prove your program correct. The best you are getting is that in the cases you chose you have shown that it works. Formal specifications can prove your program correct, so long as it is fairly small and simple. Anything larger and you will just get completely overwhelmed.

Recently the test focused nature of TDD has been called into question. From the beginning most masters (I don't think they would want to be called that) in it have said that you are driving your design, not testing your code. Most people encounter problems with this (I think) because the terminology does not match up with what you are trying to do. A Test Case is a Specification? Not in most people's minds.

Out of this has grown some movement to change the terminology to save TDD from itself. One project that I have found interesting is rSpec for Ruby. A wonderful article that explains this a lot better than I can is Dave Astels. I found it so interesting that I decided to see how well I could get something like this into perl. I've always loved perl's testing facilities, but they have never quite given me the right framework to really express that my .t files are really my specification.

So I just hacked together (no way this could be used for a project yet, but I'll probably start using it on my projects and growing it that way) something that looks like rSpec for perl. I'm calling it pSpec for now, but might change that. Since I haven't actually used rSpec or jBehave yet, I don't know if I have large gaps in what I'm doing, but it is an interesting experiment so far.

What does it look like?

use pSpec;

context "x variable", is {
    my $x;

    setup {
        $x = 2;
    };

    specify "should be 2", as {
        should_be $x, 2;
    }
};



I won't bother anyone with the implementation for now :)

Jan. 4th, 2007

beer

Essential Difficulty

In Lawrence Lessig's book "Code" a major topic is the need to decide about things that have never needed to be decided before, they could be taken for granted. The book is a study of changes that have been prompted by computing and the internet. These changes in many parts come down to things that were not possible or were expensive before are possible and cheap. This has caused a need to re-evaluate what is legal/illegal, right/wrong, possible/impossible. It seems innocuous enough, but it has major ramifications. The law can no longer assume that something will still be done (on the internet) because there isn't somebody saying that it can't. That doesn't work anymore. Now you need someone explicitly saying that it *can* be done.

Why? Well because only what we make possible is possible inside these logical constructs we call computers. The system will do anything but will only do what you tell it to. So if you never tell a system to log that a user has logged on, you'll never have a record that a user logged on.

Fredrick Brooks talked about essential versus accidental difficulty in software development in his essay "No Silver Bullet." He credits the terms and the ideas to Aristotle. An essential property of an object is one which cannot be removed without removing some part of the identity of the object. An accidental property is one which does not need to be there but is for some reason or another. An accident property can be removed with no change to the object.

In "Code" the computers removed many accidental difficulties in law. An example would be copyright law never dealing too much with what exactly constitutes "Fair Use" because the technology at the time made it difficult to copy without limits. Computers removed the accidental difficulty of copying and forced us to deal with the essential difficulting of intellectual property ownership.

Today I was thinking about other essential versus accidental difficulties. One thing that came up often in my work was getting people to think about requirements in a system for logging. The problem here is that in most people's every day lives they have no need to think about such things. Common objects do not need to record explicitly what they did and what happened to them. It is an accidental (or is it?) property of a bridge that when it fails the structure will be bent, the metals will be stressed, and it will hit the ground leaving marks. Unfortunately when a software system fails it does not neccisarily leave a trace. Unless the system logs out what happened as it failed there is very little left for anyone to piece the situation back together.

Gathering software requirements removes all of the accidental properties of the object and leaves just the essentail ones. Sometimes we want those accidental properties to remain because we use them to reason about the world around us. The hard part is that until they are gone we might not even realise that they are not essential to the system, but are *essential* to our *use* of the system.

So maybe this is an argument for developing software based on uses of a system, rather than based on the properties of a system. Thoughts? I think the jury is still out on it.

Nov. 29th, 2006

beer

Striving for an unknown outcome

I've been thinking about my future recently. Where am I going? What do I want to do? What kinds of decisions do I need to make to get there? How long will it take?

Along with all of those questions are also the tradeoffs that will need to be made (and tradeoffs are not bad, in fact sometimes the constraints that force tradeoffs make you realize what really is important). More or less money? More or less responsibility? Live in a country where it isn't my native language or not? Travel a lot? The list could just go on.

These lists will never shorten. As soon as I have figured out one thing then a new question will come up, be revealed, scream for attention. The question right now is how can I get to those. If I don't work on the current questions new questions will be slow in coming, learning will be slowing in accumulating, and life will become static (and static life is death). So what do I do about it.

Well, one of the only things that I can think of right now is to try setting a goal for myself. This isn't a goal that I can necessarily do, but one that I would like to do and think is possible. If I achieve this goal I don't know what the outcome will be right now, but if I don't I am pretty certain about the outcome.

I need to improve, I need to grow. So here is my goal: I know my future has something to do with software development. I'm in a graduate program about software engineering. I have personal interest in the subject and I have some small amount of experience in the field. I am going to try to write one entry a week, of no pre-determined length, about software development. I don't know what topics I'll cover. Sometimes I won't meet my goal, but I'll try to reach some portion of it. If I can't write about software maybe I'll write about beer, or cooking, or Germany, or politics. But I will write.

Let's see how this goes.


Practice doesn't make perfect. Perfect practice makes perfect.

Nov. 7th, 2006

beer

Back to school

I started school a few weeks ago. Now I have to get into that school groove again, while at the same time getting up to speed in a new language. And by a new language I don't me a new programming language. I mean German.

I'm in graduate school in Germany. It is really starting to sink in. Mostly when I have to get in front of the class and say something. It really is frusturating to not be able to say exactly what you mean, but I guess I have that problem in English, too. :)

The homework hasn't been too bad yet. I'm waiting for it to pick up though. The most aggrivating thing about it is that it isn't very well organized (I'm not always quite sure what is expected from the homework).

Let me see, what else? Not much really, just living the German life. We live in a really nice part of the city and have pretty much everything we could want. It is also amazing what less stress will do to get rid of stomach problems. I still think it is funny that moving half way around the world to a place where I didn't have a home for 2 weeks is less stressful than my job was.

Previous 10