/devel/talk

Extension-based content negotiation and nested routes with Restlet

I've been working with Restlet to expose a RESTful api interface to the data model for one of my projects. Restlet is a super flexible library allowing one to configure and access all the properties of HTTP through a REST-oriented API.

The application bootstrapping options are also super flexible, allowing one to configure how routes are resolved and nest routes for cascading capabilities. I ran into a small caveat when I tried to configure extension based content negotiation. Basically, the idea of extension based content negotiation, is that instead of using "Accept" headers, one can append a mime extension to their request URI to request a particular return format. Say, we have a http://localhost/api/resource uri, one can request xml or json formats by simply doing http://localhost/api/resource.xml or http://localhost/api/resource.json. Of course your resource has to support these formats. The documentation on this type of content negotiation is non-existent. I had to scour a bunch of users group messages and javadocs before I figured it out. I figured I'll shared if someone else is interested.

My applications is written in Scala, so examples will be provided as such. I'm sure any experienced developer can easily discern the java equivalent.

First, in your application bootstrapping, you must turn on the extensionsTunnel option. Here is my code, which also demonstrates nested routes. Then, in your resource you must conditionally infer the MediaType provided and emit the representation of this resource based on it.

First, the root router's matching mode must be set to Template.MODE_STARTS_WITH, otherwise it will try to match based on full absolute uri path and not find any nested resources. So the matching mode is very important in the case where you're working with nested resources.

Second, you set the extensions tunnel property to true: getTunnelService.setExtensionsTunnel(true). This will turn on the extension tunneling service and perform content negotiation based on the URI's extension. Note: if an extension is not provided, it will resort to first available representation supported by the resource. It can get more complicated I believe based on other configurations, but this is what happens in the most simple scenario.

Now, with content negotiation on, the resource has to conditionally infer the proper MediaType requested and provide its representation for the MediaType. In Scala this is very elegantly done using the super flexible match/case construct. This construct can be used as Java's switch statement, but it is way more powerful and allows for advanced pattern matching. As you can see, I check for both xml and json media types and provide the proper representation. The supported media types are handled through @Get annotation. For more info, see Restlet's annotations and Resource documentation.

Now, accessing the resources yields the following results:

  $ curl http://localhost:8080/api/v1/test.xml  
  Hello from Restlet

  $ curl http://localhost:8080/api/v1/test.json  
  {"message": "Hello from Restlet"}

NOSQL Databases for web CRUD (CouchDB) - Shows/Views

There are many applications that easily land themselves to CRUD paradigm. Even if only 80% of the application's functionality is pure CRUD, one can benefit from a simpler storage model. For so many years many (including myself) thought that storing application state which needs to be used in an enterprise-grade applications, meant we had one option, an RDBMS. Not that other models weren't available, but their prevalence was not as high which made one question the quality/stability and long term health of such software. So we've gotten accustomed to approaching state persistence by sticking everything into one hole. If it didn't fit, we trimmed it, cut it, squeezed it, stumped on it, but we made it fit. Then when it was time to pull out, ah, we repeated the procedure. ORMs are one of the most popular remedies for such procedures. But if one has ever developed a complex data model and actually took the time to think about the data access strategy, on both ends, application and RDBMS, you'd quickly run into many limitations of the ORM model. I guess you can abstract away the impedance mismatch only so much, but watch out for these leaky abstractions. So if you're still rusty on your SQL and the relational theory because the great Gods promised that you'll never have to worry about it if you use ORM, you better get to learning, unless you're planning maintaining a ToDo List application for the rest of your life.

So, with this out of the way, let's talk about real data persistence. So there are many applications (especially web applications), that don't lend themselves very well into the relational persistence model. There are many reasons for this, but those who have ever had to beat their heads against the wall to bend the relational model to persist data know what I mean. By the time you're done, you're using and RDBMS to store non-relational data and all the benefits of the relational model are moot. You might as well store your data in an excel spreadsheet. So what are some of these reasons?

1. Highly dynamic structure (relational schemas are rather static, if you're doing it the relational way (no tall/skinny tables))
2. Data model is not very relational. That speaks for itself, but many don't really know when and how to identify this, as we've spend so much time identifying relations that don't exist or are irrelevant to the application.
3. Your relational schema is denormalized to the point where you're no longer benefiting from relational database features like enforcing consistency and reducing redundancy in the data.
4. Your relational database is bending backwards to accommodate your read/write throughput even after you denormalized (which itself is a reason to look elsewhere) and optimized, forcing you to continuously have to scale up to allow for increased load.
5. You continuously marshall/unmarshall data (ORM???) to persist it and then to transform it to another data format.

Touching a bit more on bullet #5; Lots of software is written using good OO principles of encapsulation. Encapsulation is the heart of software abstractions and is probably the most important principle. But people tend to abuse it. Abstractions are good when they add value, but marshalling and unmarshalling one data structure into another without a good apparent reason, other than you don't have to learn how to deal with a particular data structure, I'm not sure is such a good reason. So many software projects use ORM for the sake of not learning SQL, but how far can you actually get? ORM is a perfect example of a leaky abstraction. So many projects retrieve data from a web view in JSON or url-encoded format and marshall that into objects, only to validate the data and persist it using an ORM. So now you've unmarshalled the data from JSON to an object graph to just then marshall it again into a SQL query to send to the database. Do we really need these superfluous steps?

I'm sure there are other reasons I haven't mentioned here. These reasons I personally faced when making my decisions.

A rational way of deciding on data persistence is not to automatically start writing a DDL script or grab your ER diagram tool, but rather look at what data you have, how would this data persist in a "natural" way, how does the client software need to access this data, what are the performance/scalability considerations and then go out and look at different persistence models to find the best match.

In my latest project, I had to think about a way to persist hierarchical data. This data will be accessed through some web medium (browser, mobile client, etc...) majority of the time. One of the web interfaces will be an ajax enabled web app, another will be an iphone and/or adroid app. JSON is communication protocol lingua franca of the web. Some will argue it's XML, but I'll keep my XML opinions to myself at this point.

CouchDB is a document database which one would call key/value store. It allows for storage of JSON documents that are uniquely identified by keys. Sounds interesting, not really. There are tens of other databases that have same capabilities, so why CouchDB? Well in one short sentence: CouchDB is build one the web and for the web. So what does that really mean? Well, besides the JSON storage structure and its innate ability to scale horizontally, they've build some pretty awesome features that makes it very appealing for a particular type of an application. The task is to decide whether the application you're building is that application. So not to make this post any longer than it already is with my rant, let me describe and demonstrate some of the features that I've used over the last few days and why they are relevant.

Please make sure you have couchdb 0.10.* version installed as well as curl command line utility. For installation instructions see http://wiki.apache.org/couchdb/Installation

Once couchdb is installed, you can start it using the couchdb command. Depending on your setup, you can do...

sudo couchdb

A little bit of a background though before we get any further...

We're going to store hierarchical data, which JSON is a natural fit for. One of the issues we have, is that in our industry, there are numerous data standards and they are all defined either in XML or in some delimited rectangular format. One major use case involves performing CRUD operations on the data from variety of sources (web app, mobile app, etc...) as well as being able to emit this data in one of the industry standard formats for integration purposes.

CouchDB exposes a RESTful API, so it's rather easy to use it from any language which supports HTTP. Most popular languages have abstraction libraries on top of that, to abstract away the HTTP abstraction. Here is a list of available clients: http://wiki.apache.org/couchdb/Basics. For our purposes we're going to use curl, a command line utility which allows us to make HTTP requests. So let's see how we can easily accomplish this with CouchDB.

Now that CouchDB is successfully running, let's create a database and insert some sample data...

curl -X PUT "http://localhost:5984/sample_db"

Above line create a database called sample_db. If the command is successfule, you will see the following output: {"ok":true}. Now lets add three files to this database. The JSON data files which we're sending below are found in code snippets below labeled accordingly, so make sure they are in the directory from which you're running the below commands.

curl -X PUT -d @rec1.json "http://localhost:5984/sample_db/record1" curl -X PUT -d @rec2.json "http://localhost:5984/sample_db/record2" curl -X PUT -d @rec3.json "http://localhost:5984/sample_db/record3"

Again, each command should yield a JSON response with "ok" set to true if the add succeeded. Here is what one would expect from the first command: {"ok":true,"id":"record1","rev":"1-7c15e9df17499c994439b5e3ab1951d2"}. Again, ok is set to true making this a success response. The id field is set to the name of the record which we created. You can see that names are set through the URL as they are just resources in the world of REST. The rev field displays the revision of this document. CouchDB's concurrency model is based on MVCC, though it versions the documents as it updates them, so each document modification gets it's unique revision id. You can read more about this in CouchDB's architecture and API documentation.

rec1.json

rec2.json

rec3.json

Now that we have the data persisted, let's talk about some strategies for getting the data out.

CouchDB supports views. They are used to query and report on the data stored in the database. Views can be permanent, meaning they are stored in CouchDB as named queries and are accessed through their name. Views can also be temporary, meaning they are executed and discarded. CouchDB computes and stores view indexes, so view operations are very efficient and can theoretically (and I believe practically) span across remote nodes. Views are written as map/reduce operations, though they land themselves well for distribution. Here is an example of a map function in a view. (Reduce functions are optional if your query requires aggregating result sets)

There are other two really cool features, which allow more effective data filtering and transformation. These features are shows and lists. The purpose of shows and lists is to render a JSON document in a different format. Shows allow to transform a single document into another format. A show is similar to a view function, but it takes two parameters function(doc, req), doc is the document instance being iterated and request is an abstraction over CouchDB request object. Here is a simple show function...

The xml function and inlines you see here is the e4x which adds native support as a part of ECMAScript and is implemented in the embedded javascript engine Spidermonkey, which CouchDB uses.

This show function, takes a particular JSON record and turns it into XML. Creating a show is pretty simple, you just encapsulate the function above into a design document and create the record through PUT.

Here is the design document for the show above...

xml_show.json

Once you have the design document, create it...

curl -X PUT -H "Content-Type: application/json" -d @xml_show.json "http://localhost:5984/sample_db/_design/shows"

Note: in (..../_design/shows), shows is just a name of the design document, you can call it what ever you want

Now let's invoke the show...

curl -X GET "http://localhost:5984/sample_db/_design/shows/_show/toxml/record1"

Here is the output

So, that was super easy, we stored our document which required no code on our behalf and then we retrieved it with minimal effort by using ECMAScript's e4x facilities.

So how would I transform a record collection or view results into a different format? Well, this is where lists come in. Lists are similar to shows, but they are applied to the results of an already present view. Here is a sample list function.

Again, you encapsulate this list function into a design document, along with a simple view function...

xml_list.json

Now, we create the design document

curl -X PUT -H "Content-Type: application/json" -d @xml_list.json "http://localhost:5984/sample_db/_design/lists"

Once the design document is created, we can request our xml document listing all person records

curl -X GET http://localhost:5984/sample_db/design/lists/list/toxml/all

And the output is

So you can see how shows and lists are very useful and provide a convenient way to transform views into different formats.

As you can see, we created the database and stored data. No code was required to make that happen, just collect the data through your application and make a CouchDB REST request. We also added some custom functionality of transforming the data for multiple client consumption by using shows and views. In my opinion, CouchDB is a great step towards what one could call web/cloud scale database. It has awesome abilities to integrate with web technologies and it can scale to support the ever increasing web scale of data. In other words, it fits some application models like a glove.

I barely even scraped the tip of the iceberg of what CouchDB can do. We haven't talked about result aggregates which can be achieved with map/reduce, we also haven't discussed data validation and security. These features might be a top of some future posts.

Annoying javax.net.ssl.SSLHandshakeException exception

This exception has to be the most annoying one I've faced over the years with Java. I'm not sure which moron's wrote the SSL library, but did they think about providing an option to disable ssl certificate validation? I wasn't sure it was a requirement to have a valid certificate. I mean sure, it's nice and provides that worm fuzzy security feeling, but when I'm developing and/or testing, can you please provide some way to disable this annoying thing? Either way, I dug into this today and figured it out. It's actually as anything else in standard JDK, 100+ lines of code which they could of provided out of the box a simple boolean switch, instead your have to implement factories, interfaces, etc... WTF? Just to turn off certificate validation? Talk about over-engineering stuff.

So here is the code, which you can copy and paste into your project, instructions on how to use it are below...

import org.apache.commons.httpclient.protocol.Protocol;
import org.apache.commons.httpclient.protocol.ProtocolSocketFactory;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import java.io.IOException;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.net.SocketAddress;
import java.net.UnknownHostException;

import javax.net.SocketFactory;

import org.apache.commons.httpclient.ConnectTimeoutException;
import org.apache.commons.httpclient.HttpClientError;
import org.apache.commons.httpclient.params.HttpConnectionParams;
import org.apache.commons.httpclient.protocol.SecureProtocolSocketFactory;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public static class TrustAllSSLProtocolSocketFactory implements ProtocolSocketFactory {

    public static void initialize() {
        Protocol.registerProtocol("https", new Protocol("https", new TrustAllSSLProtocolSocketFactory(), 443));
    }

    private SSLContext sslcontext = null;

    private static TrustManager trustAllCerts =
            new X509TrustManager() {
                public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; }
                public void checkClientTrusted( java.security.cert.X509Certificate[] certs, String authType) {}
                public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
            };

    /**
     * Constructor for TrustAllSSLProtocolSocketFactory.
     */
    private TrustAllSSLProtocolSocketFactory() {
        super();
    }

    private static SSLContext createSSLContext() {
        try {
            SSLContext context = SSLContext.getInstance("SSL");
            context.init(null, new TrustManager[]{trustAllCerts}, null);
            return context;
        } catch (Exception e) {
            throw new HttpClientError(e.toString());
        }
    }

    private SSLContext getSSLContext() {
        if (this.sslcontext == null) {
            this.sslcontext = createSSLContext();
        }
        return this.sslcontext;
    }

    public Socket createSocket(String host, int port, InetAddress clientHost, int clientPort)
            throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(host, port, clientHost, clientPort);
    }


    public Socket createSocket(final String host, final int port, final InetAddress localAddress,
                               final int localPort, final HttpConnectionParams params
    ) throws IOException, UnknownHostException, ConnectTimeoutException {
        if (params == null) throw new IllegalArgumentException("Parameters may not be null");
        int timeout = params.getConnectionTimeout();
        SocketFactory socketfactory = getSSLContext().getSocketFactory();
        if (timeout == 0) return socketfactory.createSocket(host, port, localAddress, localPort);
        else {
            Socket socket = socketfactory.createSocket();
            SocketAddress localaddr = new InetSocketAddress(localAddress, localPort);
            SocketAddress remoteaddr = new InetSocketAddress(host, port);
            socket.bind(localaddr);
            socket.connect(remoteaddr, timeout);
            return socket;
        }
    }

    public Socket createSocket(String host, int port) throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(host, port);
    }

    public Socket createSocket(Socket socket, String host, int port, boolean autoClose)
            throws IOException, UnknownHostException {
        return getSSLContext().getSocketFactory().createSocket(socket, host, port, autoClose);
    }

    public boolean equals(Object obj) {
        return ((obj != null) && obj.getClass().equals(TrustAllSSLProtocolSocketFactory.class));
    }

    public int hashCode() {
        return TrustAllSSLProtocolSocketFactory.class.hashCode();
    }
}

Now all you have to do is call TrustAllSSLProtocolSocketFactory.initialize() anywhere in your application initialization code or right before you access any https resources, either through the URL class or through any other library, like HttpClient.

Hope this helps, though it's still a pretty ugly hack IMO.

Grails 1.2 dependency management

Grails 1.2 comes with a great dependency management. You no longer have to manually manage the dependencies and don't have to resort to using maven for this task. Grails uses Apache Ivy under the hood for transitive dependency management. Ivy allows to set up multiple resolvers and resolution patterns though allowing you to adapt to maven repositories and many other, even custom repository schemes. Grails takes it a step further and allows you to do this with a pretty nice looking DSL.

Application dependencies are configured in BuildConfig.groovy file and they look something like this...

grails.project.dependency.resolution = {
  inherits "global" // inherit Grails' default dependencies
  log "warn" // log level of Ivy resolver, either 'error', 'warn', 'info', 'debug' or 'verbose'
  repositories {
    grailsHome()
    mavenCentral()

    mavenRepo "http://download.java.net/maven/2/"
    mavenRepo "http://repository.jboss.com/maven2/
  }
  dependencies {
    // specify dependencies here under either 'build', 'compile', 'runtime', 'test' or 'provided' scopes eg.
    runtime 'com.mysql:mysql-connector-java:5.1.5'
  }
}

By default, grails pulls dependencies from your grails installation library. You can also uncomment (as you see above), the mavenCentral() repository. These are the defaults and don't require any more configuration. You can also specify a custom maven-compatible repo URL using mavenRepo.

There is also a flat dir resolver, allowing you to resolve dependencies to some local directory. You can do this as:

flatDir name: 'localRepo', dirs:'/some/path/to/repo'

So I naively tried using flatDir to resolve a local maven repo, which of course didn't work, since it's not defined with the maven artifact resolution pattern. In order for that to work, you have to define your own resolver for now. I just submitted a patch to add a mavenLocal() resolver. Should make it to HEAD shortly.

So, for now, you can do

  private org.apache.ivy.plugins.resolver.DependencyResolver createLocalMavenResolver() {
    def localMavenResolver = new FileSystemResolver();
    localMavenResolver.local = true
    localMavenResolver.name = "localMavenResolver"
    localMavenResolver.m2compatible = true
    localMavenResolver.addArtifactPattern "${userHome}/.m2/repository/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
    def ivySettings = new IvySettings()
    ivySettings.defaultInit()
    localMavenResolver.settings = ivySettings
    return localMavenResolver
  }  

Then use this method as...

resolver createLocalMavenResolver()

resolver method basically allows you to specify your own Ivy resolver. But that seems like too much work for something as common as local maven repo, especially since anyone who's going to use maven dependency artifacts, probably will install some locally.

So once the patch makes it in, you can just uncomment...

mavenLocal()

By default this resolver will resolve dependencies in home_dir/.m2/repository, but you can also specify your own path to the repository by just passing it as an argument...

mavenLocal("/opt/maven/repository")

Grails 1.2 is beginning to look very exciting.

GORM custom id mapping

I'm not sure why this is so undocumented. There is documentation about all these options, but you must collect them all from different sections. So I figured I'll document it here for the sake of people finding it, before they post it to the grails users list for the thousandth time never really getting a straightforward response it seems.

GORM defaults to using 'id' field as the primary key, so it injects it if you don't explicitly define it. You can easily change its type from Long to say String, but some Hibernate baggage follows. Basically, anything outside of the default Long id will not work, unless you also map is as an 'assigned' id, though hibernate will not try to automatically assign a long id during persistence of a new object. There are also cases that besides the type changes, you want to have the field name something else. I mean, when you have a User class, you might want to refer to its primary key as username and not just a plain generic id.

Here is how you completely remap the primary field (non-composite).

class User {
  String username;
  String password;

  static constraints = {
  }

  static mapping = {
    table 'users'
    id generator: 'assigned', name: "username", type: 'string'
  }
}

This will allow you to use username as the primary key field on this domain class, as opposed to id. id field no longer exists and is completely replaces by a string field username. The proper database schema is also created with username column as the primary key.

Choosing a web development framework/toolkit

I'm sure I'm not the only lunatic that spends many hours well into the night thinking about web frameworks, but then again, maybe I am. This is all exacerbated by the fact that I work for startups, so requirements are much different that say someone working for an established corporation that has various standards and practices in place. I left the corporate world 4 years ago and haven't looked back. I love the dynamics of the startup environments and my personality fits very well with its culture and pace.

So some of the questions I battle with are, which framework should I use for this new project, or am I using the right framework for my current project? Is the framework and language it's written in supports writing applications in a powerful, flexible, fast, scalable way? A lot of the criteria I just listed are not as much framework as design and architecture of your applications and infrastructure, but frameworks can make it easier or harder to achieve such a desirable architecture.

The issue is not as pronounced for other non-web applications, mostly due to the fact that most Turing-complete languages are capable of performing the same job as any other, the only question is the programmers preference, proficiency, and the availability of some framework/abstraction to make your life a bit easier. In some languages, writing these abstractions is a breeze or in some instances they are not explicitly available because some or lost of boilerplate is reduced through various language idioms.

But web development is so complex these days, that simple abstractions are not enough. Anyone that thinks either hasn't created a serious web application or posses some information that I'd be willing to pay to have:-) Sure, with the knowledge of HTTP and some gateway protocol, whether its CGI, Java Servlets, WSGI, etc..., one can do almost anything that's possible on the web, but that's a pretty bad criteria to have in the age of ever so complex applications/features. One doesn't want to rewrite something from scratch. Authentications/Authorization for example, although many applications have a pretty custom authn/authz scheme, 80%+ of what's needed is boilerplate that I nor any experienced application developer cares to reinvent. I'd rather be doing more challenging things, not sure about you.

So many frameworks conceal some amount of boilerplate from the developer through abstractions. Right now, there seems to be two kinds of web framework camp schools of thought:

1. I've done this many times before, trust me, you don't need anything else. I've extracted this, not made this up. The 20 applications that I've developed with this framework and extracted all of its generic concepts is all you'll ever need. Here is my convention down your throat. You think you need XYZ? No dumb ass, you just have no clue and your brain has all the baggage of a previous framework. Come on, open your mind, do you really need XYZ? You do? Well #$%@ you, go use someone else's framework.

2. We don't know what the developer wants, he might want bar or foo or barfoo or foobar or foobarbarfoobarbar, or what ever they wake up and desire that day. We'll come up with abstractions that can be extended by other abstractions. But wait, what if the developer wants to extend those other abstractions, oh, well we must allow them to do so, so here are some more abstractions. Before you get started, here are 50 different things you must do, with xml or code bootstrapping.

The second camps sound nicer, more sane in some instances, and you're overwhelmed with the flexibility that makes you believe that nothing is impossible. But that's further from the truth.

I've mostly used the #2 frameworks, as with many years of development, I've developed quite a convention of my own. No, it's not that I'm not open minded old timer that is scared of change, actually I love change so much that I find spending many unproductive nights hacking something in a completely different language/frameworks, exploring the ever so unpopular technologies, etc... But my conventions have grown empirically, though I'm not easily swayed to go back 10 years ago when I had no experience and relearn from the same mistakes I've already made to just come to most likely the same conclusions. People do this all the time, in every field and I don't have grant money nor a big corporation blindly investing in my useless use of time.

I'm pretty big on DDD and OO and all the abstractions that come with it and I need a framework that allows me to do so, without forcing me to mix relational and OO concepts by forcing me to use a weak ORM, or no ORM, or an ORM that they choose for me. I also am more than capable of deciding whether I need a Repository data layer and not just a plain simple DAO layer. I know what I'm talking about, at least I think I do, so I don't need any 20 year old telling me that using recordsets that masquerade as domain objects is just fine. Maybe for you, but I have a different opinion. It's possible that your todo list will do just fine, but not my software that might start off with 10 domain objects concepts and grow larger as functionality is added. Ok, enough with my discontent with the 20 year old programmers, there are many of them that are brilliant, they'll just have to learn (or not) as the time goes on.

Now with all that's mentioned above, I also am very aware of over-engineering and aren't we all so good at it. So, when I'm not in the mood to over-engineer something, I'd like a simple way of accomplishing a task. Without all the enterprise application pattern abstractions, etc... Sometimes I just want to create a quick prototype, start off simple, then grow it if needed into a production ready piece of software. Eventually through constant refactoring, I'll add the necessary patterns/abstractions as my requirements grow or change.

With all the above rant, I'm yet to find a framework that can do this. Rails maybe comes close, only one problem. ActiveRecord is very limited and sucks. Besides the mapping limitations, you also bind your domain objects to the relational model and are constrained into modeling your domain in a unnatural way. This might be fine if you're building a small app in its isolation, but it's a huge technical debt if you're building a service model on top of a persistence model that might be used/reused by other domains or services, etc... I want to have a consistent, coherent domain which is available as a service to other services, like webapps, background processes, etc... Good luck doing this with Rails. It is true that when I'm first starting my app this might not be a requirement, but like any other startup company, we have a vision and if that vision is realized, we'd rather not have that much technical debt to pay before we can move forward.

So after all the rant above, what is it that I want? Here is a list. It's not comprehensive, just things I can think of at this time. I'll update it as I think of anything else.

1. Support for MVC (most frameworks have pretty decent support for that now)
2. Extensible MVC (need to be able to extend the way controllers and views operate. Some frameworks do it by convention and limit you to a set of popular conventions.)
3. Allows you to build your domain in isolation. (I want my domain model to be completely decoupled from any web technologies, persistence, etc... Just a plain OO domain model)
4. Gives you very flexible persistence options. (I might decide to use a fully featured ORM (ActiveRecord is not fully featured), or I might want to use SQL, or heck, I might want to do both for efficiency or to scratch a morning itch, who cares about the reason, please let me choose. Oh, and one more thing, what if I don't want to use a SQL database at all? I want to use a native XML store or better yet a key/value store. Even if it's just to piss someone off, I want to do this and one should be able to accomplish that pretty easily. I'm not asking for a mapper for these stores, simply just don't make your framework bound to some relational store though making the work of turning this dependency off a 5 hour chore.)
5. Supports AJAX (I should be able to easily render JSON or XML views, without much plumbing or lots of mappings and annotations. The authentications and forms support should also expose some form of ajax compliant interfaces, so that forms and authentication can be done using ajax if I choose so. Be able to easily parse submitted data into some data structure and validate/synchronize it with the domain model.)
6. Bindings (All frameworks have some sort of bindings. Some of them are limiting. I don't want to create command objects to just simply bind the data and than synch with my domain model. If I have domain object graph(s), I should be able to bind it in the view layer. Bindings should be customizable. In Spring MVC for example, you can only bind one input control to a field or set of fields in the command object, but what if I want to bind 3 input fields that collectively represent one field in the domain object, I'm out of luck, unless I use javascript to first serialize those input fields into one field. That really sucks.)
7. Support RESTful, stateless, and other web concepts in a straightforward way. (I want to be able to configure every part of HTTP and the web and make the application work, look, interact in my way that's compatible with the web, not your way. Some component based frameworks make that harder that it should be, like the fact that they are inherently stateful by default. Some make it hard to support RESTful or custom URI schemes, because they transfer state through URL rewriting. All of these problems don't exist in some frameworks, like Rails, Spring MVC, Grails, etc..., so I know it's possible.)
8. Validations (Most have fully fledged validation support, but I can't say that it can't be made a bit easier. I do like Spring's flexible validation support.)
9. Forms (This is a big one. Can you provide a flexible way of creating forms and layouts. I mean seriously, we're developing forms today the same way we've developed forms 15 years ago. Every other aspect of development has moved on, but we're still doing bullshit html form controls. XForms is a way out, but no browser support and pretty hard to integrate support from vendors like Orbeon and Chiba makes the standard useless. Can we either embrace it or come up with something else. Am I the only one that gets an anxiety attack every time I think about creating yet another interactive form that doesn't do anything much differently than the form I created 4 months ago for a different project, though I either have to copy and paste all the cruft or start from the absolute scratch. Wow, that's sad IMO.)
10. Scalability. I know this one again is not up to the framework, but as I mentioned before, the framework can make it easier or harder to achieve. For example, inherently stateful frameworks that require either session affinity or replication of session state, make it very hard to horizontally scale. Yes, I know you can scale with replications tools available out there, but any synchronous replications is not linearly scalable. So any such frameworks makes it harder. There are many other criteria that can make a framework more scalable than others, but in general, statelessness, stability, and speed makes it viable for faster scalability tunes.)

Ok, that's it for now, I need to vent before my brain allows me to think of other things I've encountered of my never ending framework journey. I'd really love for someone to just say, hey you're wrong, there is such a tool(s), here it is. I'd be eternally grateful. Many will say that it's useless to complain, if I see a need, help create the framework or functionality that you think is needed. I wish I had more time, until then, I'll continue to grunt and develop my own inner frameworks to make things easier. One day, if I have time, maybe I'll devote some time into making some existing open source framework better. I've had more time years ago and contributed to quite a few open source projects that I'm truly starting to miss it now. I still occasionally submit a patch or two to a framework I'm working on after fixing an issue or adding some feature, but at times I'm in such a hurry to move on to the next task, I don't have the time to package or generalize it enough to make it useful for everyone else.

Right now, I'm working with Grails after starting a project in Spring MVC and not being able to deliver functionality as quickly as I wanted. I'll have to live with some issues I found with Grails when I was using it about 4 months ago, like the fact that you must use hibernate or gorm, crappy groovy stack traces, etc... Hey, there is always something one might not like, but I really like Grails and am hoping that now that it's in the hands of SpringSource, they'll spruce up the documentation to be more like Spring's awesome documentation and clean it up a bit.

Update: I wanted to reiterate a bit on the Grails in regards to isolated domain model. Grails does allow you to create and isolate your domain model and its persistence, unfortunately you have to twist it's arm if it's anything outside of GORM. You don't have to add classes to domain directory, but wtf is it there for? Also, it would be nice if the grails team provided a way to specify which classes in the domain directory are persistent or not. I mean, a domain model != persistent entities. So transient classes and other domain artifacts should also be grouped together. Putting them into src/groovy sucks personally, because I have to navigate two directories now to look at what's supposed to be a coherent domain model.