Saturday, August 14, 2010
I will be writing more posts on this blog shortly.
Saturday, July 17, 2010
In our handleRequest implementation, we simply generate a new session id for every new user using our own IdGenerator. Notice how we can use the request.getUri() to check that the requests have been made to the expected URI which is ‘/sessionTest’. The QueryStringDecoder from the Netty API helps in extracting request parameters as name-value(s) map. If the request provides the username, we simply create a new session id and store the user session details in our local memory map in SessionIdHolder. We send the response to the user – ‘Session ID is <…..>’. In the next request, the user can copy this session id from the response and send it as a request parameter. Then the server extracts the user session details mapped to the session id, after stripping off the server instance id, and finds out the corresponding user name. It sends the response ‘Welcome user <….> to server <…>’. If it does not find the session in its map, it returns the response ‘No valid session, please login with user’.
The sessions can be shared across servers using replicated caching or a shared database. However, to keep our example simple, and to illustrate load balancing with session stickiness, we have not replicated the sessions.
The sendResponse method sends a response to the user channel from which the request was received. It sets the desired response status – 200 for OK, 403 for UNAUTHORIZED. The HttpResponseStatus from Netty API provides all the HTTP response constants. We also set cache-control and pragma headers to ensure that responses are not cached by the client browser. We also set the content-length header. If the request had a CLOSE header or was using HTTP1.0 protocol with no KEEP_ALIVE header, we close the client socket channel after providing the response. If not, the same socket channel will be used for multiple HTTP request-response cycles between the client and server.
Now that we have the handlers ready, we write the main class to start the HTTP server.
java apachetest.SessionTestServer netty1 . 9080
Started server at port 9080 Press any key to stop server
We get back a response on the browser window – ‘Session id is ….’
Now, we copy the session id and change the URL address at the browser to http://localhost:9080/sessionTest?jsessionid=0140C3892B4F5348.netty1
- where the ‘jsessionid’ request parameter value would be the session id which was assigned to us, in the last response.
We now get the response – ‘Welcome user Archanaa to server netty1’
At the console where netty1 process has been started, press any key (like Enter). The server process will stop with the message – ‘Shutting down server’
Now, let us use Apache Web Server for load-balancing two such HTTP servers with session stickiness.
After installing the Apache Web Server, which would listen at port 80 on your machine, go to conf/ directory and make the following changes in httpd.conf configuration file.
- Uncomment the modules mod_proxy, mod_proxy_balancer, mod_proxy_http and mod_rewrite.
- Add the following VirtualHost configuration to the end of the httpd.conf file.
Now let us start the two server instances like so-
java apachetest.SessionTestServer netty1 . 9080
java apachetest.SessionTestServer netty2 . 9081
Similarly, in a different browser window or tab, let us connect with a different user http://localhost/sessionTest?user=Ankur and we would get a new session id.
Now let us change the request to give the jsessionid parameter in the request instead of the user and see the results.
No matter how many times we connect with the same session id and in whatever order, we always get the response from the correct server showing that session stickiness is working properly.
Now let us stop one particular server instance, say netty1 and try to connect its user. Now if we fire a request from our browser with netty1’s session, it would be failed over to the next instance, that is netty2. Since we have not replicated our session, we get the response –
Meanwhile, the other user is able to connect as before successfully with netty2’s session.
Thus we see that our simple experiment has worked well.
There is no limit to how we can use JBoss Netty to build our own HTTP server. In the above example, we have kept the handler synchronous. But there is no restriction for saving the HttpRequest along with a unique asynchronous completion token in a map and doing asynchronous request processing in a different thread. Once the request processing is done, the request can be retrieved from the map and we can send the response as we have done in this simple example above. To add robustness, we can keep a time-to-live for requests and on timeout, remove the corresponding request and send a REQUEST_TIMEOUT response to the client.
With that I conclude this series and I hope I have been able to showcase a fine alternative to using Servlet API and Containers for building simple HTTP servers.
Sunday, July 11, 2010
Saturday, July 10, 2010
“Each request/response object is valid only within the scope of a Servlet’s service method, or within the scope of a filter’s doFilter method. Containers commonly recycle request/response objects in order to avoid the performance overhead of request object creation. The developer must be aware that maintaining references to request/response objects outside the scope described above may lead to non-deterministic behavior.“
With the ability to store references to request and response references in memory, now one is free to loosely decouple the layers and queue up requests for processing and response data to be rendered.
The benefits can be profound. I have recently used the above approach of keeping the layers loosely coupled and found that I could attain a huge throughput from one machine itself whereas in the earlier approach I would have required multiple horizontally clustered machines and application servers. The reason, I believe, is that here we are decoupling the layers such that each layer has its own thread or thread pool concentrating on the specific responsibilities of that layer; rather than the earlier approach of making threads span through all three layers. The front layer concentrates on I/O and very little CPU processing; the business processing layer would typically have more of CPU-bound activities; whereas the Data Access/EAI layer would again have more of I/O related activities. This division enables the CPU to more optimally switch its attention to different layers. Moreover, one layer does not affect the lifecycle and processing of another layer. Thus the front layer is free to accept more and more HTTP requests and work on them.
To design such an application, though, one must be careful to appropriately time-out and clean up request and response objects, not to mention a more sophisticated programming and design with multiple threads to keep the layers loosely coupled.
In fact, possibly realizing this, Servlet 3.0 specification has sought to remedy this by introducing Asynchronous request processing. With Servlet 3.0, The HttpServletRequest has a new method startAsync() which returns an AsyncContext object that caches the request/response object pair. This AsyncContext object can be stored in a memory-repository and the servlet method may return immediately so that more requests can be handled. Additionally, the HttpServlet should be annotated with the asyncSupported attribute set to true so that the Servlet container does not commit the response when the Servlet method returns.
Servlet 3.0 would help standardize the asynchronous request processing API for the JavaEE developer community. But it would take some time for Application Servers to be JavaEE 6 compliant with stable versions of their product. The good news is that most of the application servers already support Asynchronous Request Processing with non-standard APIs and at the risk of sacrificing portability, one may use these APIs till then.
- Tomcat 6 provides a CometProcessor interface which can be implemented by Servlets.
- Jetty 6 provides Continuations
- WebLogic 9.2 provides AbstractAsynchServlet and FutureResponseServlet
- Websphere version 7 provides Asynchronous Request Dispatcher API
But the fact remains that most of the frameworks built around this Servlet model and based on the Core J2EE Patterns, be they presentation frameworks like Struts, JSF, Spring MVC or bridge frameworks like Seam, is synchronous and they have not yet advocated any samples using Asynchronous Request Processing paradigm. And most of the JavaEE developer community is heavily influenced by reference architectures and blueprints like the Java Pet Store application and does not follow out-of-the-box approaches.
In my opinion, this has been the result of the mentality that Servlet Containers are the be-all and end-all solution for HTTP server side applications and restricting oneself within the confines of the Servlet framework.
- For example, earlier versions of some containers used a thread pool and allocated threads-per-client socket connection and recycled used threads once connections closed. Of course, this means that there is a cap to the number of clients that can be supported with this approach since after a few hundred threads, there can be severe context switching. This model has been abandoned for good reason in newer versions of the same containers. Most application servers and containers now provide the thread-per-request model where each HTTP request may be read and processed by a thread from a thread-pool and after processing of the request, the thread is recycled back. But this goes to show that not all application servers / containers are made equal.
- The mechanism used by the server to read the request as part of the socket communication may also vary. Servers can adopt Native IO or Java NIO mechanisms to implement either the reactor pattern or the proactor pattern. The proactor pattern is more sophisticated, scalable and efficient than its reactor counterpart. True proactor implementations require extensive OS support and Native IO. Using Java NIO’s non-blocking read, a near-Proactor implementation can be achieved.
As in the reactor approach, the thread-pool invokes the Servlet’s threads but the read is actually being done from an in-memory buffer rather than the client socket channel.
Let us compare the two approaches.
In the reactor variation, since the application thread pool is reading from the actual socket, one saves on the memory consumed. There is no buffering up of data except for the actual request data that will be used by the application. When one is concerned about the possible memory consumption due to unknown request sizes, this would be an optimal approach.
On the down-side, consider that the clients are sending data to the server over a fragile medium with low bandwidth like GPRS. I happen to have some experience developing server-side communicators for GPRS. GPRS actually uses unused time division multiple access (TDMA) slots of a GSM system to transfer data, for example, when there are gaps and pauses in speech. This means that the transfer of data can happen very slowly depending on the voice traffic and moreover, it would arrive in bursts rather than a complete packet. The time gap between bursts of data is arbitrary, depending on the availability of time slots. So, if the reading is done by the Servlet in the context of the thread-pool, a precious container thread gets blocked till more data can arrive on the socket. This can bring down the scalability of the application. So, for low bandwidth transportation mediums, such a server could prove sub-optimal.
In comparison, the proactor variation would have much better throughput and scalability for low-bandwidth transport mediums since the event de-multiplexer part would take care to buffer up data in memory and the thread-pool would read and process completed requests from in-memory buffers. On the down-side, this may be unnecessary when you are assured of excellent bandwidth on your LAN or WAN and would be sub-optimal if you have memory constraints. Of course, this would require very careful implementation of the container or server with stringent memory handling and timeouts and users of such containers may have to read the documentation very carefully to configure optimally. If not, it could even lead to memory leaks.
Most Application Servers/Servlet Containers actually use the reactor pattern, because most web-site based applications can assume good bandwidth from clients and keeping in view the hazards of inefficient memory management. Personally, I was hit by this behavior when I used Weblogic8.1 Application Server and its Servlet Container for handling HTTP POST data from GPRS based clients.
In general, for designing server-side applications which only use HTTP as a transport medium, I would recommend using a proactor pattern-based approach. To handle memory constraints, a proper timeout mechanism should be implemented to detect idling of clients and cleanup buffers.
Sunday, July 4, 2010
- People believe that Application Servers and their Servlet containers provide some great sophisticated and almost magical services that they can never do without for any serious production based application.
- A very large community has generally built web-site based applications or has some web-site component for exposing their business to human consumers. If they are using and investing in Application Servers and/or Containers anyway, why not just use the Servlet container.
These are in fact, quite often valid reasons. But one must pause to think - is there no flip-side at all? Have we not lost out on anything at all due to this approach? And do the above reasons hold true for all HTTP server applications?
Not long ago, there was a time when people thought that EJB = J2EE and that every J2EE application must have an EJB container. In fact, I was one of them. Then one fine day in 2006 I came across a book - "Expert One-on-One J2EE development without EJB" (2004 Edition) by Rod Johnson and Juergen Hoeller, the creators of Spring Framework. That book completely changed the way I thought of JavaEE applications and I consider it a turning point in my professional career. There was an entire chapter - "EJB, Five Years On", which effectively dispelled all the myths I had ever entertained about J2EE and EJB. One by one, the chapter showed me that whatever services Stateless Session Beans (often the only EJBs used) provided- whether Declarative Transaction Management, Remoting, Clustering, Thread Management, Instance Pooling, Security and so on- there were alternatives and in fact, EJB was not the best answer to these concerns for all scenarios. 6 years after the publication of this path-breaking (in my eyes) book, the impact of Spring is for all to see. It revolutionized JavaEE development, the JavaEE6 specification is greatly inspired by it, a large part of the developer community has not looked at EJB since the arrival of Spring and it is understood that Spring shops are unlikely to move away from their favorite framework so that the community may now be split into Spring shops and JavaEE shops. (Read Adam Bien's weblog JAVA EE 6 IS NICE, BUT IT IS TOO LATE...).
I think the reason Spring framework and the book I mentioned above were so effective was that the creators of Spring tried to understand and discuss the philosophy behind the EJB2.1 and J2EE specification and then see whether there were any alternatives or better approaches. Not only that, there was a respect for the capability of the developer and architect community urging that middleware developers are not semi-competent to be unable to make decisions for their application requirements and they should not be deprived of control or ability to use alternatives.
In the same lines I would like to discuss about the need to use Servlet Containers for all HTTP server side processing. Let us keep our minds open and ask some real questions as to when and why we need Servlet containers and when can we do without them. An important pre-requisite is to have confidence and respect for our own competence as developers and architects to control the programming language and platform we have painstakingly learnt and developed our expertise in over several years.
An excellent book “Head First Servlets and JSP” explains the value of containers for server-side HTTP applications. It first challenges a Java developer to think of her responsibilities “if you had Java but no Servlets or Containers”. Then it expounds “What does the Container give you?” and lists the following –
- Communications support (i.e. listening on server socket and HTTP protocol encoding and decoding as well as HTTPs support)
- Lifecycle management (of the Servlet and the entire web application)
- Multithreading support
- Declarative security (by adhering to JAAS Security Standards)
- JSP support
One may add the following as well to the above list -
- standardized deployment contract in the form of war files and declarative deployment descriptor in the form of web.xml.
- A standard API to represent HTTP requests and responses to extract data from HTTP requests (headers, URL parameters, request/response body.
- Session Management
- Pluggability with major web servers (like Apache HTTPD) for reverse proxy, load-balancing with session-stickiness and failover.
One must be thankful to the containers and the Servlet specification for the standardization and the above fundamental features offered which help developers concentrate on the business problem at hand. Without this support, the huge amount of ground-work to be done would be daunting to the best of us. Some of these features are just indispensable for web applications - no questions asked. I say this also because there are a whole lot of GUI MVC frameworks like Struts, JSF, Spring MVC which have been built around the Servlet specifications. To use these invaluable MVC frameworks for churning out your web-sites, you would of course require a Servlet container.
Then what exactly are we attempting to discuss about? First, as architects or designers, one must find the key quality attributes required for the business application and decide which of the Servlet Container facilities we absolutely need, which ones we can do without, which attributes are we prepared to trade-off for more important ones and whether there are alternatives. As a young JavaEE developer/designer/architect, I was also among those who thought that Servlet Containers are THE solution for developing Web/HTTP Server-side Applications. By not considering the special quality attributes and trying to twist Servlet Containers and other JavaEE platform offerings, I had blundered and what was worse, for a long time I didn’t even know that I could have done better by adopting a different paradigm. I have since become wiser.
Before proceeding, let me make it clear that if you need to make a web site for thin browsers with HTML for human users using JavaEE platform, I am of the opinion that Servlet Containers would most likely be required at least to a certain extent. Though of course, now with the emergence of RIA technologies like Flex, JavaFX, etc, there is a whole new paradigm for web-site development.
However, consider the following interesting scenarios where there is scope for exploring alternatives to Servlet Containers-
- Say you need HTTP as a transport protocol. You have your own application level protocol in the form of some XML format or some binary protocol (custom or Google protocol buffers and so on). This application level protocol or format is just using HTTP as a means of transport instead of raw TCP/IP sockets. Typically this is done because HTTP is seen as firewall friendly.
- Or else you are using the RESTful facilities provided by HTTP - GET, POST, PUT and DELETE to provide a simple web based service. Once again HTTP is providing a convenient protocol especially in the form of name-value URL parameters to specify your query string and formulate your request. But essentially it is being used as a protocol and a format, not for displaying an HTML GUI page.
- Even for GUI applications, there has been a re-emergence of thick RIA clients enabled by Flex and JavaFX and Silverlight which take care of the GUI rendering entirely at the client side and seek business data from HTTP based server-side applications. Once again, we have the same premise - they don't need the server side to do any rendering for them, just to provide HTTP-based service and provide the data in some mutually agreed application level format embedded in HTTP request/response format.
- One can also include the AJAX/Comet requests which are frequently sent by clients to servers and have to be responded to in minimal time with maximum throughput and also scalability for handling numerous such requests. Often the client side script is capable enough of rendering or just un-hiding certain portions of the page after applying the correct values.
- Sometimes the medium between the client and server can be fragile like wireless (GPRS) and the transport may behave in strange ways - like sending chunks of data in packets and with very low bandwidth.
In my next posts, I will attempt to go under the hood and read between the lines to evaluate the Servlet Container features, for such scenarios.
Saturday, July 3, 2010
However, my project manager-cum-architect at the time (early 2006) was perturbed with the idea of trying out a 'new' and 'untested' architectural approach and as I had not much architecture experience at the time to counter the arguments, I gave in and used a regular Core-J2EE-patterns inspired 3-tier architecture utilizing a commercial application server's servlet container.
When the first generation of the complete solution was built, deployed and made live, I became more and more convinced that my first inclination to use NIO was justified.
Here were the reasons-
- I was building a communication server which used HTTP as a means of transporting the actual application data. I was not putting up a web site for human users with an HTML GUI. As such I did not need lots of the facilities of the servlet containers and was in fact hampered by a few restrictions as I will elaborate below.
- Pre-Servlet3.0 specifications only supports synchronous architecture and a thread-per-connection or thread-per-request model. Servlet3.0 specification has only recently been released and the majority of application servers are yet to comply with this specification. For more information, read the article Asynchronous processing support in Servlet 3.0. Our solution was built and deployed much before that. Of course, application servers also have non-servlet-standard APIs for asynchronous HTTP in their servlet engines (read article Asynchronous HTTP and Comet architectures). But I had no awareness of it at the time.
- The transportation medium was GPRS for which bandwidth is quite low. Moreover, in developing countries, the quality of GPRS is not as consistent as Europe or United States. Furthermore, our devices were mounted on moving vehicles which could move at very high speeds changing cells and GPRS connection could be quite fragile. Under the hood, GPRS is a packet oriented service offered on 2G networks and as such manages to send only a few packets so long as communication channels are not used up by voice communications. That means the entire HTTP request would arrive in packets or chunks or bursts over GPRS. Often we would also get HTTP requests that were incomplete and would eventually time out. For example, if the GPRS device was trying to send 512 bytes our server would get only 78 bytes and then the request would time out. In the thread-per-request model, this would mean that one precious thread of the container would be blocked just trying to read the complete request and would eventually timeout. Depending on the read timeout value set in the application server, this could mean as much as 30 secs which is a huge amount of time. What we required was an engine which could do a non-blocking read and assemble the entire HTTP POST request in memory and only then call a servlet or a handler to do the rest of the processing. In other words, we needed the proactor pattern instead of the reactor pattern. And while lots of application servers use NIO or native IO, they internally use the reactor pattern!
The case for NIO was thus clear. But I still was having a hangover of the Core J2EE patterns and the numerous MVC frameworks built around Servlets for presentation tier. So I would probably have used NIO frameworks in the typical synchronous style. Then along came our seasoned architect who reviewed our solution and proposed a radically different - distributed and asynchronous architecture. I realized that if I used a synchronous approach with the reliability that data sent by the clients was getting successfully processed by the backend, I would have to block my thread till the backend processing of the data sent by the vehicular devices was done.
The other possibility would have been to use reliable JMS, but with the store-and-forward technique of reliable JMS, we would have probably sacrificed the desired high scalability and throughput for the reliability. Besides that was rather like arm-twisting the architecture to comply with servlet container constraints.
What I needed was an HTTP engine which would not impose the following restriction of pre-Servlet-3.0-
“Each request object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method. Containers commonly recycle request objects in order to avoid the performance overhead of request object creation. The developer must be aware that maintaining references to request objects outside the scope described above may lead to non-deterministic behavior.”
An engine that would allow me to do this-
I had had experience building NIO TCP/IP servers with Apache MINA but the MINA releases were taking a long time and HTTP protocol was not directly supported. Yes, there had been an asyncweb project from safehaus built on MINA0.8, but the project had been migrated to Apache and was not developed further to be compatible with latest versions of MINA. It was a big risk using Apache MINA+Asyncweb.
Grizzly, I found, had too many options - plain NIO framework, Embeddable HTTP Web Server, Embeddable Comet WebServer, Embeddable Servlet container. I was simply baffled. It seemed a great learning curve and I had very little time. Moreover, this post from Grizzly's creator alerted me that Grizzly would by default, use the reactor pattern rather than proactor, which I was not comfortable with for my scenario.
In contrast, JBoss Netty's HTTP support was short and sweet - with no-frills. All I had to do was to use the ready-made HTTP Protocol encoder and decoder and attach my own HTTP request handler. After all, HTTP for my scenario was just a protocol over TCP/IP sockets carrying the application data in its body.
Within no time, I had my prototype ready and I was confident that I was on the right track. The performance test reports were extremely heartening. But could Netty fit in my scenario?
Turns out that it far exceeded expectations. That simple HTTP communication server I built using Netty could sustain the concurrent load of 10000 clients with the desired throughput of 500 requests/sec on a single desktop PC with 1 GB RAM and 1 CPU in our lab. And I am not kidding! It was celebration time for all of us. The memory utilization was well within the 812MB heap size we had granted to the JVM and the average CPU utilization was 78%. In other words, we were utilizing the full resources that the machine offered.
My reasoning for building a scalable HTTP communication for GPRS devices with NIO, asynchronous HTTP was thus vindicated by using JBoss Netty Framework. Thanks to Trustin Lee for making this splendid framework.