For anyone interested in groundwork who may not be a web developer, the following discussion explains how web-based applications work and a gives an overview of the decisions that a developer must make when designing and implementing a web-based application.

A reasonably knowlegable person might think of the web as a large collection of web pages, served to their browsers over the internet from web servers around the world. This does describe a large percentage of the web, but it's not entirely accurate.

There are actually 3 types of web-based content: static, semi-static and dynamic.

Static content typically exists in the form of files on a web server; for static content to change, a person must manually modify the file. Static content is good to use if the content needs to change very infrequently or if only small amounts of it need to change. Good examples of static content are personal home pages or corporate web sites.

Semi-static content is similar to static content except that the files are modified or generated periodically by software at the web site instead of by a person. Static content is good to use if the content needs to change regularly, but does not need to be updated in real-time. News sites or online catalogs are often semi-static.

Dynamic content is content that reacts to user input or displays information at or near real-time. Web based applications such as online games or web-based training systems are good exmaples of dynamic content.

There are actually several different kinds of web-based applications, but almost every kind can be categorized into one of two general architectures: client-server applications and client applets.

In the client-server architecture, a browser plays the role of the client and displays content that the server generates. The server is responsible for the processing while the client displays the interface.

In the client applet architecture, the entire application is downloaded to the browser and runs locally. The application may periodically contact external servers for data, but the bulk of the processing is performed by the client.

Both architectures have strengths and weaknesses and neither is appropriate for every application.

Since the groundwork class library provides a means for writing client-server applications, this document focuses on that architecture.

Typically, five components compose the client-server web-based application architecture: The user, the browser, the web server, the application and the database. These components are combined in series such that each component interacts with the next component in the series. In some implementations, the web server and application are actually combined into a single component. In others, the web server, application and database are all three combined into a single component. So variations on the theme exist, but fundamentally the architecture is the same.

A point worth mentioning here is that the interactions between the different components are highly transient in nature. Each component makes a request then waits for a response or waits for a request then sends a response; with relatively long periods of idle time between activities. This behavior is fundamentally different from the behavior of other types of applications and has a signifigant effect on application design.

When a user interacts with a browser, he or she requests an address. This address is more formally known as a URL or Uniform Resource Locator.

A typical URL might look like this:

http://www.firstworks.com/groundwork/documentation/index.cgi

But URL's can be much more complex. The following URL illustrates every possible URL component.

http://user:password@www.firstworks.com:80/groundwork/documentation/index.cgi/default?variable1=value1&variable2=value2

From left to right, the components of the URL are:

• protocol - Indicates what kind of server the url is attempting to access, usually one of http, https, ftp or news.
• user name - For pages that require user/password authentication.
• password - For pages that require user/password authentication.
• address - The DNS name of the server.
• port - Each service that a server provides listens on a different port, http servers usually listen on port 80, https servers on port 443, ftp servers on port 21, etc. The protocol implies a port, but it can be overriden using this component.
• path - Analagous to a folder or directory, indicates where to find the page on the web server.
• page - The filename of the content we're trying to access.
• path info - An infrequently used component that can be used to pass data.
• get string - A frequently used component that can be used to pass data; composed of encoded name/value pairs, seperated by ampersands.

Some of these parts are probably familiar to the average web surfer, while other parts are likely to be unfamiliar. If you've ever seen a long, strange looking web address, then it probably contained some of the optional parts. Before URL's became marketing tools, long, ugly URL's were much more common.

Communication between the browser and the server is done over a low-level protocol called HTTP or Hypertext Transfer Protocol. The browser primarily uses HTTP to request information from the server and the server primarily uses HTTP to respond to the browser.

Along with each unit of information sent to the browser is a string of metadata called a MIME type. The MIME type defines what kind of data is being sent. MIME stands for Multipurpose Internet Mail Extensions. It was originally designed for identifying email attachments, but was later incorporated into the web. There are actually two parts to a MIME type: the type and subtype. The type is a general description; such as "text", "audio", "video" or "image". The subtype is a more specific description; such as "plain", "html", "gif" or "mpeg". The full MIME type is given as type/subtype; as in text/plain, text/html, image/gif or video/mpeg. Since browsers handle different types of information differently, the MIME type is very important to the browser.

The most important MIME type to most web developers is "text/html". HTML stands for Hypertext Markup Language. Units of HTML are often called pages. When a browser receives a page of HTML, it renders it in a browser window. The HTML language has syntax for defining the look, feel and layout of data as well as for defining relationships between the current page and other information on the web.

A commonly used protocol for communication between a web server and an application is called CGI or Common Gateway Interface. This protocol provides a message passing interface based on environment variables, standard input and standard output. Web based applications based on the CGI protocol are often collections of standalone binaries or scripts called CGI's. When the web server receives a request for a CGI, it forks that CGI off as a seperate process. The web server then passes it the path info and query string and waits for a response. The response is usually a page of HTML which the web server then passes back to the browser.

CGI scripts must be interpreted by a scripting engine. Most scripting engines interpret the script and load modules to provide additional functionality. Scripting languages are usually both easy to learn, and portable across operating systems, but offer poorer performance than binaries for a number of reasons. Scripting engines and modules are usually large, general purpose binaries which take a long time time and lots of memory to load and unload. Interpretation of scripted code is also slower than binary execution. And finally, scripting languages often make less efficient use of memory than compiled languages.

Binary CGI's typcally perform better than scripts. Primarily, since execution is faster than interpretation, binaries just run faster. And, since a binary can be statically linked and stripped, it can also be made into a very small, very fast, self sufficient unit. Additionally, compiled languages give talented programmers the capacity to manage memory more efficiently than scripting languages do. Binaries do have a downside though. They have to be recompiled or sometimes ported to run efficiently (or at all) on different operating systems or hardware. Compiled languages are also more difficult to master than scripting languages and often lack the high level functionality that make scripting languages attractive.

In architectures where the web server and application are combined, the web server typically forks off a scripting engine or binary module as a thread of the web server itself. There is no need to pass any information to the new thread, since threads have access to the address space of the web server itself. Threads are also less costly to fork than processes. For these reasons, this method is generally considered to be more efficient than the CGI method. It has it's share of pitfalls though. For one, server API's are proprietary and don't port to other servers. Additionally, on architectures that don't have kernel support for threads, the threads only run during the time slice available to the web server, so the priority of the web server must be carefully tuned so the operating system context switches to it often enough to address to the number of hits it's receiving. The more general purpose the machine is, and the more seperate, active processes are running on it, the more difficult it is to get good performance out of threaded server API based applications. This issue is usually solved by having a machine dedicated solely to serving web applications. The final pitfall is that every module in the entire application must be loaded when the web server starts, even if some of those modules are rarely used. This can require lots of memory.

Aside from server API's and CGI's, another approach to writing web-based applications is the web application server approach. This approach is difficult to describe because the term "web application server" has become sort of a catch-all for any application that isn't CGI or server API based. Usually a web application server consists of a small, fast, special purpose web server with it's own special purpose API and often a GUI interface design tool and file manager. Since web application servers are usually special purpose, they are typically very good at doing some things and very bad at doing others. They also tend to be very closed systems that run on a limited number of operating systems and interface with a limited set of databases. Rapid application development is promoted by web application server development environments, but developers can quickly run up against the limitations of the environment and frequently have difficulty overcoming those limitations.

Most web-based applications need to access some kind of database. This may simply be a bunch of flat files arranged in the filesystem or it may be a SQL database running on a different machine altogether. Either way, the application must be capable of speaking the appropriate language (via API calls) to whatever database it needs to talk to. Sometimes, the database exists long before the web-based application that accesses it does. Consequenlty, the availability of API's for that database can be a limiting factor, and drive every decision that has to be made when developing the application. Further detail in this area is way beyond the scope of this discussion, but since the bulk of web-based applications access a database of some kind, it's certainly a subject worth touching on.

Assuming that the web-based application is either CGI or server API based, a few fundamental decisions have to be made about the general structure of the application before development can begin. A fairly common inclination is to write a single, large script or binary that does everything the application needs to do and selectively call parts of it to deliver each page. This is a bad idea from a performance perspective because in transient systems, the time it takes to load, unload and execute are all exaggerated by larger scripts or binaries. Smaller is generally better, from a performance perspective, so writing seperate, optimized scripts or binaries to deliver each page of an application is usually a better choice than a single monolithic app.

Each page-delivering component has the following life cycle:

• Launch.
• Receive information from the web server.
• Send an HTTP header back to the web server.
• Send a response to the web server.
• Shut down.

The launch stage involves the forking of a process or thread, some memory allocation and a filesystem read to load the script or binary into memory.

Receiving information from the web server involves reading environment variables and/or standard input in the case of a CGI or directly reading web server memory in the case of a server API thread.

Where does the web server get the information from? The browser passes some of it using HTTP and the web server defines some of it by itself. HTML forms take user input and pass it to the web server through one of two methods: POST or GET. If the web server receives information through the GET method then it passes it to a CGI by setting the QUERY_STRING environment variable. If the web server receives the information through the POST method then it passes it to a CGI through standard input.

The next stage, sending an HTTP header back to the web server should be done early in the execution of the script or binary. Some web servers won't wait long before killing a process if they don't receive a header from it. The simplest header is an HTTP statement of the page's MIME type such as:

Content-type: text/html

More complex headers include other HTTP information. To process content, browsers require more than just a statement of the MIME type, so when the web server receives the header, followed by 2 carriage returns, it takes a look at it and fills in any missing information. When using CGI's, one way to speed things up is to send a complete header and force the web server to just send whatever it gets to the browser. This is called sending a non-parsed header. To let the web server know that the CGI is going to send a non-parsed header, the filename of the CGI has to start with "nph-".

At this point, the browser is waiting for some content to display. Typically, this is the point where you query your database, do some calculations or perform other magic based on the parameters passed from the browser and format the results using HTML. CGI's will output this HTML directly to standard output while server API based programs will use API calls to write the output back to the browser.

The shutdown stage is the simplest of all: memory is deallocated and the component exits.

Applications that are developed inefficiently ultimately cost more to deploy.

Due to the transient nature of web-based applications, a very simple mistake to make is underestimating the capacity of an application to consume system resources. When developing and testing a web-based application, even the most inefficient implementation will run very quickly compared to other types of applications. You might be inclined to think then that since the components of an application are so small, take so little time to run, and require so little memory; that you can write them in whatever language is easiest to learn, run them on whatever old computer you have lying around and use lots of fancy graphics on every page. If the application is rarely run and resides on a machine which hosts other rarely run applications, then you might be right. If the application is destined to get a lot of hits though, then it's possibly the worst mistake you can make. While individual hits on an application may consume undetectable amounts of system resources, hundreds of hits per second can grow that insignifigant quantity to monstrous proportions. The choices of server and network hardware, operating system, filesystem, languages, API's, scaling techniques and countless other factors should be driven in no small way by the analysis of the level of activity that the application is likely to receive.

In the end, throwing in more hardware to improve the performance of an improperly designed application will reach the point of diminishing returns; and it will have become costly than long before that.

Hopefully, this discussion has shed some light on an unfamiliar topic for non-developers and struck a familiar chord with developers. For more information, press your browser's back button and proceed to the next discussion in the list: Introduction to Groundwork.