My little project in building a virtual affiliate store around a large number of product feeds is progressing slowly, mainly due to other real work. But it has made some real progress.

I already have the product feeds (all ftp based) loaded and indexed in various ways including raw text search (inverted indexes based on stemming and lots of stopwords). I have a lot of set operations done as well. Everything runs in memory only (this is the read-only portion of the system). This memory-based query engine is designed to be distributed (accessed via REST) running on top of an embedded Jetty server and using XStream to marshall the data.

The biggest problems so far are (1) building a universal category tree and mapping all the merchant products into it (everyone has their own scheme) and (2) decided what type of interface to provide.

The first issue is basically an informational architecture problem. I looked at a lot of shopping sites and tried to discern what motivated their choices, then built myself one. Sadly I can't see any way yet past manually mapping the store categorizations to mine and then dealing with unknown new ones as they appear. The real solution would be to analyze all the information on a product and build a system capable of mapping the data automatically. That will become necessary when I add larger merchants like Walmart (850K products) and iTunes (2M+).

The interface question is interesting as I see three choices (1) pure ajax (2) pure html and (3) both.

A pure ajax site has the advantage of being much faster to use, given a UI that emphasizes many different ways to slice-and-dice the products. The downside is that Google sees nothing much and you don't get the discovery from people searching for products via Google.

If I provide an interface that ultimately lists all the products from all merchants in a discoverable URL scheme (e.g. /products/flowers/roses/) then all the data will be found in search engines (baring being punished for duplicate content). The downside is a ton of processing as the engines devour all the pages (between Walmart and iTunes that's 300K pages or so). I can cache this information (perhaps as zipped HTML) but it's still a ton of bandwidth and time.

I would like to do both so that people can find the "store" via Google, but still have the benefit of alternate ways to find (and build RSS feeds from) stuff in the store. Performance of all this is very predictable and not too difficult to scale (most of the work is happening in memory time). If I choose to support reviews and comments then I have to build a more robust database architecture (something to preserve the information); currently the query engine is totally read-only and updated once per day (the data feeds update that way). Currently I can load 3M products per hour into my Dual G5 development box.

So this is an interesting project so far, but I need to eat and pay bills so I can't work as much as I would like.

There are many shopping sites but I see directions that haven't been explored yet. All my investment is my time so far, so once it goes live (whenever) it doesn't have to big huge to be successful.

More later.

My Tags: web applications store java ajax

Found this series of articles on the yahoo UI blog. Some interesting stuff. Pardon if you all read this already. There is so much information on the web it's easy to miss it the first time around.

Performance Research Part 1: What the 80/20 Rule Tells Us about Reducing HTTP Requests

Performance Research Part 2: Browser Cache Usage - Exposed!

Performance Research Part 3: When the Cookie Crumbles

My Tags: links web applications performance

The project I am working on at home will require an architecture to support a high volume of ajax calls slicing and dicing data from a fairly large database. Since the data is only updated once a day, there are interesting options on how to build the system.

Web applications can be roughly divided into two types: mostly read and some write. The first type is mainly used to show the user some information based on some type of query, which could be explicit (like a query by example form) or more commonly based on a UI that gives them options. The second may do the same but allows updating of information by the user. It's rare for an application to not have some update features so what type it is may be hard to pin down. I view the difference as whether the user can update the information they are querying.

Amazon, for example, has a lot of write features (ordering, building lists, commenting) but they mostly don't affect the catalog of items (other than availability).

Digg may spend most of its time showing similar content but users are actually modifying the data as they interact with it.

The site I am trying to build will be more like Amazon, in that the majority of the information only changes rarely (like once per day) and that other interaction is peripheral. This makes it easier to use all sorts of caching strategies.

When I worked out the problem in 1998 with Consumers Digest Online we were using WebObjects, which would run separate processes (called an instance) in the same server (or multiple servers). Originally my digested search information (about 20MB mostly a big tree structure and associated indexes) was loaded into each instance, which somewhat limited how many we could run in a single server box. Later we modified the cache data (which was all static data) to run inside a shared memory space (HP/UX). This architecture then looked like this:

We were able to support as many as 2000 simultaneous users with this structure, even though the search engine was fuzzy with a lot of derived data searches. The average response time through the cache for any query was subsecond. There were something like 40,000 packages of features, 20,000 products, and an almost infinite combination of potential results. You couldn't cache individual results as each search was unlikely to ever appear again. The data changed once per week, and was processed for the caches after the new database was loaded. All display data came from the database directly.

During my year of working on projects for Sabre I wrote a white paper for them on how I would build a replacement for their monster reservation system (with its 8000-per-second query requirement) based on a similar but larger architecture. I don't think it went anywhere but eventually they built something with a more traditional database caching architecture. In their case the flights don't change too often but the reservations do so it is more of a type 2.

For my new application I will be managing a wide variety of slice-and-dice searches through a much larger space (iTunes for examples has 2M+ tracks) but at least the information isn't derived from the database data like in the CDOnline system (with its complex required and disallowed package combinations). The biggest issue is that in building an Ajax from end you need fast response even under heavy loads, since queries are 95% of what the application will do.

The architecture for this system looks something like this:

The point is to replicate the cache servers which have enough RAM to store the data required for searching all in memory. As the data in the database is updated only once per day, it can be optimized for the fasted possible searching when loaded into the cache server. Of course you could build something more traditional which would allow the database to manage the caching or use a caching framework (of which there are plenty both free and for-pay).

In my case I want to manage the searching myself as most of it involves deeply nested trees, which are hard to optimize in a relational database (Oracle has some benefits here but I've never used them). RAM is always faster than hard drives (at least until the new Flash drives get cheap enough) and since I know exactly what I will allow the users to do, I can build the cache precisely for those needs.

The pleasant thing about this architecture is that I can not only grow it as needed, but I can also distribute it over multiple data centers if necessary. This is basically what all of the big companies do. I expect to be able to manage as much as 100 transactions per second, all without any massive investment in servers (money I don't have anyway).

One other note is that I use Jetty as my appserver, which is really fast for Ajax calls with its continuation based architecture.

My Tags: ajax caching web applications

A few more points to consider when building or managing a web application project.

1. It's been __ months and there's nothing to see

Seeing is believing, and in a web application it shouldn't take long to get something running. If a significant project has no visible progress in the first third of the development schedule if you are using SDLC/waterfall or the first few iterations of an agile project, then you should start to worry. If the team has been working for 6 or 9 months and there is no URL you can go to to view the current application running, then odds are the application isn't ever going to be complete.

You would think that this is obvious, yet I have seen it happen enough to think it's not that uncommon. Sometimes it's the development team that is incapable of making progress due to inexperience or lack of ability; even more commonly it's not their fault as the basic requirements keep changing (and not in an agile way) or management can't make needed decisions.

This isn't only a problem with web projects of course, but in a web project the browser already contributes a lot of functionality; all you need is some static HTML to demonstrate what is coming. I have heard people make excuses about building frameworks or firm foundations or working out detailed designs, but the only way for someone other than the developer to judge progress is by seeing something run.

The side benefit to seeing visible progress is that the customer can get an idea of what is being built and contribute to the shaping of the solution they are paying for. Even if you are not doing a agile project, having the customer be able to make comments on a live piece of software is a good thing (even if it will torpedo your schedule); after all it is meeting the needs of the customer that is the whole point of the project.

For developers I can only add that it is in your best interest as well to work on both the front and back ends of a project. Management and customers really appreciate seeing something work far more than dry project timelines and status reports.

2. You keep your source code where?

You would think in this modern era that everyone uses source code management systems (SCM), yet I have seen fairly large companies where source is "managed" on shared directories, or even kept on developer's hard drives. Even if they have an SCM there is often no attempt at organizing the repository, which is treated as nothing but a file dump. One place I worked at briefly on a contract had a manager who withstood 3 months of arguments before allowing the developers to set up an SCM system, arguing that using one would add too much time to project schedules.

Your SCM system should hold all source code, scripts, schemas, etc; basically everything necessary to build the entire system from scratch. It should also be backed up continuously and carefully (another thing I have seen people fail to do), and tested regularly. The repository should be designed (and the software chosen carefully) to make it easy to manage multiple versions of all projects for all contributors.

Many times only certain people will use a repository, such as the programmers, but other folks like DBAs, web designers, and QA will use other systems for their resources. It may not be as convenient for them but keeping all of the parts necessary to build and maintain a web project in an organized place is a huge benefit. Sometimes this can be a big pain, such as when I used Documentum in my projects I couldn't easily save the state of the DocBase in any meaningful way (and my employer was too cheap to pay for any real possibilities). Even keeping snapshots of external resources at regular intervals is better than nothing.

The question to ask is, if our place of business burned to a crisp, could we rebuild and continue work on projects if all we had was a backup tape of the source repository?

My personal favorite is Subversion but there are many others.

3. Management says 'let's worry about systems architecture when the app is finished'

Another wonderfully bad idea. Let's design a new car and worry about the engine when we're done. Planning on how your application will be served up should begin at or even before the coders get started. What usually happens when you just throw the application at your existing environment is lost connections, bad customer experience, crashes, and hours of painful overtime while you try to scale or fix things live. This tells your customers your application is broken (even if the code is perfect) since most people don't know or don't care how the internet works.

Today I tried to log into my Linkshare account and the login page didn't come up (different subdomain than the rest of the site); instead I get a warning from the browser that the server didn't exist. They fixed it after a while (not sure how long it was out) but still, an ordinary person might think they had done something wrong or the site was gone. Imagine you are a financial services company holding my money for retirement and your website goes up and down all day, how would I feel about my money and you?

I actually worked at one of these once and would not put my money there for anything. One of their major web applications did go down at random points every night (due to a manual backup that killed database connections with no warning) with no meaningful message to the users.

4. Management says 'let's worry about testing when the app is finished'

Ditto to the last section. QA doesn't start after the developers go on vacation. It starts the same day as the development does (or even earlier). Not only should testing be planned early but the testing environment (see the last section) should be developed early as well. The aforementioned financial services company failed to spend enough money on the QA environment to mirror the architecture of the production system, so when they rolled out a new customer portal, people started seeing other folk's financial data on their pages. Oops. The app worked fine in the limited QA environment, which had a single server. The production environment had two, and the problem involved id's being shared that were not unique in the cluster.

Oops! is not a development methodology.

My Tags: it web applications project management

I get so tired of going to websites that provide interactivity and discovering they suck megatons. This is especially irritating in Fortune 500 companies (like Bell/AT&T/whatever they are called this week with yellowpages.com). None of this is rocket science, yet I am amazed at how companies with deep pockets manage to produce drecky sites. This is your public face, would you go on a date with mud on your face and spinach in your teeth?

At the risk of giving away all of my (not so secret) secrets, here are some useful things to do.

1. Put javascript and css in their place

Javascript and css should be kept in separate files and not embedded in your web pages. I don't care that your IDE can parse and syntax color 8 simultaneous languages in one file, it's hard to read, hard to debug, and brittle when there are multiple people editing the page. I'm ambivalent about adding event handlers upon loading instead of in the html itself but it's not a bad idea. If you want to avoid linking to files then include them in the html using your language or framework include feature.

2. Use compression

All web servers and browsers are capable of dealing with compressed content. Turn it on. It doesn't cost anything, saves bandwidth, makes dialup users happy and dogs wag their tails. You can save more by compressing your javascript files than by removing all the spaces manually. In any case it's easier to code something you can read.

3. If you put a DocType in your page, validate against it

Of all the things you find in the wide world of web, crappy HTML is by far the most common. Why put a DocType in every web page and then serve up invalid content? I always work in strict XHTML even though it isn't served up that way, simply so that I can be assured it's well written. It's a discipline thing. Use the web developers toolkit or Firebug and keep checking all the time. I don't care if IE 6 shows your site correctly. I bet IE7 doesn't. Who would buy a car that only ran on blacktop and not on concrete roads?

4. Don't air your dirty laundry in public

If your web application is properly tested (you do have QA right?) the end user should never see an error message. God forbid your code throws an exception and vomits all over the end user with debugging information. One site I went to threw a monkey wrench and then showed me a page full of every server and application level variable it could find, and then for good measure included them in a comment in the web page. I have also seen so many messages like "could not connect to database" and "timeout occurred reading database" I felt like visiting the competition. And usually did.

If you have to demonstrate poor QA and show an error page at least put it in context of the site and don't scare your customers with it. Architect your logging system to track problems and provide information to the programmers. Don't be like one employer I had who hid the logs from the developers out of some stupid security paranoia.

Whatever you do, don't include your test code and development comments in your web pages or javascript. yellowpages.com happily loads their unittest.js file on every page. Not that I want to test for them.

5. Use modern web design

Table based sites are so 19th century. Yet look around the web and see the fruits of old web design. CSS is your friend. Dreamweaver is your enemy. If your web designers are unable to build sites by hand using modern techniques and style with CSS then don't give them Dreamweaver until they can. There are millions of sites devoted to modern web design so there is no excuse to not learn how. If you can't write HTML by hand then reading the articles won't make an impression. Web design is programming + art. Accessibility, usability and maintainability are all made easy doing things the modern way and treating it as an engineering task.

6. Support all modern browsers

There aren't that many anyway. Other than IE6 and IE7 all the others are pretty similar. Saying you only support IE is stupid these days as there are two fairly different versions, with IE7 being somewhat closer to the others (yet screwed up in its own way). Code to the standards, adjust for IE6 and IE7.

Would you shop in a store that said "Whites Only!"? No, and I don't want to shop in an online store that says "IE only". Even worse are the places where they don't tell you you're not supported until the last minute. It's like going to a store and after filling up your cart, having the cashier tell you "we don't serve your kind here". Not real smart, telling customers to get lost. They will.

7. If your application has special needs, check up front

If you use javascript, flash, cookies, or something special (like a plugin) make sure you check on whatever the first page the customer sees and if you absolutely can't do anything with out it at least let the customer know up front. If you can work without it but have to degrade service either make it transparent or let them know what doesn't work. I don't want to have to explain your site to my mom.

8. Test your application usability with real users

It's amazing how many sites out there are so hard to operate. Sometimes I wonder if anyone other than the engineer actually used the site for its intended purpose. If you offer search, can anyone find useful stuff with it? Is your navigation clear? I went looking for some security hardware for my door and wound up at a Honeywell site, wherein I couldn't find anything useful and gave up after several fruitless moments. Then I went to their competition. Engineers, QA people, managers and the like are not users. If I can't find something I want on your site, then how will my mom?

9. Use a real database

Access is not a real database. Neither is Foxpro. If you can't afford (or don't want to buy) something like Oracle then use MySQL, or even better Postgres (or my personal favorite H2). Access is a toy. Would you drive a lawn mower on the highway? I've always been amazed at companies that have mission-critical information stored on someone's desktop in an Access database. And don't get me started on people who manage data using shared Excel spreadsheets.

10. Don't use platform specific functionality

ActiveX controls are the most evil things you can use. They lock you into a platform, and yet don't work easily in IE7 anyway. Just say no. If you have to Flash is always a better choice since it is available to most people.

11. Realize your customers don't all have T3 lines

An average directory page on amazon.com is around 350K and has 43 separate HTTP requests. Yikes. At least 20% (or more) of people in just the US still have dialup. One friend works by day on internet backbone switches and goes home where he has nothing but dialup available.

Compress, combine and cache were possible. I found that my biggest bandwidth hog was prototype.js. Once I compressed it and delivered it that way it almost vanished from the list. Just because I have a 6MB/sec cable line doesn't mean you aren't paying for it on your end. Most browsers only do 2 connections at a time so why make it hard on your customers and yourself.

12. Deal about accessibility and internationalization early

If your webstore or product site only caters to rock-climbing englishman, maybe you can get away with ignoring the needs of the disabled and those who speak other languages. Think about this before you build.

13. Test early, test often

You do test your web application? Testing should begin long before it's complete. Test the code, test the database, test the usability, test the environment and infrastructure. Test as if people would die if you didn't. Test as if your customers would all leave you if you fail (they will!). Test on all major browsers. On my Macbook Pro I can run IE6, IE7 (I need a bit more disk space), Firefox, Safari and Opera. I am sure a Fortune 500 company can afford more.

If you do all of these things then life will be good, people will love you, you will get rich and everyone will shower you with goodies.

Maybe yes, maybe no; but at least you can know your web application doesn't suck.

My Tags: web applications programming html it