The Most agile Project I Ever DidAugust 04, 2015
When people think of Agile they usually think of a specific process like Scrum or Kanban. This story is more about dealing with continuous change, driven by the CEO, in what seemed insane at the start but actually ended up being a lot of fun.
This was three years ago around the time of iOS 6 beta. I worked in the mobile team for a well known travel company (now sadly just a brand of our biggest competitor, nothing here exists any more). We had an iOS app supporting all of our product lines (hotel, flight and car), an iPad app we had inherited from another office that had been closed, plus mobile web and a mobile API that made our main web app usable by our mobile apps.
As usual I don’t mention who I worked for but it’s not hard to figure out.
On a Friday where our manager was about to leave on a 3 week vacation halfway around the world, and with a week to go on a major release of the iPad app, he came back from a meeting with a white face. The CEO had been talking with a board member the day before who wanted to know why we weren’t competing with a hot new startup that was getting a lot of press and only sold hotel rooms for the current day. Yeah, them.
The CEO said “sure we can have an app in the App Store in three weeks.” That’s why our manager had that sinking feeling. Of course three weeks means two weeks since Apple usually takes a week to approve a new app. We knew it wasn’t possible.
Naturally we were told not to start yet either, management first had to figure out what brand the app would be released under—that would determine the design. Our manager told us to finish our iPad work first and then left the country.
We had an American brand and a European brand which operated mostly independently and had radically different design philosophies. This seemed strange since the app would only be released in the US.
Finally on the next Thursday the management arguments wound down and they picked the European brand which made no sense whatever, but oh well.
We had just hired a new designer for our mobile team who could also do a little coding in Objective-C and knew Interface Builder, so he and the two of us iPad programmers would be doing the work, plus a Java dev would do whatever mobile server side changes we needed and interface with the back end folks.
So now we were told to start building … what? We really had no idea other than the obvious, book rooms for the current day. Now we already did that in our existing app, in fact 70% of our iPhone app bookings were for today, but what was desired was something more like the competition, streamlined, and only showing rooms vaguely defined as special. What it should look like or how it should flow or even who would use it was unknown.
What we did know was the the following Thursday our test lab had already booked people to come in a try out the new app! Yikes!
So that weekend our designer started guessing about a design and throwing it into IB, and I had to figure out how we would be calling our API. Now I had two choices, the existing iPhone app code or the newer iPad app code. The iPhone app had no separation between UI and API and was impossible to break apart. I knew the iPad app code was not very good either but at least I saw it was possible to carve it up. It used a common framework called RestKit (very old version) and the code—which we didn’t write, some third party did—wasn’t great but it was the only option. Trying to build new code was not possible in this insane timeframe. So I surrounded the ugliness with a new API so at least we could change it later.
On Monday we got together and put a whole app together and starting connecting things between the pages and the API and other necessary business logic. The third programmer, the designer and I then worked the first 3 days to get as far as we could. Note we had no special data, we were just using the existing hotel search to work with.
The back end changes were problematic. Our main web app only supported a single business unit and since this would require new analytics and tracking and customer service support for the first time they would have to figure out how to change their code to do this. At this point they had no idea how long it would take.
We forged ahead to get ready for Thursday with whatever we had. On that morning it was actually running with search, list, details and a rudimentary checkout page, enough so that the 5 or so testers could look at it and give feedback. In between users we were making changes based on the previous person’s comments, and fixing some bugs! They also had them try out other apps including the target.
By this time it was obvious to everyone that the app would not ship the next day! I volunteered a guess at two months. The CEO from what I could tell seemed OK with this.
Now as we were starting on this crazy project, one of the process guys decided it was a good time to have us try out Kanban! Our mobile team was extremely Lean (the rest of the company was nominally Scrum) so this seemed a strange time to do something new. So he trained us–yeah in the middle of this first week— on the tools and had the project manager set the project up. We thought this was nuts, Kanban cannot be used in a blitzkrieg project where no one knew yet what the app should do or look like. We decided to play along with lip service. More on this later.
What Does It Do?
Everyone was excited about the project and we had to start making builds for every executive to try out. I had to collect UDIDs from all of their phones. Everyone had suggestions.
One of the major issues was our back end services were abysmally slow. Our normal app and web process was the user entered a date range and a location which triggered a search for hotels with available rooms for those dates. This performed terribly; on average it took 10 seconds to return a list of 150 hotels from our 100,000+. Our competitor had only a few hotels in each city so search time was zero; you just picked a city and there they were. Ten seconds may as well be forever.
So we thought about limiting the search to a subset of hotels. Sadly the search code on the server was such a convoluted mess that all we could do was ask for fewer hotels to speed it up a bit. Not good enough.
The other issue was our typical enter-dates-and-location was too much effort. So we decided to follow the competition and up it a little by having the user pick today, tomorrow or the next day in some fashion. Our product manager had a bright idea to figure out what our top 40 markets were by rooms booked. We would then provide just a list of these cities in the app, plus a local search where we would take the time hit. The user would then pick the day and a prebuilt list would be shown to pick from to see details on that hotel. If a nearby search then all we could do was run the regular slow search call. Unlike the competition we would allow the user to stay up to 7 days.
For the cities every few hours we would run all possible searches, 40 X 3 X 7, and cache the results in our mobile server. This way the results would come back instantly.
Figuring It Out
So now we were getting somewhere. The user would pick a day and a location and get a list of possible hotels to pick from. Once they had looked at the details and made a selection of a hotel and a type of room we gave them the option of picking more days up to seven. Sometimes this would fail since the chosen room would not available or at least discounted the entire time and the user would go back to the list to pick another. Since hotels are generally only 70% booked for this evening most of the time it would work.
Now the competition featured discounted rooms (in fact they negotiated rates and rooms every morning) so we had to try to do something similar. Our business was of course a seller of all types of hotels with many kinds of discounts including opaque (think Hotwire) but the hotel search only returned everything. So our server programmer built a filter that looked for any hotel that had a room for that date or dates with any kind of discount of at least 20% off of what we considered regular price. This required some back end changes to provide more information. Basically it was a hack to show better prices by eliminating all the regular priced rooms. Occasionally for a city for a day there would be none because the back end was notorious about returning empty lists.
So all we changed in our mobile API was adding the alternative search and a couple parameters to other calls.
Now that the data was taking shape we were still struggling with the UX (how the user interacted with the app) and UI (what it looked like) amidst all the suggestions. Everyone had ideas and we did nothing to discourage them. Of course this meant that every page went through many changes. The first UI the user saw for example went through 17 revisions during the 56 builds that I made. Each revision of each page was done in code and IB which sounds crazy but it allowed us to show people and get real feedback. One time our designer was checking in a revision of the initial UI screen while people next to us were discussing doing something yet again different. He looked at me and laughed and hit the revert button.
The funny thing about this process was that people were seeing constant changes to the app almost immediately, there were no iterations other than a daily build (sometimes more). Many of the executives and others ran out of suggestions before we ran out of patience! I think some became embarrassed thinking they were slowing down the CEO’s pet project. By giving people this constant feedback we were able to hone the flow, the appearance and the features much faster.
Now during this time we were having a daily Kanban meeting with the process guy and he became frustrated that everything was piling up in the In Work column. He complained that we should be finishing each item before starting on the next. It took some time to get him to understand that everything was constantly being changed every day—that was the nature of this wild process. Not only that we had no official QA assets to even test even if something was complete. Our CTO at the time didn’t believe in hiring QA. At least we had a lot of people trying it out.
The three of us kept a shared file in some online note taking application to remember all the things we still had to do or change, that was our real “process”. We did have a daily meeting where various groups communicated where they were which I suppose is a process as well.
As we approached the 8th week the app started to get a more finished appearance, we did a couple more user test sessions in our lab. People stopped making more suggestions as they seemed happy to see it getting done.
A New Wrinkle
Naturally there had to be a new wrinkle. Someone decided we needed to add credit card scanning to the app to make it cooler. There was a new startup that offered the first (I think) credit card scanning technology and we would be the guinea pigs. It was still in beta but they promised it would be done soon. It worked by taking images with the iPhone camera, sending them to servers in Europe where the number and date were identified and returned to the app. It had a lot of issues; I remember helping them fix memory crashing bugs in their code. It was slower than typing in the credit card number yourself but seemed like a cool extra feature. No other app had this yet either.
By the ninth week the app seemed pretty solid, the back end people had figured out how to run a new business unit, support was being set up, our caching was working well, and it was time to look at submitting it to Apple.
We worked the weekend before submitting it on a Monday. Other than a few hours the first weekend at home, this was the only overtime we worked on this project. Yes, despite all the craziness we went home every day at a normal time.
While Apple was busy doing whatever they do, the CEO decided to make the app cooler and told our supply people to lower the margin on every room the app would show to zero. Turned out zero was not allowed so we set it to a penny. This violated every agreement we had with all the hotels and would undercut all of our apps and the main website, but hey what the CEO wants…
Apple approved the app and it went live. We had a great rep at Apple and they featured the app in iTunes so we had great downloads. The reviews were pretty good and it was clear some people booked with the app every day which is fairly rare for a travel company.
The CEO ordered a cake in the shape of the app icon, a its daring shade of magenta, and had a big celebration. From then on for the next year at every company gathering he would brag about what an amazing thing we had done to ship this app, see what we can do when we work together, and of course every other group got sick of hearing this ad nauseum.
The app was getting a fair amount of business but now the management was afraid the rash decision to make our margins near zero would fatally affect our income so they decided not to promote the app at all. It wasn’t our normal brand so people thought it was some new company although you could find it by “other apps by this developer” it got no help from anything other than the Apple featuring which eventually went away.
Now we all thought this was strange since we just went through all this work and cake and now it was left to rot. Eventually they raised the margins back to normal and all you got was rates you could get from our other apps and website, but at least with all the usual discounts in one place. People still used it but over time it got less and less bookings. We dropped in the travel category down to 150 or so.
Let’s Waste $10,0000
Our mobile director got permission to try one thing. There are many companies that promise for money to get you ridiculous download numbers and pump you up to the top of the App Store. So despite not caring about the app we bought $10,000 worth of one of these services. I watched it run and sure enough we got tons of downloads and shot up the charts all the way to #2. However all the downloads came from Egypt and China and other countries we did not have customers in. After two days the money ran out and it fell back to the 150s. I don’t think we got a single booking out of it unless a few people downloaded it in the US because of the high ranking.
Of course there are lots of companies who do this to maintain their App Store ranking and I believe it costs a lot more today. We never did it again for anything else.
Two years ago this month we found out we were being sold to our biggest competitor and slowly over time everyone was laid off. All of our technology was sunsetted including this app.
Was it fun? Yes it was. We designed and experimented with real code. We took on suggestions from everyone and tried out everything that made any sense. It was the first iOS app built with ARC which made it much easier to write. It rarely crashed at all. Occasionally the back end would hiccup and book a room three times but it was easy to prove and customer service usually refunded all the money. People who used the app liked it even after the ridiculous margin was removed because it was so quick to book in the popular cities.
The design influenced everything we built after this and many design features were copied into our other and future apps. As a learning tool it was great.
To us it was the essence of agile, free of hard process, responsive to all requests, completed in a reasonable timeframe (two months) and appreciated by those who used it and liked by Apple.
Yet it was an idiotic project that made no business sense at all! We could never really compete with a well funded startup laser focused on a small target that negotiated rates on a daily basis. Of course they are still in business though I have no idea if they ever became profitable (contacting hotels every day is not very scalable cheaply). I do remember seeing a tour of their offices online and the area reserved for their cornhole game was bigger than our entire mobile group’s office space!
In the end I don’t regret any of the project though as it was fun to build and getting something done that started so vaguely was very rewarding.