How do you build a product like Flickr or LiveJournal? What are the issues of scalability? Where do you start? How do you learn to scale? This was the theme of the talk at the SDForum event on Software Architecture and Modeling SIG.

The talk was mesmerizing. For almost two hours, ideas and thoughts flowed like water, out of Cal. He would pause to answer some questions, get back to his thought stream. Almost felt as if he was thinking aloud. I think most of us did not even want to inturrupt. It was like watching a science fiction movie. Thoughts were racing in my mind. How did it start? Where did this brilliant young man learn all this from? Obviously other members of the audience had similar questions. Someone asked him what Flickr looked like when they started.

Cal talked about the humble beginnings of Flickr with a few servers in a colocation. Here is the picture of Flickr in its current configuration.

Cal’s slides are here. Cal mentioned that lots of people took inspiration from Live Journal’s back end architecture, which, incidentally is completely built out of LAMP (Linux, Apache, MySQL, Perl/PHP/Python/Ruby). A presentation on Live Journal’s backend is here.

I walked away from this talk with several thoughts:

• This stuff is hard and takes a lot of dedication to improvement
• Managing data(bases) seems to be one of the most critical components of scalability
• LAMP is worth serious consideration since there seems to be so much experience in the industry in using it
• PHP is worth looking at too. Two of the products I like most - Wordpress and Flickr both are built using PHP and both seem to scale reasonably well
• It is fascinating to observe great minds at work. There is so much intelligence in the world, so there is still hope