Just another Blog

Another cute error message

2009-08-03T09:14:00.003+05:30

Firefox seems to be getting a personality too! This error message, which I saw the first thing after starting up my laptop, made me smile :-)

So, it's not only web applications that have cute error messages, now.

A post I made earlier about Flickr's "down for maintenance" message.

How to make your Hive cluster blazingly fast

2009-07-30T21:01:00.007+05:30

Squeeze it! Literally.

Always, always, always - compress your tables. Most of your Hive query execution time is not going into doing complex computations. Aggregates, averages, and counts across tons of weblog data are not exactly computationally complex. It's the reading & writing of tons of data that's taking time.

In computer science parlance, it's not CPU bound, it's I/O bound.

If your data is already compressed (which in all probability, is the case), then nothing can beat the speed at which you can import it into Hive. Simply use the LOAD DATA LOCAL command to load compressed files into a TextFile table. Hive will automatically detect the compression and decompress on-the-fly when executing queries.

Leaving aside the data import times, here are some (unscientific) benchmarks for a SELECT COUNT(1) query on my raw data tables. From 106 sec down to 60 sec. At 10% of the storage space.

Storage	Row count	Table size	Query time
Uncompressed	8,259,720	7,686 MB	106 sec
Compressed – GzipCodec (RECORD)	8,259,720	4,773 MB	101 sec
Compressed – native gzip	8,259,720	736 MB	60 sec

Now, each of these tables was put through a moderately complex Hive query [1], which processed the data using two custom map/reduce scripts, and inserted the procssed data into a new table. Sure, the initial ETL phase is not significantly faster, but look at the amount of space I'm saving.

Storage	Resultant row count	Resultant table size	Query time
Uncompressed	1,561,633	1,608 MB	699 sec
Compressed – GzipCodec (RECORD)	1,561,633	N/A [2]	563 sec
Compressed – native gzip	1,561,633	86 MB	510 sec

I'll try to run some complex queries on the table with processed data to compared between compressed and uncompressed storage.

Learn how to compress your Hive tables at Hive Wiki – Compressed Storage

[1] Take a look at the section titled "Processing the raw data (ETL)" here
[2] I forgot to record the table size before dropping the table :-)

Using Hive for weblog analysis

2009-07-24T14:23:00.009+05:30

Introduction

I've been playing around with Hadoop since the last fortnight to see how it performs with our weblog data processing jobs (Apache access logs). Right now we're using a blink-and-it-breaks system running a bunch of custom Perl scripts for log processing and MySQL for data storage. It's running on a single server class machine. It takes it about 3-4 hours to extract, transform, and load (into MySQL) one day of weblogs (this includes identifying sessions based on a 30min timeout). Another 1-2 hours to aggregate data and generate various CSV reports.

So, a total of 6-7 hours, when it's not crashing.

With Hadoop + Hive I've brought the first phase down from 3-4 hours to under an hour! Anywhere between 15 minutes to 45 minutes to copy log files into a raw Hive table (depending upon how much replication you've configured your cluster to have). After that, 12 minutes to extract, transform, and load into a new table (including session identification).

I gave Cloudbase a shot before Hive. Cloudbase was *dead* easy to setup and get off the ground (I love their simple approach to UDFs & UDTs). However, it's performance was not as good as Hive. On the other hand, I spent quite a lot of time trying to even import my custom log formats into Hive (I'm looking at you GenericUDF). But once I got started, man, was I blown away! Take a look at my Twitter updates on Cloudbase and Hive comparisons.

Cluster setup: I managed to corner 4 desktop machines in the office for my experimental cluster. They are Core-2 Duo 2.4 GHz machines with 1GB of RAM and standard SATA hard disks. All four are connected to each other using a switch.

Setting up the Hive tables

Here's how I've set up my Hive tables:

The following table (raw) stores the raw, unprocessed log files. I've partitioned the table by date so that the ETL operations can be run on bite sized chunks consisting of a day's worth of data.

create table raw(line string) partitioned by(dt string)
row format delimited fields terminated by '\t' lines terminated by '\n';

The following table (hits) contains the processed & filtered hits, partitioned by date. Within each partition the rows are clustered by Apache user-id (aid) and within each cluster they are again sorted by page view times (ts). According to Hive Wiki this clustering & sorting can improve efficiency in certain queries. Although it does lead to an increase during the ETL phase. I'm really not sure what the buckets do - and 1,000 is just a number I pulled out of thin air.

create table streamed_hits(ip_address string, aid string, uid string,
                          ts string, method string, uri string,
                          response string, session_id string,
                          session_start string, pv_number int,
                          clickstream string)
partitioned by(dt string)
clustered by (aid) sorted by (aid, ts) into 1000 buckets
row format delimited fields terminated by '\t' lines terminated by '\n';

Importing the raw data

Here's how one import data into the raw table. The import process is a simple HDFS file copy operation[1]. The more replication you have, the larger number of nodes the data needs to be copied to. Hence, the more time it takes during this operation.

load data local inpath '/weblogs/20090602-access.log'
into table raw partition(dt='2009-06-02');

load data local inpath '/weblogs/20090603-access.log'
into table raw partition(dt='2009-06-03');

Processing the raw data (ETL)

Here's the Hive query for the ETL operation. I'm making sure that uninteresting rows are discarded upfront using the WHERE clause in the inner query. I had earlier put this in the outer query (which was causing LOTS of uninteresting rows to go through the parsing logic). The (apparently, in hindsight) stupid placement of the WHERE clause was bumping up the completion time of this query from 15mins to 2+ hours!

I'm using a custom map script to read my log files, which are in a 'non-standard' format. There is another way to do this - UDFs & SerDe - but according to the discussion I had on the mailing list, these features are not fit for newbie consumption yet. The DISTRUBUTE BY & SORT BY is essential for my session & clickstream identification script to work (thanks to the Hive Wiki).

from
   (from raw
   select transform line using 'parse_logs.pl' as     ip_address, aid, uid, ts, method, uri,
                                                   response, referer, user_agent, cookies, ptime
   where lower(line) rlike '^(\\S+) (\\S+) (\\S+) \\[(.*?)\\] "(.*?)" (\\d+) (\\d+|-) "(.*?)" ".*?(mozilla|msie|opera).*?".*'
           and not line rlike '^(\\S+) (\\S+) (\\S+) \\[(.*?)\\] "(\\S+) (/images.*?|/styles.*?|/javascripts.*?|/adserver.*?|.*?favicon.*?) (\\S+)".*'
           and dt='2009-06-30'
   distribute by aid sort by aid, ts asc) parsed
insert overwrite table streamed_hits partition(dt='2009-06-30')
   select transform parsed.ip_address, parsed.aid, parsed.uid, parsed.ts, parsed.method, parsed.uri,
           parsed.response, parsed.referer, parsed.user_agent, parsed.cookies, parsed.ptime
   using 'identify_sessions_and_clickstream.pl' as ip_address, aid, uid, ts, method, uri,
                                   response, session_id, session_start, pv_number, clickstream;

Custom map/reduce scripts used

Here's the parse_logs.pl Perl script. Please excuse my Perl - I'm not a Perl programmer. I was just trying to reuse the funky regular expressions that we had lying around in our current log processing system. You use plug in any script, that reads/writes from standard input/output, into Hive.


#!/usr/bin/perl

my %monthNum=(
   "Jan" => 1,
   "Feb" => 2,
   "Mar" => 3,
   "Apr" => 4,
   "May" => 5,
   "Jun" => 6,
   "Jul" => 7,
   "Aug" => 8,
   "Sep" => 9,
   "Oct" => 10,
   "Nov" => 11,
   "Dec" => 12,
);

while (defined($line = )) {
   if (($host,$user,$apache,$rfc931,$method, $url, $ver, $status,$size,$referrer,$agent,$cookies,$ptime) = $line =~ m/^(\S+) (\S+) (\S+) \[(\S+ \S+)\] "(\S+) (.*?) HTTP\/([0-9\.]*)" (\d+) (\d+|-) "([^"]*)" "([^"]*)" "([^"]*)" (\d+|-)$/) {
       # everything found -- nothing to be done
   }elsif (($host,$user,$apache,$rfc931, $method, $url, $ver, $status,$size,$referrer,$agent,$cookies) = $line =~ m/^(\S+) (\S+) (\S+) \[(\S+ \S+)\] "(\S+) (.*?) HTTP\/([0-9\.]*)" (\d+) (\d+|-) "([^"]*)" "([^"]*)" "([^"]*)"$/) {
       $ptime="";
   }

   $ts="";
   # Converting date to yyyy-MM-dd hh:mm:ss
   if (($day, $monthname, $year, $hour, $minute, $sec)= $rfc931 =~/^(\d{2})[\/-]([^\/-]+)[\/-](\d{4}):(\d{2}):(\d{2}):(\d{2})/) {
       $month=$monthNum{$monthname};
       $ts=sprintf("%04d-%02d-%02d %02d:%02d:%02d", $year, $month, $day, $hour, $minute, $sec);
   }


   $agent=lc($agent);
   $user=lc($user);
   print "$host\t$apache\t$user\t$ts\t$method\t$url\t$status\t$referrer\t$agent\t$cookies\t$ptime\n"
}

Here's the identify_sessions_and_clickstream.pl Perl script. Again, same disclaimer about my Perl as above. Also, please excuse the extremely naive way in which I'm trying to construct the clickstream. That work in progress.

This script depends on the fact that the incoming data is sorted on Apache user-id and timestamp (that's what the DISTRIBUTE BY & SORT BY in the Hive query achieve). Apart from identifying 60min sessions I'm cleaning up the URLs to make them more 'generic' (removing query strings, removing trip IDs etc.)

#!/usr/bin/perl

use Date::Parse;

$session_duration=60*60; # in seconds
$prev_apache=undef;
$prev_ts=undef;
$pv_number=undef;
$session_id=undef;
$session_num=undef;
$session_start=undef;
$clickstream='';

while (defined($line = )) {
   chomp($line);
   ($ip_address, $apache, $user, $ts, $method, $url, $status, $referrer, $agent, $cookies, $ptime) = split(/\t/, $line);
$url, $status, $referrer, $agent, $cookies, $ptime\n";
   $url =~ s/(.*?)\?.*/$1/i;
   $url =~ s/^\/(trains|flights)\/itinerary\/.*?\/(.*?)/\/$1\/itinerary\/itinerary-id\/$2/;
   $url =~ s/^\/(trains|flights)\/itinerary\/(\d+)$/\/$1\/itinerary\/itinerary-id/;
   $url =~ s/^\/(activate|reactivate|reset)\/.*/\/$1/;
   $url =~ s/^\/share.*/\/share/;
   $url =~ s/^\/account\/trips\/.*/\/account\/trips\/trip-id/;
   $url =~ s/^\/trains\/stations\/[0-9]*/\/trains\/stations\/numeric-id/;
   $url =~ s/^\/trains\/stations\/[A-Za-z]*/\/trains\/stations\/alphanumeric-id/;
   $url =~ s/^\/hotels\/info\/.*/\/hotels\/info\/hotel-id/;
   $url =~ s/^\/places\/hotels\/.*\/images.*/\/places\/hotels\/images/;
   $url =~ s/^\/places\/hotels\/images.*/\/places\/hotels\/images/;
   $url =~ s/^\/newsletters\/images\/.*/\/newsletters\/images/;
   $url =~ s/^\/index.*/\//;
   $url =~ s/\/$//; # remove trailing slashes

   if(!defined($prev_apache) || $prev_apache ne $apache){
       if($apache eq '-' or $apache eq '')  {
           # TODO -- use IP address & user_agent to identify sessions?
           $pv_number='';
           $session_start='';
           $session_id='';
           $clickstream='';
       } else {
           $pv_number=1;
           $session_start=$ts;
           $session_num=1;
           $session_id="$apache|$session_start";
           $clickstream="$url"
       }
   } elsif($prev_apache eq $apache) {
       if((str2time($ts)-str2time($prev_ts))<=$session_duration) {             $pv_number=$pv_number+1;             if($pv_number<70) clickstream="$clickstream|$url" pv_number="1;" session_num="$session_num+1;" session_id="$apache|$session_start" clickstream="$url" prev_apache="$apache;" prev_ts="$ts;">

[1] A small warning here, if your Hive session is running on one of your active nodes. Beware that because of HDFS default policy your Hive node's disk is going to be filled up first. This might lead to a disk space problem if your HDFS partition is being used by other processes as well. I ran into this and had to run the HDFS balancer, which, btw, is extremely slow! (even after increasing the dfs.balance.bandwidthPerSec property).

Capitalism and the environment

2009-07-07T10:41:00.004+05:30

Sadly, this is not a joke - it's the truth. All large-scale projects see "environmental concerns" just as a nuisance they have to put up with it. Something that will eventually have to yield, because, well, you just can't stop "development", can you?

Why aren't existing trees part of "beautification drives"? Why does beautification always mean pouring massive amounts of concrete on the ground and building railings and felling of trees?

On that note, I would recommend 'Small is Beautiful: Economic as if People Mattered' by E.F. Schumacher as compulsory reading to everyone studying the pseudo-science of economics. And to everyone who thinks growth is good - perpetually.

Directi Ads on Facebook

2009-07-06T14:54:00.003+05:30

Nice :-) This must be resonating with the uber-geeks really well. Check out the careers page as well -- "Open Techie Positions"

My comments to Ministry of IT about Section 69A of IT Act

2009-07-02T13:39:00.003+05:30

Emailed to grai (at) mit (dot) gov (dot) in as given at http://www.mit.gov.in/default.aspx?id=969

Dear Sir,

With reference to the Draft Rules under section 69A of the Information Technology (Amendment) Act, 2008, available at http://www.mit.gov.in/download/sec69A_Rules.pdf , I would like to bring to your notice the following comments --

(i) Lack of neutral party: There is no provision in the Draft Rules where the request for blocking information is examined by a neutral third-party (say, a court designated for this purpose) before being enforced. This leaves the law open to abuse by persons "in power" who want to curb any criticism (published on the Internet) aimed towards them. In other words this creates a situation where the plaintiff, himself is the judge.

(ii) Lack of representation: There is absolutely no provision in the Act which gives the publisher (against whom the request for blocking has been initiated) a right to a fair hearing in front of a neutral third-party. I realize that on the Internet, which lends itself to anonymity, identifying & notifying the publisher of a particular information resource is difficult. However, there needs to be a process in place using which the Designated Officer first needs to notify the publisher against whom a Request for blocking is pending. This process may include: (a) emailing the contact person mentioned on the website/resource, (b) notifying the ISP on which the website/resource is hosted, (c) general notice on the Minister of Information Technology's website, etc. This is analogous to various "Notices" that any government agency serves before taking drastic action against a person -- like sealing of property, or disconnection of electricity/water supply.

I have worked in the IT industry for 5 years and I understand the need for tighter regulation and stronger cyber-laws. However, these need to be balanced against (a) right to freedom of speech & expression, and (b) right to a fair trial. In my opinion, the Draft Rules completely oversteps these constitutional rights in a bid to provide the governing agencies with more power.

Regards,
Saurabh Nanda.

A SIM-card along-with the National ID card?

2009-07-02T12:10:00.005+05:30

Taking on from Sanjay Swamy's post at Medianama, I think it's an excellent idea. Sure, there are the rough edges that one needs to smoothen out, but that's the case with any scheme of this magnitude. The kind of opportunities that open if your national ID can be part of your mobile SIM card are amazing. Especially in the mobile banking and m-commerce area.

Picture this: Your bank and telecom provider, both as part of their KYC, will need to know your national ID. Now, if you have the ability to link-up your SIM with your national ID (both smart cards), you can perform a fully authenticated financial transaction through your mobile phone. Just think of the number of services that can be made available to the rural population through such a thing!

The biggest challenge this idea will face is from people getting paranoid about
(a)a telecom operator "owning their identity", and
(b) not everyone having a mobile phone.

The National ID should be a complete in itself even without a SIM or a mobile phone. Something like the new vehicle registration smart cards you get in Maharashtra nowadays. However, it should have a detachable SIM card built right into it. If you want to use the additional features of your National ID, just remove the SIM and plug it in. You can keep your current phone number and even switch operators -- thanks to Number Portability.

The telecom operator (or anyone else on the mobile network) will NOT have access to your identity unless you authorize them during a transaction which requires such an access. It's just that if the SIM and your National ID live on the same hardware (which is, the smart card) you can link them up with ease, as and when required!

ICICI Bank -- Email shy?

2008-11-21T13:14:00.004+05:30

Has anyone ever been able to successfully email ICICI Bank? I've tried twice and failed. Today was my second attempt. After being asked a series of questions (four, I think) I finally get this form.

Even after asking my account number (which will give them instant access to all my contact details -- KYC, remember) they still want me to type out my email address, phone number (along with STD code -- mobile numbers don't work), and complete contact address.

And finally, the damn thing doesn't work!

What's wrong with just giving an email address? Saves them from the development and maintenance cost of developing this stupid tool. Isn't customer support on email cheaper than doing it over the phone?

Update: A friend tell me that the form works on IE and not on Firefox. Irks me even more!

Email archival without mailing list software?

2008-10-30T13:10:00.002+05:30

I believe that the lack of archives in business communication is a serious enough problem to solve. Especially in these times, when a lot of brainstorming happens on email amongst team members located in different geographies.

I've tried using Basecamp for all business communication and succeeded only partially. Although Basecamp can serve as a good archive for email communication, it just doesn't beat the simplicity of adding any random email address to the CC list when you hit "Reply all". You have to first add a user to a project, then add him to the message, and then reply to the message. This, of course, will work only if the person knows how to use Basecamp. Otherwise, add the additional overhead of evangelizing Basecamp over regular email before getting him to accept this new mode of communication. [1] If something confidential is being discussed, there's also this fear of archiving your trade secrets on someone else's servers.

Probably there's a good use-case to add a new feature in SMTP servers -- an archive header and an email archiving module. If an incoming email has the archive header (X-Archive: true) the SMTP server will archive the email by storing a copy of that email and making it accessible on a web page. All replies to that email (and further replies to those emails) will automatically get archived, regardless of whether they have the archive header or not [2]. The server can also add a footer to each email in the thread to direct the user to the archive. Optionally, the archiving module may also archive email attachments and make them available on the company intranet (or the web).

Various mail clients will also need to be modified to present a user with a checkbox while composing the mail that sets/unsets the archive header.

I think this feature can be easily hacked into postfix (or sendmail) and a couple of open source email clients (like Evolution and Thunderbird).

Alternatively, this can also be achieved by having a record-keeper email ID (something like archive@somecompany.com). Any email CCed to the record-keeper will be archived and available on the company intranet (or the web). The only drawback of this is that the archival chain can break very easily if someone removes the record-keeper from the CC list. This approach can probably be an easier path towards implementation, both from the client's as well as the server's side. (Aren't such archival modules already available?)

If anyone wants to collaborate on this project with me, please drop me an email at saurabhnanda (at) gmail (dot) com. I've been wanting to work on a decent tech project for quite some time now.

[1] Given, recipients of Basecamp messages can now simply reply to messages just like normal email, it's still not perfect. All formatting (even line-breaks) are stripped. And you still have to log on to Basecamp to compose a completely new message.

[2] I think it's reasonable to assume that since the parent wanted the email thread to be archived, it must be archived in its entirety.

In Support of "Top Posting"

2008-10-30T13:01:00.002+05:30

All over the web, on mailing lists, especially in the open source geek circles [1], top posting is frowned upon. Even while bottom posting, you're not considered cultured if you don't truncate the parent post to the exact specific part(s) your reply pertains to. I, too, used to follow this religiously until recently.

After three years of communicating with non-geeks (you know, the business and marketing types), I realized that a lot of business communication was simply not possible in the typical quote-reply-quote-reply format that is prevelant on mailing lists. This is because of a couple of reasons, which I'll try to enumerate in the following paragraphs.

One of the reasons given for not including the parent post in its entirety is that anyone wanting to refer to the thread can always look up the mailing list archives. Cultured folk should not waste bandwidth and screen real estate by repeating what is available a click away. This is not the case with typical business communication, which starts between two people (with usually their bosses on the CC list -- to be "kept in the loop"). Along the course of a reasonably sized email discussion, the CC list grows exponentially (believe me)! First the bosses' bosses are added -- if the small fry realize that they'll need to save their ass if something goes wrong. Suddenly something is under the purview of a chappy in some other department and him and his boss and boss' boss are added to the list. Later someone realizes that the legal department also needs to be "kept in the loop". Within no time you're having an orgy on email! With each new addition to the CC list you need a place where the latecomers can read-up on the discussion till then. In normal business communication there are no "archives". The latest email, itself, serves as an archive. I've had a tough time catching up on the discussion where each email was regularly truncated by the participants. I had to piece together more than 25 different emails forwarded to me as the "archive" -- believe me, it wasn't an easy task. Therefore it's much better to preserve the entire thread in the body and add your reply to the top of the email.

A side-effect of not having an archive is that you can't really follow the quote-reply-quote-reply pattern ("Comments inline...") very effectively. If five different people are communicating, you suddenly lose track of (a) who said what, (b) in reply to what, and (c) in reply to whom. With an archive, each post can easily be identified on all three parameters. Take away the archives, and within two or three replies you lose track of (a) and (c) very easily. Inline comments, even in business communication, are useful when each paragraph or point in your email can be answered independent of the others. For example, when you're seeking answers to a list of technical questions. However as soon as you want to reply to the reply of your questions you start losing track of (a) and (c). One decent solution I've noticed is prefixing your reply with your name in square brackets. So, a typical email after a couple of back and forth replies looks like:

* Can I pass the customer's shipping address instead of the billing address in the API?
[Alice] It's possible, but why would you want to do it? You already have the billing address in the DB, don't you?
[Bob] Yes, we do -- but we don't want to share it due to privacy concerns.
[Alice] Okay, then you can pass the shipping address.

I've observed a significant side effect of following the quote-reply-quote-reply model of communication -- especially on tech mailing lists. A lot of discussions regularly morph into flame-wars, mudslinging matches, or "preachers preaching the lesser mortals." Don't get me wrong -- the communication style is not the sole reason for discussions going off track. Obviously a lot depends on the attitude of the people involved in the discussion, but I feel the communication style definitely promotes the degeneration of the discussion. When you read a post you generally disagree with, it's very tempting to pick each line, quote it, and try to rebut it individually. It's very easy to get lost in the trees for the woods.

[1] These are the only circles which seem to have a respectable mailing list culture.

Overestimating users of technology

2008-10-15T12:12:00.004+05:30

Most software products assume too much about their users. After reading Alan Cooper's "The Inmates Are Running The Asylum" (at Amazon and at IndiaPlaza) I realized how much I used to presume about a user's comfort level with technology while designing products. I now consciously switch off the developer/geek side of my brain while writing product specs or mocking up UI prototypes.

Here's a post by David Pogue on what everyone assumes that everyone else knows but is wrong. I thought I would know every single thing on the list but was surprised to find a keyboard shortcut that was new to me - using shift + space to scroll up in a web page.

You can tap the Space bar to scroll down on a Web page one screenful. Add the Shift key to scroll back up.

Yahoo! recently released their research findings about an OpenID usability report. They were not surprising:

None of the users had heard of OpenID before, and none of them even noticed the OpenID sign-in box displayed below the traditional email/password login form on the site. [...] Observing these tests was more than a bit frustrating for the Yahoo! OpenID team, and the test subjects may have been distracted by the sounds of the groans and head-pounding coming from the other side of the one-way mirror. Certainly there is a lot of work to be done on the OpenID UX (user experience) front.

(At the risk of over-simplification, OpenID is a common username/password using which you can login to multiple websites without creating individual username/passwords for each.)

Scary: CSRF and REST

2008-09-30T17:58:00.007+05:30

The latest exploit to hit the web is CSRF. It's nothing fancy, very simple to execute and understand. In a nutshell:

You're signed in to website A

You open a new tab and visit website B. Website B is a malicious website (or is a trusted website which has the potential of being malicious because of allowing user generate content-UGC).

Website B has an image tag (not necessarily, but the simplest to inject via UGC) which points to a predictable URL on website A. For example, <img src="http://www.a.com/account/delete?confirm=yes" /> which basically deletes your account on website A.

Your browser will try to fetch the "image" from website A, merrily sending your session cookie along with the request. And BOOM! On website B you don't get the image ('cuz there is none) but your account has just been deleted from website A.

This is the simplest scenario I've explained. Read the Wikipedia article CSRF to understand how it can be executed in a variety of ways and situations.

Over the past year or so I've been going ga-ga over RESTful architecture and stateless servers. I've tried making URLs predictable and discoverable. Personally, I click on the "keep me signed in" checkbox whenever on browsing the web on my laptop. This just turns my world around!

Does it mean that we now need to make sure we're not signed-in to one site while visiting another? Or that all web applications now have to be stateful and URLs have to be non-predictable? Which means that while generating a page the webserver will have to add a random token to each URL and validate that when the browser requests the URL? The horror!

Any thoughts, anyone? This sounds like impending doom!

Review: Rock on!!

2008-08-29T12:57:00.003+05:30

So, last night I was conned into buying discounted paid preview tickets for Rock On screening at Metro Adlabs. I went there all hyped up thinking of catching a glimpse of the Rock On crew and general P3 types crowd. But I was let down. Cast/crew chhodo, acchi ladkiyon ki bhi kami thi! Anyways, but I was not let down by the movie.

Superb music - especially if you like rock (you'll like it even otherwise). Heard such good Hindi rock after a long time - nice lyrics and singing. Superb acting by the entire cast - never knew Arjun Rampal could act well. Superb singing by Farhaan Akhtar.

Movie is a tad slow and dwells upon each character and relationship and lets it build. But, I guess that's required. That's how the audience connects with each character.

Loved the end note - "Don't download the music. Buy the CD!" Heh, I will. Hope you will too. I wonder why these guys don't sell music CDs + posters + t-shirts right outside the movie hall. I'm sure I would've bought a CD without a second thought had I seen a stall right after I stepped out after watching the movie.

PS: Vote for it on the IMDB page for Rock On!!

FM

2008-08-12T13:20:00.002+05:30

BMC aur Dharmendra mein ab zyaada farak nahin hai bacha, kutton ka khoon peene mein dono ko aata hai maza!

That's Mallishka on Red FM! Absolutely hilarious.

I'm hooked onto FM these days. Especially Rainbow FM in the mornings and late nights - they play amazing English music. The RJs mostly suck, but the music rocks.

Why do the other stations play the same songs over and over and over again? They haven't yet gotten over Jab We Met. I'm sick of listening to those songs. Get a new playlist guys!

Lazy programmer and why Lisp templates suck

2008-07-09T23:52:00.003+05:30

I was reading some random article on Rediff and happened to notice this block shoved in one corner. What caught my eye was the first question - "Are commodity exchanges responsible for price raise of food grains?" - but I got easily distracted by the sheer laziness of the developer who implemented this. Check out the screenshot below:

1 answer(s)
5 answer(s)

I mean for god's sake! On one hand we have people who're trying to teach computers natural language processing, and artificial intelligence; and on the other hand computers can't figure out if was one answer or five answers.

Then I thought, how easy it would be to implement this in ERb:
<%= post.answer_count %> <%= post.answer_count==1 ? "answer" : "answers" %>

And then I got thinking how damn painful it would be to implement this in Lisp and HTML-Template. You'd first have to create a flag for each post indicating whether it had a single or multiple answer. Then you'd have to pass all those flags (or list of flags) down to the template filler function. Then you'd have to put a big TMPL_IF statement in the template to achieve the same thing:
 answeranswers
You could of course, for each post, simply compose the exact string to be displayed ("1 answer", "5 answers") and pass that down to the template. But what's the point of having templates if you can't separate view logic cleanly.

Lisp is a great language, which will probably help you achieve a lot of intellectual orgasms (Meta-object protocol, macros, reader macros, etc.) But it sure does need a good templating tool. I tried hacking around in CL-EMB, but it doesn't really cut it.

[End rant]

IIT Kanpur Suicides - Werther Effect/Copycat Suicides?

2008-07-02T23:18:00.000+05:30

Yet another suicide rocks IIT Kanpur. Yet another round of introspection. More statement blaming the student-faculty relationship, internet usage, grading system and academic pressure. Media bites with ridiculous statements from, both, the students and the faculty.

I'm not sure how much the situation has changed from my times - and they're not too far in the past, I'm a 2005 graduate - but, I really don't think that the academic pressure is too high at IIT Kanpur. Any lower, and it'll risk getting lost amongst the gazillions of second and third rung engineering institues in the country.

Ditto with the grading system. It's not the best, but it's not the pits either. Every system has its pros and cons. And the cons are not that bad that they'd cause students to start committing suicides over them.

I think the suicides are because of an individual's intrinsic character and the Werther Effect. I first came across the Werther Effect while reading the chapter on social proof in Robert B. Cialdini's book, "Influence - The Psychology of Persuasion." The Werther Effect talks about how a highly publicized suicide can cause a number of follow up copy-cat suicides amongst people in the same situation. And suicides in IIT Kanpur (or any other IIT for that matter) are surely highly publicized. You have it splashed on the front pages of all local newspapers, the in-campus newsgroups would be flooded with posts, discussions in the quads & wings, special committees and reports, etc.

When I first read about the Werther Effect, I scoffed. But, now I've started believing in it. An extract from the Wikipedia article:

The well-known suicide serves as a model, in the absence of protective factors, for the next suicide. This is referred to as suicide contagion. They occasionally spread through a school system, through a community, or in terms of a celebrity suicide wave, nationally. This is called a suicide cluster[1]. Examples of celebrities whose suicides have inspired suicide clusters include the Japanese musician Hide and Yukiko Okada.

I think some psychology/sociology professor from the IITs should be looking at this angle as well. In my opinion, the grading system and the academic pressure is just fine. Let's not degrade the quality of IITs any further - we've got politicians to do that for us!

[HTML] The under-used tag

2008-04-19T19:21:00.004+05:30

I've seen so many web sites not using the label HTML tag. It's a very nifty usability tool whenever you're writing captions for form fields - especially checkboxes and radio buttons.

The problem with checkboxes and radio buttons is that they're small. You have to position your mouse pointer very accurately to be able to turn it on or off. With labels you can associate the form field with its accompanying caption. Clicking on the caption will turn the associated checkbox/radio button on or off. Give it a spin here:

Without the label tag

With the label tag

With text boxes clicking on the label will shift the cursor into the associated text box:

Enter your name (without label):

Enter your name (with label):

All you need to do is give the form field an id tag and associate the label with it:

<input type="checkbox" id="example2" /> <label for="example2"> With the label tag</label>

Impulse Purchase

2008-04-19T19:15:00.004+05:30

I've seen people purchase clothes on an impulse. Shoes, deodorants, shades, food, whatever. But I don't think I've seen anyone purchase a domain name on impulse!

A few days ago I was following a truck on my bike and saw the ubiquitous "Horn OK Please" written on the back. And suddenly out of nowhere I thought of "horny" please. What a cool domain name to have! I came to the office, and purchases it for Rs 599 at Net4Domains.

What the fuck do I do with it now? Any ideas?

Saurabh Nanda. Proud (confused?) owner of www.hornyplease.com!

Composing blog posts in Textile Markup

2008-04-09T13:09:00.003+05:30

Why can't I compose posts in Textile Markup? It's so damn simple and guarantees conversion to clean HTML. I hate using the rich text editor for composing posts. It's too complicated and I don't like the superfluous markup it generates. Plus I've also gotten addicted to Textile, thanks to the amount of time I spend on Basecamp.

I've submitted a feature request and also posted about it on the Blogger Help Group.

Clueless with Computers...

2008-04-08T19:11:00.003+05:30

... that's how non-geeks feel. I'm extremely sure of that now.

I had been spending some time planning my parents' trip to Kerala. I had every event (flight, car travel, hotel stay, etc.) added neatly to my Evolution calendar. (Given the number of meetings I now have to attend - I've been using Evolution avidly for the past month or so - but that's a different story). Finally, I sent out notifications for each event to my parents.

I called up my Mom expecting the entire travel-plan communication process to be a breeze. What's the complication, I thought. You get a bunch of meeting invites; you click "Accept" for each one and boom! you have a proper travel calendar right in front of you.

She got the mails alright, she even clicked "Accept" on each one of them. But I had one hard of a time trying to get her to the calendar.

Me: Okay, now that you've accepted them, you can see the schedule in the calendar.
Mom: How do I get to the calendar?
Me: Okay, do you use Outlook? [That's what most offices use.]
Mom: Yes.
Me: So, the left pane, sorry column, sorry left side of the screen will have a button for calendar. Click on it.
Mom: It's not there.
Me: Can you see your mail folders?
Mom: Yes. But no calendar.
Me: Okay, I'll call you back in 5 min.

Then I rushed to a guy with Microsoft Windows on his machine. Fired up Outlook and noted the exact location of the Calendar button. It was pretty visible. But then I realized, probably it's been turned off on her machine. I asked the guy with M$ installed, he showed me how to configure your left pane to show/hide various buttons. I was like, yeah right. "Mom look at the left pane, there's a teeny-weeny arrow there which will lead to a menu, which will have an option called 'configure', which will have various checkboxes...." Not happening!

I spent the next 10 mins trying to figure out how to get to the calendar in Outlook without clicking on an onscreen button or going through the configuration process. It's bloody un-intuitive. It's "Go > Calendar" if you don't already know. The shortcut for it is Ctrl+2.

Back to the phone:

Me: Okay, press Ctrl+2.
Mom: [After 30 sec] I did, nothing happened.
Me: Are you in Outlook? [Damn!]
Mom: Yes.
Me: Do you have a Menu item called "Go"? You know, right at the top where file, edit, etc. are? Can you see a "Go" there?
Mom: No, there's no "Go" option!
Me: Are you sure you're using Outlook?
Mom: Yes.
Me: Are you sure it's not Netscape?
Mom: [getting irritated] yes!
Me: okay, are you using Outlook on your own desktop? Or do you sign in to Outlook using the Internet Explorer? [Ah, probably she was on one of the M$ web mail box thingies]
Mom: No. It's on my desktop. We have two different icons. One for Internet Explorer and the other for Outlook Express. [!!]

I wanted to pull my hair out! And as it turn out, Outlook Express, indeed does not have a calendar application.

I just gave up on the entire cool iCal stuff. Took a screenshot of my Evolution Calendar screen and mailed it to her.

[Ruby] A few points about Ruby threading...

2008-04-06T00:35:00.004+05:30

... it's green threaded.

That means, the threading is simulated by the Ruby VM and they are not "true OS threads".

That means, if a thread in your Ruby program makes a call that the VM can't suspend, all threads running in that VM will block. In effect, your Ruby VM will "hang" until the rogue thread returns.

A few links about how threads in Ruby work:

Thread and Processes from "the Pragmatic Programmers' Guide"

Ruby Threads from "The RubySpec"

Basically, the recommended way to achieve concurrency, if each concurrent thread/process is doing even moderately complex I/O tasks, is to fork a new process. (The much-touted "shared nothing" architecture)

PS: And in case you're wondering, ActiveRecord *is* thread-safe. It's just that most apps don't use it because of the preferred way of achieving concurrency in Ruby (forking processes).

[Ruby] Madness with the 'require' command

2008-04-05T18:23:00.001+05:30

I spent around 30 minutes trying to figure out why a particular Ruby file was being loaded twice, only to be struck in my face with this gotcha.

Basically, when you use require 'file' the method Ruby uses to determine whether the file has been loaded earlier is pretty stupid. It looks for the filename passed verbatim in it's internal data structure/hash/whatever. So, if you give require 'file' and require './file' the damn file will be loaded twice.

WTF?! Seriously!

Anyways, I thought this was undocumented behaviour but I found this in the RDoc entry for 'require' - should have looked there before.

Ruby tries to load the library named string, returning true if successful. If the filename does not resolve to an absolute path, it will be searched for in the directories listed in $:. If the file has the extension ``.rb’’, it is loaded as a source file; if the extension is ``.so’’, ``.o’’, or ``.dll’’, or whatever the default shared library extension is on the current platform, Ruby loads the shared library as a Ruby extension. Otherwise, Ruby tries adding ``.rb’’, ``.so’’, and so on to the name. The name of the loaded feature is added to the array in $". A feature will not be loaded if it‘s name already appears in $". However, the file name is not converted to an absolute path, so that ``require ‘a’;require ’./a‘’’ will load a.rb twice.

The funniest post I've read in the recent past...

2008-03-15T15:02:00.002+05:30

... is Ek, Anek, aur Ekta. The Vishal Patel guy is back. And, how!

Although his website is full of gems that are going to get you LMAOWOPITOAGMTL [1], this one post about the Ek-Anek national integration cartoon is enough to get you in splits!

Checkout the next/previous links at the end of the first page... "Anek chidiyon ki kahaani sunoge?" "haan, haan >>" / "<< nahin bas"! Ha ha!

'Nuff said. (And do click on the "haan, haan >>" link to LMAOWOPITOAGMTL [1]!)

[1] Note: Laughing My Ass Off While Other People In The Office Are Giving Me The Looks!

What if you can't design for yourself?

2008-03-11T00:09:00.002+05:30

Picking up from Jason's post at SvN:

Apart from the flame war that's going on the comments section there, there is one thing about the "design for yourself" philosophy that has intrigued me. What if you really aren't designing for yourself? What if the software that you are developing/designing is not for your daily use? For example an accounting+finance system that is to be used by hardcore accountants? Do you not concede to how the end user wants the software to function? Don't you lose confidence in your "gut feel" automatically?

The products that 37signals are working on are general purpose products, as such. A project management tool, todo list, CRM application can be used by any company or department. What about specialized software? Can we apply the same principles there?

How to reset an Oracle sequence at midnight everyday

2008-03-09T10:21:00.007+05:30

Oracle, unlike MySQL, does not auto-generate values for the primary key column for you. That means, you can't "auto_increment" a column in Oracle. With each INSERT statement you have to provide the value for the primary key column yourself. Oracle has database sequences to help you generate series of numbers for this purpose.

Sometimes you end up using Oracle sequences for stuff other than as a source of values for your primary key. And sometimes you need to reset the value of the sequence back to 1 at midnight everyday. I ended up with the same problem and I couldn't find a canned solution off the Net for this. I spent a day trying to solve this and ended up learning a lot about Oracle sequences, Oracle packages and PL/SQL, the dbms_lock package, and the dbms_scheduler package.

The problem would be fairly easy to solve if resetting an Oracle sequence were a single atomic SQL statement. No, it turns out that you have to do a bit of circus involving multiple statements to reset the sequence back to 1. I figured that you basically have two options:

Drop the sequence (statement #1) and recreate the sequence (statement #2)

Change the INCREMENT BY property of the sequence to be negative of the CURRVAL (statement #1). Increment the seqeunce (statment #2). Reset the INCREMENT BY property back to 1 (statement #3).

In both the solutions, you run the risk of some other process coming along and doing a seq.nextval on the sequence while the reset is underway. Boom! A bit like getting caught with your pants down.

So, here's what I did:

Create a PL/SQL package in Oracle which acts as a proxy for the sequence. Make sure that no application module does a seq.nextval directly. All requests to get the nextval have to go through the getNextSeq() function/procedure provided by the PL/SQL pacakge.

The getNextSeq() function first acquires a lock in shared mode provided by the dbms_lock package in Oracle. Increments the sequence. Releases the lock, and returns the nextval. (Note: The dbms_lock package is a general purpose lock library. It does not mean that you acquired a lock on the sequence - that's not possible in Oracle, as some gurus told me. You just acquire a lock. Very much like a mutex lock you acquire in programming languages.)

In the same PL/SQL package, you create a function resetSeq() which acquires the same lock in exclusive mode, resets the sequence using either of the solutions given above, and releases the lock.

Now, you can create a scheduled job in the DB using the dbms_scheduler package to run the resetSeq() function at midnight everyday. In fact, you can execute that at whatever frequency/interval you want.

In the process I also realised that Oracle has the capability of doing a fair amount of metaprogramming through the EXECUTE IMMEDIATE statement. You can pass it any string and Oracle tries to parse + execute that as an Oracle statement, much like eval in Ruby or Lisp.

I spent most of my time breaking my head over why dbms_lock.request didn't seem to be working. Then I realized that dmbs_lock.allocate_unique worked on a per session basis. What that means is, if two separate connections called dmbs_lock.allocate_unique to create a lock with the same lock handle, both of them would end up getting lock objects that would behave independently of each other. In which case, effectively, both the processes were working on different locks and the whole purpose of using locks would be defeated. This is why, I'm calling allocate_unique just once, and "hard coding" the lock id in the package definition using Oracle "metaprogramming".

PS: I'm still not sure whether this is the best solution to this problem, but it's a solution that works. Please let me know if you have a better way of solving this.

Here's the script:


set serveroutput on;
declare
    lock_handle number;
begin
    dbms_lock.allocate_unique('my_lock', lock_handle, 31536000);
    dbms_output.put_line('Dropping my_seq');
    execute immediate 'drop sequence my_seq';

    dbms_output.put_line('Creating my_seq');
    execute immediate 'create sequence my_seq minvalue 1 nocache increment by 1';

    dbms_output.put_line('Creating my_pkg declaration');
    execute immediate 'CREATE OR REPLACE PACKAGE "my_PKG" AS
  procedure reset_my_seq;
  function get_my_next_seq return number;
END;';

    dbms_output.put_line('Creating my_pkg definition');
    execute immediate 'CREATE OR REPLACE PACKAGE BODY "my_PKG" AS
  my_lock_handle number := ' || lock_handle || ';
  
  procedure reset_my_seq is
    lock_status integer;
    curr_val integer;
  begin
    lock_status := dbms_lock.REQUEST(lockhandle => my_lock_handle, lockmode => dbms_lock.x_mode);

    execute immediate ''alter sequence  my_seq minvalue 0'';
    execute immediate ''select my_seq.nextval from dual'' into curr_val ;
    execute immediate ''alter sequence my_seq increment by -'' || curr_val ;
    execute immediate ''select my_seq.nextval from dual'' into curr_val ;
    execute immediate ''alter sequence my_seq increment by 1'';

    lock_status := dbms_lock.release(lockhandle => my_lock_handle);
  end; 
  
  
  function get_my_next_seq return number is
    lock_status integer;
    next_seq number;
  begin
    lock_status := dbms_lock.REQUEST(lockhandle => my_lock_handle, lockmode => dbms_lock.s_mode);
    select my_seq.nextval into next_seq from dual;
    lock_status := dbms_lock.release(lockhandle => my_lock_handle);
    return next_seq;
  end;
END;';


    -- dbms_output.put_line('Dropping scheduler job');
    --  dbms_scheduler.drop_job('reset_my_seq');

    dbms_output.put_line('Creating scheduler job');
    dbms_scheduler.create_job( job_name=> 'reset_my_seq',
    job_type=>'PLSQL_BLOCK',
    job_action=> 'begin my_pkg.reset_my_seq; end;',
    start_date=> systimestamp,
    repeat_interval=>'frequency=daily;interval=1;byhour=0; byminute=0; bysecond=0;',
    enabled=>true);
    commit;
end;
/