Jack Nicholson was an unconventional choice to play the Joker...
Not really. IIRC, when they first started floating the idea of a Batman movie in the 80s Jack's name came up all the time for the Joker. Not only was he one of the biggest movie stars of the era, but he was known for "Cuckoos Nest," "the Shining," "the Witches of Eastwick" and other films where he played a grinning maniac.
Yeah, he wasn't the long, lanky, Joker of the comic books but considering the previous Joker was Cesar Romero with his painted over mustache Jack seemed like a big step up in any analysis.