The Date specification requires lowercase d and y so the custom format can be assigned as :
SYSTEM.DATE.CUSTOMFORMAT1$="ddMMMyy";
If this needs to be applied to a column note the sequence (here being applied to column 1) : wks.col1.subformat=20; // First set the subformat to use CUSTOMFORMAT1 wks.col1.format=4; // Then set the column format to use Date