I read up the specification (RFC3484) but I am none the wiser. Linux allows applications to set the precedence value, effectively meaning that the app selects the destination address, but labels can be added (ip addrlabel) to specify a source address.
The specification says that the largest prefix is always used when looking up the label database. What do the specific label numbers define? The type of routes?