Why does pollen irritate humans so much?

The main reason is many types of pollen are covered in jagged edges. Look at these electron microscope pictures:

Pollen is also made out of one of the most sturdy naturally occuring organic material in existence, sporopollenin. This stuff is so resistant to breaking down we can find pollen that is thousands of years old and use it to create climatic reconstructions.

Finally, many plants are wind pollenated. To have any chance of hitting another flower from the same species, they throw millions (~3 million pollen spores are released from each Cannabis flower, for example). Some other plants, such as orchids, produce as few as 2 pollen grains, but they have to be "hand delivered" by bugs. These insect pollenated plants are not the ones that are irritating your nose.

Source: I studied palynology in school.

Edit: It was pointed out below that in addition to the jagged shapes, there are also defensive proteins in pollen that additionally provoke an immune response in some people. I acknowledge this is also a contributing factor.