A boutique fund financial institution, such as a fund manger, broker, or investment bank, is one that is small and specialised. In fact the specialisation is not always evident, and many "boutique" institutions are, in fact, merely small.
A boutique may offer a different approach or a specialist focus. For fund managers merely being small can be an advantage. Where they are specialised, they may focus on a sector, or on serving a particular type of client.
For example, a boutique brokerage may be able to build knowledge and experience in a particular sector, and therefore offer particularly good sell-side research to investors in that sector. A boutique fund manager may be able to invest in securities that are too illiquid for a larger investor.
Boutiques may also be built around key individuals, so that, for example, a boutique fund management company will give a start fund manager a lot of autonomy to run a fund and therefore attract investors who have a high opinion of that particular individual.