Behavioural finance is a branch of financial economics that deals with the the behaviour of investors. In common with other branches of economics, financial theory tends to make fairly simplistic assumptions about the behaviour of people - that investors act rationally to maximise their wealth (strictly speaking, their utility).
Phenomena such as investment bubbles demonstrate that investors are far from uniformly rational, and that the behaviour of markets in general shows that behavioural factors are important.
In recent years both the broader discipline of economics and the theory of finance have developed theories that do take behavioural factors into account. At the moment these studies, although interesting, do little to help investors, except to warn them against known common mistakes.