Goodwill is a balancing number that allows double entry bookkeeping to remain consistent when acquisitions occur. The fundamental reason to ignore it when valuing companies is that goodwill is based on a company valuation itself. The goodwill added to the balance sheet as the result of an acquisition is the difference between the price paid to buy the company acquired and the book value of its assets. Market cap minus net assets. At a per share level it is It is the difference between net assets per share and the share price at the point of acquisition. It allows the past share price of a part of a company to influence the current share price of the whole — something that is very difficult to justify.
Goodwill also allows the history of companies to influence their current numbers, distorting comparisons between similar companies with different histories. Suppose two companies own very similar assets (other than goodwill), but one grew through acquisitions while the other grew organically, their balance sheets will look very different. Goodwill impairments can similarly distort comparisons between the profits of companies that have goodwill on their balance sheets and those that do not.
The value of goodwill is also very uncertain. It reflects the value of the acquired business at point of acquisition, but that value will fluctuate as the prospects for the business change after acquisition. This is why large goodwill impairments are often necessary.
There are similar difficulties in dealing with other intangible assets: inter-company comparisons are distorted by different accounting policies and histories.
Goodwill is sometimes relevant to metrics used by investors. For example, when calculating a ROIC you may well wish to take into account how successful (or, probably, not successful) acquisitions have been in generating returns for shareholders. In that case you are likely to want to include the goodwill.