Compensation Surveys Are Biased

Fred Cook, Chair, Frederic W. Cook & Co., Inc.

After years of watching companies conduct and use compensation surveys, I have become convinced that there is bias in the way surveys are constructed, interpreted, and usedparticularly at the management level.

Multiplied throughout the industry, the cumulative effect of this bias has driven pay levels upwardabove the "real" marketthereby contributing to layoffs and downsizing. The latter occurs when companies realize they cannot and need not sustain survey-inflated pay levels.

There are 12 common causes of this upward bias. My purpose is to alert compensation professionals and other users of surveys to this bias so they will be able to use survey data more effectively in the future.

1. User Bias. Companies that sponsor surveys often do so with an implicit (albeit unstated) objective: to show the company as paying either competitively or somewhat below the market so as to justify positive corrective action. Those who are contracted to conduct the survey and interpret its results on the company's behalf subconsciously take on these same objectives. This user bias does not by itself cause salary inflation, but it does create the climate in which upward bias occurs and is accepted.

2. Sample Bias. Companies like to compare themselves against well-regarded, high-paying, and high-performing companies. Those firms that participate in surveys drawing data from a large number of other organizations often have the ability, through computer technology, to create a subset of participants with whom they wish to compare themselves. This typically results in a higher competitive pay line than the general survey, thereby showing the user company in a more favorable (i.e., less competitive) light in terms of pay rates.

Alternatively, the survey may be custom designed, in which case the sponsoring company will tend to select a high-paying, high-performing group of companies against which to compare itself. The survey designers will exclude low-paying (and less well-regarded) companies. It is perfectly natural to want to compare oneself against the star performers in an industry; this reflects the company's goal to become more like the leaders.

This would be fine if surveys compare relative performance as well as relative pay. Most do not. Actually, paying less than others may be quite appropriate if the company's performance is low relative to the survey sample. Likewise, high pay may be justified by high performance.

What happens, however, is that the low-paying company will downplay its relative performance and use the survey to justify pay increases, while the high-paying company justifiably maintains its high position based on its relative performance. The net effect, over time, is upward movement in competitive pay levels.

3. Survey Selectivity. Most companies have access to several surveys covering the same population. In cases where different surveys show the company in different competitive positions (some more favorable than others), compensation professionals tend to disregard, challenge, or downplay those surveys that do not show the company in the desired competitive position.

4. Scope Bias. Survey professionals accept the idea that the relative size of the organization should influence its pay, particularly at upper management levels. Yet "size" can be measured in any number of ways: revenues, equity, assets, market capitalization, net income, etc. Sponsoring organizations tend to select size variables that let them compare themselves favorably to the survey companies. Thus, if revenues are high but market capitalization is low, they will select revenues as the variable.

If every company in the survey has the ability to select the size or performance variable that favors itself, then it is technically feasible for all companies to show themselves as paying below the market. When this happens, what is the real market?

5. Compensation Selectivity. A total compensation package is composed of many elements, whereas most surveys tend to focus on a few, such as salary and bonuses. It is natural for companies that have a competitive total compensation package to be light on some pay elements and heavy on others. Companies with very generous benefit packages, supplemental executive retirement plans (SERPs), or large equity grants tend to disregard or downplay those pay elements in conducting surveys. If a company surveys only those areas where it is light (cash compensation, for example), it should not interpret or use those findings in isolation. If it does, it will be raising its total compensation levels above the market.

6. Benchmark Bias. In submitting survey data and interpreting the results, companies must match their positions against positions in the survey. In doing so, they tend to match their positions against those that have higher responsibilities and hence higher compensation. One company's positions may not have the full scope of responsibilities typical of that position in other companies. But the interpreter will tend to disregard this when comparing pay levels. Conversely, if a company's position shows up as highly paid relative to the survey, the interpreter will explain that away by saying that the position has more responsibilities, a different reporting relationship, or other factors that justify higher pay. Since very few benchmark positions are perfect matches, it may be quite appropriate to adjust survey data for differences in position responsibility. But, if this is done to explain and justify a higher-paid position, then equal efforts should be made to explain and justify those that are paid below the market.

7. Statistical Bias. Survey professionals use a variety of statistical techniques to interpret the survey results and determine where a particular company stands against the survey population. Single and multiple regression techniques are often used to supplement straight statistical averages, medians, and quartiles. A sponsor motivated by a desire to have the company appear in a good light can select, from among the statistical techniques available, those that will support this purpose. This problem will grow worse as sponsoring companies become more sophisticated in using option-pricing models to compare the values of equity-based, long-term incentive grants. By manipulating the variables and assumptions in ways favorable to the company, the interpreter may be able to make an above-competitive grant practice appear less generous and the grant practices of others appear more generous.

8. Converting Actual Bonuses into Target Bonuses. Many companies use surveys to assess and reset their target bonus practices as a percent of salary. However, most surveys collect data on actual bonuses, not target bonuses. In times of strong economic expansion and performance, it is common for actual bonuses to be above the target level. Companies that rely on high actual bonuses reported in the survey to justify raising their target bonuses lead the way in escalating total pay levels over time.

9. Results Bias. Having a strategy of paying competitively based on performance means a company is obligated to offer a competitive pay opportunity, not guarantee a competitive result. For example, a position may have a target bonus payout of 30% of salary, set so that the base salary plus the target bonus would result in competitive total annual pay when performance standards are met. However, there is no guarantee that the target bonus will be paid or that total pay will be competitive.

Companies may overlook this obvious truism when they have low or no payouts under annual bonus or long-term incentive plans in comparison with others that have made higher payouts, perhaps because of better performance. Low payouts relative to the competition may be perfectly justifiable and not a cause for corrective action. Surveys that include payouts from variable pay plans should be interpreted with reference to relative performance data. In the absence of such data, surveys should focus on competitive opportunities, not results.

10. Misinterpretation and Normalization of Special Equity Grants. Increasingly, companies make special equity incentive grants to their key people. For example, instead of making smaller stock option grants every year, a company may accelerate option grants into a single, jumbo grant. Or it may give executives a special, one-time equity grant to signal a significant event, such as a new strategy or a merger. Finally, when hiring a senior executive from outside, the employer may use special equity grants to induce the executive to take the job or to make up for benefits lost in the job change.

These special grants more often than not are included in the survey collection process. Without knowing the reasons

for the grants, the survey interpreter may assume that they are part of a normal granting practice. Even knowing a grant is a one-time special event, the interpreter may normalize it by assuming it will be repeated every three or five years. The net effect is to raise the survey averages to new levels and to create a situation where special grants by one company become built into normalized practices by the survey population.

11. Value Bias. Some employees are true "value builders" and rewarded as such. Others are "value maintainers," and some are, in fact, "value destroyers" because they are paid more than they are worth. Surveys, however, do not distinguish among these categories. When a survey includes pay data from across the value spectrum, survey averages are pulled up by the justifiably high pay of the value creators. These averages, however, are used to rationalize higher pay levels for value maintainers and destroyers as well in order to be "competitive."

12. No-Decline Bias. Compensation professionals have grown to expect that survey data will show pay rates rising every year. A conceptual framework does not even exist for handling marketplace declines in pay. However, with all the layoffs, downsizings, and early retirements of higher-paid workers, one would think surveys would show declines in market pay rates.

Even if a survey shows a decline in pay rates, the user will tend either to disregard or reinterpret the data to reflect an increase. Also, survey purveyors know that steady increases in competitive pay levels are good for business, and declines are not. This, combined with user bias, may explain why surveys continue to show increasing pay levels.

Correcting the Bias

If there were no compensation surveys, there probably would be wider disparities among company pay practices than exist today. These disparities would tend to be more economically justifiable, both on the high and the low side. Natural economic forces would operate to create a market in which people would be paid according to their relative worth. Surveys tend to narrow the disparities, particularly on the low side, because low-paying companies use them to justify pay increases that otherwise would not be justified. They tend to have less effect on the high side because high-performing companies can justify high pay and will do so rather than using the survey to depress their pay levels. By pulling up the bottom, surveys raise the average for everyone, thereby causing an upward escalation in overall pay levels.

How can you avoid the negative effect of bias? I propose three solutions. First, acknowledge that upward bias exists in surveys and act to correct it where possible. Second, downplay the importance of surveys use them as a check on where you are, not as a rationale for doing something different. And third, use relative size and performance comparisons in a justifiable and consistent manner in interpreting any survey results.