Techtel’s IT spending forecasts came our way today. It’s a fascinating piece of empiricism. Techtel’s forecast averages estimates, by a panel of interested individuals, about changes in the IT environment . What Techtel does is certainly easier than tracking spending itself, however the results are worthless. Their figures for 2004, for example, show average growth of 17.5 percent, trending up to 21.2 percent in the final quarter. IDC’s research, which is similar in findings to those of most analysts, shows growth of around one third that amount.
According to a Bitpite study of research by different analyst firms, the spread of 2004 forecasts was between zero and eight percent. Somehow Techtel’s sample returned results well outside the range of those collected forecasts. Indeed, their figure is more than double the upper limit . Because their sample is not very small, my suspicion is that there must be a huge selection bias in the way they are choosing who to survey. If that’s the case, then Techtel’s samples cannot produce statistically significant results, even if the samples were doubled or tripled in size.
Techtel needs to think carefully about the sampling and selection techniques its is using to form its panels . Techtel’s sample is used widely, for example to provide the vast majority of those questioned for an annual technology buyers study used by some of the largest US-headquartered technology firms. Their clients, and their clients’ clients, deserve better.
 The method involves asking buyers whether their spending will fall or rise, and then subtracting the former from the latter to measure the size of the gap. The resulting number is called ‘Net Increased Spending’, which reflects intended spending. A similar method is ued to calculate something they term a ‘Purchasing Index’ which reflects ‘actual demand’ for key market segments. These ‘actual’, ‘purchasing’, numbers are also surprising: the Index shows ‘actual’ ‘Enterprise Purchasing’ rising 37.5% from 2003 to 2004. Considering the far more modest change in real spending, it is hard to see the insight these data give.
 One could say, in their defence, that there could be some small print (which might not be not publicly available) that explains that their forecasts, and ‘actual’ numbers, are not measuring the same as the analyst houses. But, frankly, is that the impression that the presentataion of these data give? Imagine if one argued that the data were consistent: one could say both that overall dollar spending rose around 5.8%, and that 37.5% of enterpises were spending more, because on average those enterprises who were spending more had increased their spend by 0.15% (which is 5.8/37.5). This could give totally a different impression than Techtel’s current presentation of the data, and would be a deeply counter-intuitive argument. It seems much more probable that their research method is either broken or misleadingly explained.
 Or, it needs to be much clearer about what its research is, and is not, measuring. To say: ‘our survey shows the trends in a large sample of wide guesses about IT spending’, rather than suggesting the data show ‘actual’ and forecast spending.