Today, we will be discussing the dos and don'ts of back testing. Back-testing is a highly practical and a very real way to test the strength of certain beliefs and ideas you have about the market. Do you think that a market above the 200-day is bullish? Test it. For instance, we have found that short-term bottoms in the SPX tend to perform worse when below the 200-day line, but much better when above the line. We will discuss how to choose time periods to test and to evaluate the strength of your system.
The first thing traders should understand is that we have the technology to remove much of the subjectivenesss and mystery about the market. You need to answer these questions before you trade:
Testing also benefits the trader psychologically and many would argue that psychology is more important than market knowledge when trading. After all, an uninformed conservative trader often does better than an intelligent, reckless trader. Testing fills a trader with confidence and decreases the likelihood that he will be too scared to trade when the moment is right. Also, testing makes the trader aware of what drawdowns might look like and what sort of weaknesses his system has. Let's move on to the do's and don'ts of testing procedures.
Make sure that you don't mix your in-sample data with your out-of-sample data. For example, the best way to test is to form a list of stocks, and test them. Then optimize on a different list of stocks. Finally, test the strategy on a completely different set of stocks in a different time period from the original test. If there is no overlapping in the data, then you wont have to worry so much that you've "overfit" the data for the past. Remember that one danger of optimizing too much is that the strategy might not work unless markets behave exactly the same as they did during the optimization period. Another effective means of testing is to do simulated trading for a multi-month period. You can then test your strategy without risking money on it. Also, since you will be more emotionally detached at this point, it will be an opportune time to hone the sytems.
You should also decide how you will treat outlier trades in your system. You may want to include outlier trades if your system depends on them. Otherwise, your results may suggest that the system is not profitable when in fact it is. Be sure to keep in mind that there is more to a good system than net profit. I like to pay attention to standard deviation or coefficient of variation as measures of risk. Return is always relative to risk. Keep these points in mind when developing a system and be sure to stick to the system once you start trading. Large run-ups and drawdowns tend to cause system traders to change their systems to their dismay.
Price Headley is the founder and chief analyst of BigTrends.com.