Investigate before you invest: Common trading wisdom is uncommonly wrong.
One way that historical patterns aid our self-coaching is by helping us distinguish myth from fact. “The trend is your friend” we commonly hear. My research on the blog, however, has consistently documented worse returns following winning days, weeks, and months than following losing ones. It is not enough to accept market wisdom at face value: just as you would research the reliability of a vehicle before making a purchase, it makes sense to research the reliability and validity of trading strategies.
There are traders who make the opposite mistake and trade mechanically from historical market patterns. I have seen an unusual proportion of these traders blow up. Market patterns are relative to the historical period that we study. If I examine the past few years of returns in a bull market, I will find significant patterns that will completely vanish in a bear market. If I include many bull and bear markets in my database, I will go back so far in time that I will be studying periods that are radically different from the current one in terms of who and what are moving markets. Automated, algorithmic strategies have completely reshaped market patterns, particularly over short time periods. If you study precomputer-era markets you would miss this influence altogether. Select a look-back period for historical analysis that is long enough to cover different markets but not so lengthy as to leave us with irrelevant data is as much art as science.
My approach to trading treats historical market patterns as qualitative research data. In a nutshell, qualitative research is hypothesis-generating research, not hypothesis-testing research. I view the patterns of markets as sources of trading hypotheses, not as fixed conclusions. The basic hypothesis is that the next trading period will not differ significantly from the recent past ones. If a pattern has existed over the past X periods, we can hypothesize that it will persist over the next period. Like any hypothesis, this is a testable proposition. It is an idea backed by support, not just faith or superstition, but it is not accepted as a fixed truth to be traded blindly.
Historical testing yields hypotheses for trading, not conclusions.
For this reason, I do not emphasize the use of inferential statistics in the investigation of historical patterns. I am looking for qualitative differences much as a psychologist might look for various behavior patterns in a person seeking therapy. In short, I’m looking to generate a hypothesis, not test one. The testing, in the trading context, is reflected in my trading results: if my returns significantly exceed those expected by chance, we can conclude that I am trading knowledge, not randomness.
When we adopt a qualitative perspective, the issue of look-back period becomes less thorny. As long as we consider the results of historical investigations to be nothing more than hypotheses, we can draw our ideas from the past few weeks, months, years, or decades of trading. The basic hypothesis remains the same: that the next time period will not differ significantly from the most recent ones. With that in mind, we can frame multiple hypotheses derived from different patterns over different time frames. One hypothesis, for example, might predicate buying the market based on strong action at the close of the prior day, with the anticipation of taking out the previous day’s R1 pivot level. A second hypothesis might also entail buying the market based on a pattern of weakness during the previous week’s trading. When multiple independent patterns point in the same direction, we still don’t have a certain conclusion, but we do have a firm hypothesis.
When independent patterns point to similar directional edges, we have especially promising hypotheses for trading.
Of course, if we generate enough hypotheses, some are going to look promising simply as a matter of chance. We could look at all combinations of Dow stocks, day of week, and week of year and the odds are good that we’d find some pattern for some stock that looks enticing, such as (to invent one possibility) IBM tends to rise on the first Wednesday of months during the summer season. Good hypotheses need to make sense; you should have some idea of why they might be valid. It makes sense, for instance, to buy after a period of weakness because you would benefit from short covering and an influx of money from the sidelines. It doesn’t make sense to buy a stock on alternate Thursdays during months that begin with M—no matter what the historical data tell you.
When you’re first learning to generate good hypotheses, your best bet is to keep it simple and get your feel for the kinds of patterns that are most promising. Many of your initial candidates will emerge from investigations of charts. Perhaps you’ll notice that it has been worth selling a stock when it rises on unusually high volume, or that markets have tended to bounce following a down open that follows a down day. Such ideas are worth checking out historically. What patterns have you noticed in your trading and observation? Write down these patterns and keep them simple: these patterns will get you started in our qualitative research.
COACHING CUE
Several newsletters do an excellent job of testing historical patterns and can provide you with inspiration for ideas of your own. Check out the contributions of Jason Goepfert, Rob Hanna, and Rennie Yang in Chapter 9, along with their links. All three are experienced traders and investigators of market patterns.
LESSON 92: FRAME GOOD HYPOTHESES WITH THE RIGHT DATA
In the previous lesson, I encouraged you to keep hypotheses simple. This is not just for your learning; in general, we will generate the most robust hypotheses if we don’t try to get too fancy and add many conditions to our ideas. Ask a question that is simple and straightforward, such as, “What typically happens the week following a very strong down week?” This question is better than asking, “What typically happens the week following a very strong down week during the month of March when gold has been up and bonds have been down?” The latter question will yield a small sample of matching occasions—perhaps only three over many years—so that it would be difficult to generalize from these. While I will occasionally look at patterns with a small N simply as a way to determine if the current market is behaving in historically unusual ways, it is the patterns that have at least 20 occurrences during a look-back period that will merit the greatest attention. The more conditions we add to a search, the more we limit the sample and make generalization difficult.
The simplest patterns will tend to be the most robust.
Of course, the number of occurrences in a look-back period will partly depend on the frequency of data that you investigate. With 415 minutes in a trading day for stock index futures, you would have 8,300 observations of one-minute patterns in a 20-day period. If you were investigating daily data, the same number of observations would have to cover a period exceeding 30 years. Databases with high frequency data can become unwieldy in a hurry and require dedicated database applications. The simple historical investigations that I conduct utilize database functions in a flat Excel file. When I investigate a limited number of variables over a manageable time frame, I find this to be adequate to my needs. Clearly, a system developer who is going to test many variables over many time frames would need a relational database or a dedicated system-testing platform, such as TradeStation. The kind of hypothesis-generating activities covered in this chapter are most appropriate for discretionary traders who would like to be a bit more systematic and selective in their selection of market patterns to trade—not formal system developers.
Before you frame hypotheses worthy of historical exploration, you need to create your data set. This data set would include a range of variables over a defined time period. The variables that you select would reflect the markets and indicators that you typically consult when making discretionary trading decisions. For instance, if you trade off lead-lag relationships among stock market sectors, you’ll need to include sector indexes/ETFs in your database. If you trade gap patterns in individual stocks, you’ll need daily open-high-low-close prices for each issue that you trade at the very least. Some of the patterns I track in my own trading involve the number of stocks making new highs or l
ows; this is included in my database with separate columns on a sheet dedicated to each.
As you might suspect, a database can get large quickly. With a column in a spreadsheet for each of the following: date, open price, high price, low price, closing price, volume, rate of change, and several variables (indicators) that you track, you can have a large sheet for each stock or futures contract that you trade—particularly if you are archiving intraday data. I strongly recommend that beginners at this kind of historical investigation get their feet wet with daily data. This process will keep the data sets manageable and will be helpful in framing longer timeframe hypotheses that can supplement intraday observation and judgment. Many good swing patterns can be found with daily data and clean, affordable data are readily available.
Some of the most promising historical patterns occur over a period of several days to several weeks.
There are several possible sources for your historical database. Many real-time platforms archive considerable historical data on their servers. You can download these data from programs such as e-Signal and Real Tick (two vendors I’ve personally used) and update your databases manually at the end of the trading day. The advantage of this solution is that it keeps you from the expense of purchasing historical data from vendors. It also enables you to capture just the data you want in the way you want to store them. This is how I collect most of my intraday data for stock index futures and such variables as NYSE TICK. My spreadsheet is laid out in columns in ways that I find intuitive. The entire process of updating a sheet, including built in charts, takes a few minutes at most.
A second way you can go, which I also use, is to purchase historical data from a vendor. I obtain daily data from Pinnacle Data (www.pinnacledata.com), which includes an online program for updating that is idiot-proof. Many of their data fields go back far in market history, and many of them cover markets and indicators that I would not be able to easily archive on my own. The data are automatically saved in Excel sheets, with a separate sheet for each data element. That means that you have to enter the different sheets and pull out all the data relevant to a particular hypothesis and time frame. The various fields can be copied onto a single worksheet that you can use for your historical investigations (more on this later). Among the data that I find useful from Pinnacle Data are advance-decline information; new highs/lows; volume (including up/down volume); interest rates; commodity and currency prices; and weekly data. These data are general market data, not data for individual equities. When I collect individual equity data, I generally find the historical data from the real-time quotation platforms to be adequate to my needs.
For the collection of clean intraday data, I’ve found TickData (www.tickdata.com) to be a particularly valuable vendor. The data management software that accompanies the historical data enables you to place the data in any time frame and store them as files within Excel. This is a great way to build a historical database of intraday information quickly, including price data for stocks and futures and a surprising array of indicator data.
If you go with a historical data vendor, you’ll have plenty of data for exploration and the updating process will be easy. Manual updating of data from charting platforms is more cumbersome and time-consuming, but obviously cheaper if you’re already subscribing to the data service. It is important thing that you obtain the data you most want from reliable sources in user-friendly ways. If the process becomes too cumbersome, you’ll quickly abandon it.
As your own trading coach, you want to make the learning process stimulating and enjoyable; that is how you’ll sustain positive motivation. Focus on what you already look at in your trading and limit your initial data collection to those elements. Price, volume, and a few basic variables for each stock, sector, index, or futures contract that you typically look at will be plenty at first. Adding data is never a problem. The key is to organize the information in a way that will make it easy for you to pull out what you want, when you want it. As you become proficient at observing historical patterns, you’ll be pleasantly surprised at how this process prepares you for recognizing the patterns as they emerge in real time.
COACHING CUE
Consider setting up separate data archives for daily and weekly data, so that you can investigate patterns covering periods from a single day to several weeks. You’d be surprised how many hypotheses can be generated from simple open-high-low-close price data alone. How do returns differ after an up day versus a down day? What happens after a down day in which the day’s range is the highest of the past 20 days? What happens after three consecutive up or down days? How do the returns differ following a down day during a down week versus a down day during an up week? You can learn quite a bit simply by investigating price data.
LESSON 93: EXCEL BASICS
In this lesson, I’ll go over just a few essentials of Excel that I employ in examining historical market data. If you do not already have a basic understanding of spreadsheets (how cells are named, how to copy information and paste it into cells, how to copy data from one cell to another, how to create a chart of the data in a sheet, how to write simple formulas into cells), you’ll need a beginning text for Excel users. All of the things we’ll be reviewing here are true basics; we won’t be using workbooks linking multiple sheets, and we won’t be writing complex macros. Everything you need to formulate straightforward hypotheses from market data can be accomplished with these basics.
So let’s get started. Your first step in searching for market patterns and themes is to download your historical data into Excel. Your data vendors will have instructions for downloading data; generally this will involve copying the data from the charting application or from the data vendors’ servers and pasting them into Excel. If, for instance, you were using e-Signal (www.esignal.com) as a real-time data/charting application, you would activate the chart of the data you’re interested in by clicking on that chart. You then click on the menu item Tools and then click on the option for Data Export. A spreadsheet-like screen will pop up with the chart data included. Along the very top row, you can check the boxes for the data elements you want in your spreadsheet. If there are data in the chart that you don’t need for your pattern search, you simply uncheck the boxes for those columns.
On that spreadsheet screen in e-Signal, if you click on the button for Copy to Clipboard, you will place all of the selected data on the Windows clipboard, where the data elements are stored as alphanumeric text. You then open a blank sheet in Excel, click on the Excel menu item for Edit, and select the option for Paste. That will place the selected data into your Excel spreadsheet.
If you had wanted more historical data than popped up in the e-Signal spreadsheet-like screen, you would have to click your chart and drag your mouse to the right, moving view of the data into the past. Move it back as far as you need and then go through the process of clicking on Tools, selecting Data Export, etc. If you need more historical data than e-Signal (or your current charting/data vendor) carries on their servers, that’s when you’ll need to subscribe to a dedicated historical data source such as Pinnacle Data (www.pinnacledata.com).
If you need data going back many years for multiple indicators or
instruments, you’ll want to download data from a historical data
vendor who has checked the data for completeness and accuracy.
If you’re using Pinnacle Data, you can automatically update your entire database daily with its Goweb application. The program places all the updated data into Excel sheets that are stored on the C drive in a folder labeled Data. The IDXDATA folder within Data contains spreadsheets with each instrument or piece of data (S&P 500 Index open-high-low-close; number of NYSE stocks making 52-week highs) in its own spreadsheet. Once you open these sheets, you can highlight the data from the historical period you’re interested in, click on the Edit menu item in Excel, click on the Copy option, open a fresh, blank spreadsheet, click on Edit, and then click on the Paste option. By copying from the Pinnacle sheets an
d pasting into your own worksheets, you don’t modify your historical data files when you manipulate the data for your analyses.
Personally, I would not subscribe to a data/charting service that did not facilitate an easy downloading of data into spreadsheets for analysis. It’s also helpful to have data services that carry a large amount of intraday and daily data on their servers, so that you can easily retrieve all the data you need from a single source. In general, I’ve found e-Signal and Pinnacle to be reliable clean sources of data. There are others out there, however, and I encourage you to shop around.
When you download data for analysis, save your sheets in folders that will help you organize your findings and give the sheets names that you’ll recognize. Over time, you’ll perform many analyses; saving and organizing your work will prevent you from having to reinvent wheels later.
Once you have the data in your sheet, you’ll need to use formulas in Excel to get the data into the form you need to examine patterns of interest. Formulas in Excel will begin with an = sign. If, for example, you wanted to calculate an average value for the first 10 periods of price data (where the earliest data are in row 2 and later data below), you might enter into the cell labeled D11: “=average(C2:C11),” without typing the quotation marks. That will give you the simple average (mean) of the price data in cells C1 through C10. If you want to create a moving average, you could simply click on the D10 cell, click the Excel menu item for copy, left-click your mouse and drag from cell D11 down, and release. Your column D cells will update the average for each new cell in column C, creating a 10-period moving average.
The Daily Trading Coach Page 42