Big Mike's Trading Blog

Day trading futures, discussing money management and trade management techniques, and more

MultiCharts vs. NinjaTrader backtesting and optimizing strategy

In this video I conducted a head-to-head comparison of MultiCharts and NinjaTrader evaluating which performed better for backtesting and optimization.


The full discussion is on the forum here:
http://www.bigmiketrading.com/platforms-indicators/3328-video-multicharts-vs-ninjatrader-strategy-backtesting-optimization.html

The data was provided by IQFeed, for ES S&P 500, 5-minute bars over a period of 2 years (3/22/2008 to 3/22/2010). RTH market hours were used. Where possible, every setting was exactly duplicated in both NinjaTrader and MultiCharts.

The latest version of both platforms was used:
- MultiCharts 6.0 beta 2
- NinjaTrader 7 beta 11

Before we begin, you need to know a few things. First, if I made any mistakes here it was not intentional and if you should feel free to correct me. Second, this video and the below post summarizes how I feel. Keyword being "I". Each trader has their own method and everyone has their own ideas of what is important and what isn't. I've focused on what I know, and what is important to me. It may have little relevance to you, or you may find that things I have left out that mean nothing to me in fact mean much to you.

Here is the video comparison, remember to watch in HD mode and full screen for best viewing:
Review: MultiCharts vs NinjaTrader backtesting strategy from Big Mike Trading on Vimeo.


My rig is a Core i7 920 overclocked to 4ghz with 12GB of ram, hardware RAID 3Ware 9690SA with (4) 750GB drives in a RAID 0+1, running Windows 7 x64. No antivirus and minimal background processes were running during the test.

Raw execution speed:
MultiCharts 6.0 beta 2: 6m 50s
NinjaTrader 7 beta 11: 6m 55s

Platform features:
MultiCharts has IOG (intrabar order generation) which allows it to submit orders intrabar. This is equivalent to NinjaTrader's "calculate on bar close = false", but without all the nasty side effects and repercussions that one must suffer through when using COBC=false in NT. An example would be on a 5-minute bar, MultiCharts can look intrabar and see that your condition (in this case, our moving averages and CCI) were met intrabar (say on minute 3 out of 5) and submit an order without waiting for the bar to close.

NinjaTrader is not capable of submitting intrabar orders in backtesting. Live strategies can submit intrabar orders using calculate bar close = false. Trying to simulate IOG within a backtestable strategy in NinjaTrader requires writing custom code to use MTF and try to work-around the problem, and it adds a great deal of complexity, plus MTF doesn't even work in current beta's of NinjaTrader.

MultiCharts also has Bar Magnifier. This allows MultiCharts to more accurately determine the OHLC (open/high/low/close) for each bar. If you are using a 5 minute bar but have enabled Bar Magnifier, you can select an improved resolution such as 1 minute, or even 1 tick, so you can accurately know the OHLC order. Why is this important? Without it knowing the OHLC, if your strategy enters on the Close of the prior bar and has a profit target that falls within the High of the current bar but also has a stop that falls within the low of the current bar, how is the platform to know if the trade was a winner or loser? Was the high made before the low, or vice versa? One condition would cause a winner, and one condition would cause a loser. MultiCharts has the solution with Bar Magnifier. With bar magnifier turned off, MultiChart's is set to assume the worst-case scenario, which is far better than assuming the best case and then being in for a rude awakening when running the strategy live.

NinjaTrader assumes the best case scenario. If the high of the bar is higher than your profit target, then your profit target will be met. The only way to try to work around this is to again develop a custom complex strategy that uses MTF and submits orders to a smaller time frame but manages entry signals from the larger original time frame.

MultiCharts has far superior reporting capabilities. The backtest performance report is far more detailed and has loads more information. You can also export it to Excel with far more detail than what NinjaTrader offers in its export.

NinjaTrader does have one nice feature MultiCharts is lacking, the ability to narrow the report down by hour (say 8am-9am) so you can see what time periods during the day your strategy performs at its best and at its worse. Nicely done. I'll be making a feature request to MultiCharts to add this.

The language behind the strategy:
All the fancy features in the world won't help if your platform simply cannot be made to do what you want. MultiCharts uses EasyLanguage, which is obviously widely used by TradeStation. On the other side of the ring is NinjaTrader which uses C#.

I've always been a programmer and have learned many languages, all self taught. But I don't think I am the norm when it comes to successful traders. And in fact, I think for the most part a computer geek that knows how to program probably makes a poor trader, because their time is spent wrapped up in code and not in psychology -- where the real money is at

Alright, so EasyLanguage is just that - easy. I've picked up all the basics in about two days. It's intuitive. In fact, it's so intuitive I've often be struck and just how easy it is. Not easy as in simple, but easy as in -- in just works!

On the other hand, C# is incredibly powerful. If you want your Ninja strategy to use neural networks, land on the moon, cook you breakfast and not even break a sweat, then C# is the way to go.

But what if you could have both? In essence, MultiCharts lets you do both because you can use external DLL's which means your EasyLanguage indicators can call and reference C# DLL's!

I know that there is a die hard debate of EasyLanguage vs. C#. But having used both now, I don't think there is a clear winner. The right choice will be decided by what the individual traders needs are and what they are trying to accomplish. My final advice on this topic is this: more complex is rarely better.

Usefulness of backtest, which is what it's all about, right?:
With MultiChart's IOG (intrabar order generation) and Bar Magnifier, I find myself only interested in the MultiChart's backtest results.

Since NinjaTrader has neither feature, you are left with either accepting known-faulty results, or trying to write an extremely complex MTF strategy to work around these limitations and get you to the end result you want. Writing MTF strategies is very complex, and doesn't even work in the latest beta's of NinjaTrader.

Final comments:
In my opinion, MultiChart's is the clear victor. But, I am not happy that MultiChart's soundly trounced NinjaTrader. I own NinjaTrader and have been a diehard fan. But in the last two years the NT7 snafu has just left me flabbergasted and sickened. There are only so many times you can hear "we're unable to duplicate that" and be treated like you are the only person experiencing problems before you just give up. The same is true of "we'll add that to our list of considerations". After two years of waiting, NT7 still doesn't address half of the things I felt were important, and instead adds dozens of things that were meaningless to me, and even removes some functionality that used to be there!

Competition is good, and I think that NT7 has been a very humbling experience for the NinjaTrader executive team. If they can recover, then they hopefully won't make the same mistakes again. The competition is catching up quickly, and MultiCharts 6.0 is just the beginning. MultiCharts plans to release v7.0 of their product around the end of June 2010, and TradeStation 9 is also coming soon. No one is standing still, and in the end, traders win.

You can find more discussion on MultiCharts here:
MultiCharts, MultiCharts, MultiCharts...

You can find more discussion on NinjaTrader here:
The Truth: NinjaTrader

I encourage and welcome you to post your own reviews. I have focused on what is important to me so I am biased in that regard, and you will focus on what is important to you. Hopefully with enough information people can make some education decisions without having to go through all the leg work.




You can participate on the trading forum here:
http://www.bigmiketrading.com/platforms-indicators/3328-video-multicharts-vs-ninjatrader-strategy-backtesting-optimization.html

Mike

0 comments:

Post a Comment

Note: Only a member of this blog may post a comment.