Big Mike's Trading Blog

Day trading futures, discussing money management and trade management techniques, and more

MySQL tick database

Finally, as of about a week ago, I have a fully functioning C# "indicator" for NinjaTrader that captures tick data including bid/ask/last/volume and shoves that over into a MySQL database.

This is something I've been wanting to do for a while. I now have a separate box that runs Ninja 24/7 and captures tick data from ES, NQ, TF, YM, ZB, FESX, FDAX, and 6E. I want to add some currency pairs as well, but I need to work out issues with Zen Fire and AMP Trading (resolution seems to be to use a GAIN account for live data feed instead of Zen).

I plan to use this massive tick database to trade better. I want to develop solutions that analyze the data and make better trading decisions based on probabilities. Many people refer to this as neural networks (NN), whereby the system adapts to the data and refines itself along the way.

I can't simply rely on built-in tools within Ninja because of massive code issues that severely limit the amount of historical tick data you can utilize. That is why I wanted to build my own database. I figure it will take weeks, possibly months, before I start to have a real solution in place to analyze the data with. But, by then, I'll have lots of data!

The database is already over 1GB in size!

I am also exploring uses for logging real-time trades into the database that my automated strategies place. This is a separate project that I am working on, whereby the automated strategy can also learn from past trades and adjust itself as it goes. There are a few threads at the Ninja forums I've started about this, here is one of them.



iwannatoscript said...

Great Mike,

i am following yours' blog.

You write all things that i am looking for.

i'd like to build a database in mysqlto collect data like you.

Why don't you make a how to about building database for ninja?

I'd like to use the data in excel or calc (i prefer this because it opensource and better without problem of anyupgrade) analisys.

I really hope that you could hear this post.

i try in ninjatrader forum but not many people understand the power and how good os this kind of works.

Keep in touch.

Big Mike said...

Hi iwannatoscript,

I am still working out some bugs with the MySQL database. I've improved it quite a bit already, so that it queues up sql insert commands into an array that is configurable, so you can only contact SQL server every 100 ticks, or whatever you want -- instead of every single tick like the original design.

But there are still some problems that need to be solved. Once they are, I will post the code. We think alike regarding the "how to" tutorial, I plan to do that.

You can easily access the MySQL db via Excel, I did it in a matter of minutes from start to finish. I agree there is a lot of potential in doing so, in fact, that was the original reason I started recording tick data to the database.


iwannatoscript said...

Thaks Mike,

I am happy to hear you.

Sorry my English is basic.

I'd like to do something, not only recive all works that you are doing.

I am interested on learning these things.

Please, if you want, try to think somethings that we could do togheter contact me by email or on skype. I have many ideas.


Post a Comment

Note: Only a member of this blog may post a comment.