In this article I will describe how our backtested algorithm can be used in live algorithmic trading. My broker provides me with the TWS (Trader WorkStation) API which is the solution that I use to build my trading application, obtain market and view account details such as open positions and available cash.
The basic idea of my trading strategy is to rank hundreds of stocks by value (
For the value factor I need price and fundamental data. The price data I request from my broker. I have a market data subscription for that. For fundamental data such as earnings and equity I use my own data source. For the later stay tuned as I will be making this fundamental data available sometime in the future.
I used Python to develop this strategy. You might need to signup for a paper account at your broker and make sure they are using TWS. For a full TWS API please refer to the official documentation.
The TWS API Architecture
I will not go into all the API details since the official documentation describes everything pretty well. But I believe it is good to at least describe the idea behind the TWS API architecture. Some of the text below is taken straight out of the official documentation.
First you need to start TWS on your machine to allow for incoming connections. TWS acts as a server to receive requests from the API application (our Python client/strategy) and responds by taking appropriate actions. The first step is for the API client to initiate a connection to TWS on a socket port where TWS is already listening.
Once the TWS is up and running and actively listening for incoming connections we are ready to write our code. We have to implement TWS API’s two major interfaces: EWrapper interface and the EClientSocket.
The EWrapper interface is the mechanism through which the TWS delivers information to the API client application (our strategy). By implementing this interface the client application will be able to receive and handle the information coming from the TWS.
The class used to send messages to TWS is EClientSocket. Unlike EWrapper, this class is not overriden as the provided functions in EClientSocket are invoked to send messages to TWS. To use EClientSocket, first it may be necessary to implement the EWrapper interface as part of its constructor parameters so that the application can handle all returned messages. Messages sent from TWS as a response to function calls in EClientSocket require a EWrapper implementation so they can processed to meet the needs of the API client.
TWS API programs always have at least two threads of execution. One thread is used for sending messages to TWS, and another thread is used for reading returned messages. The second thread uses the API EReader class to read from the socket and add messages to a queue. Every time a new message is added to the message queue, a notification flag is triggered to let other threads know that there is a message waiting to be processed. In the two-thread design of an API program, the message queue is also processed by the first thread. The thread responsible for the message queue will decode messages and invoke the appropriate functions in EWrapper.
In Python TWS API the EReader thread is automatically started upon connection to TWS. There is no need for user to start the reader.
Let us go ahead and implement the EWrapper interface first since that is where the data from TWS will be coming in. The code snippet below does not contain all the methods implemented for EWrapper as my intention is to illustrate the basic idea. The full implementation will be listed at the end of this article.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
class Wrapper(EWrapper): def __init__(self): wrapper.EWrapper.__init__(self) def accountSummary(self, reqId: int, account: str, tag: str, value: str, currency: str): super().accountSummary(reqId, account, tag, value, currency) global account_summary account_summary[tag] = value def accountSummaryEnd(self, reqId: int): super().accountSummaryEnd(reqId) global account_summary_end account_summary_end = True def tickPrice(self, reqId: TickerId, tickType: TickType, price: float, attrib: TickAttrib): super().tickPrice(reqId, tickType, price, attrib) if tickType == TickTypeEnum.CLOSE: global close_price close_price = price def nextValidId(self, orderId: int): global next_valid_order_id next_valid_order_id = orderId |
It is important to understand that the methods implemented in EWrapper will be called when the respective data (e.g.: account summary) that has been requested by the client will become available. Depending on the data requested the method might get called repeatedly as can be the case with tick data or it can be called as low as once. For the later the end is usually signaled using the counterpart *End method.
Let’s look at our EClient subclass that will be requesting our data from TWS. Again I am not showing the entire client implementation at this stage.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
class Client(EClient): def __init__(self, wrapper): EClient.__init__(self, wrapper) def get_close_price(self, symbol): contract = Contract() contract.symbol = symbol contract.secType = "STK" contract.currency = "USD" contract.exchange = "SMART" contract.primaryExchange = "NASDAQ" global close_price close_price = None reqId = self.get_next_request_id() self.reqMarketDataType(4) # 1 for live, 4 for delayed-frozen data if live is not available self.reqMktData(reqId, contract, "", False, False, []) while close_price is None: sleep(0.1) self.cancelMktData(reqId) return close_price def get_portfolio_value(self): request_id = int(time()) global account_summary_end account_summary_end = False self.reqAccountSummary(request_id, "All", "$LEDGER:ALL") while not account_summary_end: sleep(0.1) global account_summary return float(account_summary["TotalCashBalance"]) def get_valid_order_id(self): global next_valid_order_id next_valid_order_id = None self.reqIds(numIds=1) while next_valid_order_id is None: sleep(0.1) return next_valid_order_id |
Notice how the client receives an EWrapper object in its constructor. In the above snippet see how we are calling the TWS API methods using the keyword self(e.g.: self.reqAccountSummary(...)) from within our methods (e.g.: get_portfolio_value(...)). The API methods are available to us since our client is a subclass of EClient. In order to make it easier to read the code and distinguish between API provided methods and method written by me I will use snake_case as my naming convention for my methods.
Finally we create a new class called Strategy that is both an EClient and EWrapper that will be connecting to TWS.
Once the client is connected, an EReader thread will be automatically created to handle incoming messages and put the messages into a message queue for further process. If you are using the TWS API with other programming languages you might need to pass an EReaderSignal object to the EClientSocket's constructor. This object is used to signal that a message is ready for processing in the queue. In Python the Queue class handles this task directly so we don’t have to pass in any EReaderSignal objects to our client’s constructor.
Looking inside the EClient's connect(..) method we see that the EReader thread is created for us. This thread is used to read from the socket and add messages to the queue.
|
1 2 3 4 5 |
def connect(self, host, port, clientId): ... self.reader = reader.EReader(self.conn, self.msg_queue) self.reader.start() # start thread ... |
We also need to trigger Client::run(), where the message queue is processed in an infinite loop and the EWrapper call-back functions are then called. Let us briefly look inside the run() implementation of EClient. You don’t have to worry about this since this is already made available to you.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
def run(self): """This is the function that has the message loop.""" try: while not self.done and (self.isConnected() or not self.msg_queue.empty()): try: try: text = self.msg_queue.get(block=True, timeout=0.2) if len(text) > MAX_MSG_LEN: self.wrapper.error(NO_VALID_ID, BAD_LENGTH.code(), "%s:%d:%s" % (BAD_LENGTH.msg(), len(text), text)) self.disconnect() break except queue.Empty: logger.debug("queue.get: empty") else: fields = comm.read_fields(text) logger.debug("fields %s", fields) self.decoder.interpret(fields) except (KeyboardInterrupt, SystemExit): logger.info("detected KeyboardInterrupt, SystemExit") self.keyboardInterrupt() self.keyboardInterruptHard() except BadMessage: logger.info("BadMessage") self.conn.disconnect() logger.debug("conn:%d queue.sz:%d", self.isConnected(), self.msg_queue.qsize()) finally: self.disconnect() |
In order for the client to be able to trigger EWrapper call-back functions we need to provide it with a concrete EWrapper implementation – a Strategy instance in our case. We do that using constructor dependency injection.
|
1 2 3 4 5 6 7 8 9 10 11 |
class Strategy(Wrapper, Client): def __init__(self, ipaddress, portid, clientid): Wrapper.__init__(self) Client.__init__(self, wrapper=self) self.connect(ipaddress, portid, clientid) thread = Thread(target=self.run) thread.start() setattr(self, "_thread", thread) |
The wrapper methods are then triggered in the infinite loop listed in the run() method above using the Decoder object as soon as a message is available in the queue:
|
1 |
self.decoder.interpret(fields) |
The decoder interprets the fields and knows what method to call on the EWrapper object.
That is basically the high level overview of the TWS API architecture. Next we will be focusing on the concrete trading strategy.
The Strategy
The first thing to do is instantiate a new Strategy and connect to TWS:
|
1 |
app = Strategy(ipaddress="localhost", portid=1234, clientid=0) |
Next we will be fetching some stocks from our database that we will be processing:
|
1 2 3 4 |
fundamentals = Fundamentals('database.ini') y = year() q = last_quarter() stocks = fundamentals.get_stock_profiles(y, q) |
Fundamentals is my proprietary API that communicates with my fundamentals database. I am currently working on making this publicly available in the future. But for now you can assume that this returns a list of StockProfile objects from a MySQL database. Each StockProfile represents a company and has properties such as price_earning, price_book and industry. year() and last_quarter() just return the current year and quarter and are passed in get_stock_profiles(...) to fetch fundamental data for that reporting period.
We will now request the latest close prices for each stock in our list from TWS:
|
1 2 |
for stock in stocks: stock.price = app.get_close_price(stock.symbol) |
Filter out companies for which we don’t have enough data and companies whose industry is either Financials or Utilities:
|
1 2 3 4 5 6 7 8 9 10 11 |
def filter_stock_profiles(stock_profiles): filtered_stock_profiles = list(filter(lambda stock: (stock.price_earnings is not None) and (not math.isnan(stock.price_earnings)) and (stock.price_book is not None) and (not math.isnan(stock.price_book)), stock_profiles)) filtered_stock_profiles = list(filter(lambda stock: (stock.industry is not None) and ( stock.industry != 'Financials') and (stock.industry != 'Utilities'), filtered_stock_profiles)) stocks = filter_stock_profiles(stocks) |
Once we have filtered out all the unwanted companies we proceed with ranking them by value in descending order and fetching the top and bottom stocks for which we want to place an order:
|
1 2 3 4 5 6 |
def rank_stocks(stock_profiles): return sorted(stock_profiles, key=lambda stock: 1.0 / (float(stock.price_earnings) + float(stock.price_book)), reverse=True) stocks = rank_stocks(stocks) longs = stocks[:MAX_LONG_POSITION_SIZE] shorts = stocks[-MAX_SHORT_POSITION_SIZE:] |
Next we calculate the weights that will be used to rebalance our portfolio later. We want the top ranking stocks to have a heavier weight in our long position and for the shorts we want the lowest ranking companies to carry a greater weight. That is, we buy more stocks of our top companies and sell more of our worst companies. Notice also that I don’t check a stock’s liquidity or if I can actually trade a particular stock. It is probably a good idea to do that in order to diminish market impact. It might also be a good idea to invest less in stocks with very low liquidity. I don’t do that here for the sake of simplicity.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
portfolio_value = app.get_portfolio_value() amount = portfolio_value // TARGET_NUMBER_OF_POSITIONS my_positions = app.get_positions() amount_available_for_long = (portfolio_value // 2) number_of_long_units = (MAX_LONG_POSITION_SIZE * (MAX_LONG_POSITION_SIZE + 1)) / 2 amount_per_long_unit = amount_available_for_long // number_of_long_units for rank, stock in enumerate(longs): dollar_amount = (MAX_LONG_POSITION_SIZE - rank) * amount_per_long_unit amount = dollar_amount // stock.price stock.portfolio_weight = (amount * stock.price) / portfolio_value stock.order_amount = amount amount_available_for_short = (portfolio_value // 2) number_of_short_units = (MAX_SHORT_POSITION_SIZE * (MAX_SHORT_POSITION_SIZE + 1)) / 2 amount_per_short_unit = amount_available_for_short // number_of_short_units for rank, stock in enumerate(shorts): dollar_amount = (rank + 1) * amount_per_short_unit amount = dollar_amount // stock.price stock.portfolio_weight = (amount * stock.price) / portfolio_value stock.order_amount = -1 * amount |
Before we go about rebalancing our portfolio we will be exiting any positions that we have in companies that are not included in our ranking.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
order_book = [] # Exit positions that are not ranking any more for symbol in my_positions: position = my_positions[symbol] if position.position > 0 and not any(stock.symbol == position.symbol for stock in longs): order = MyOrder(position.symbol) order.description = "Closing long position." order.type = "EXIT" order.amount = -1 * position.position order.currency = position.currency order_book.append(order) for symbol in my_positions: position = my_positions[symbol] if position.position < 0 and not any(stock.symbol == position.symbol for stock in shorts): order = MyOrder(position.symbol) order.description = "Closing short position." order.type = "EXIT" order.amount = -1 * position.position order.currency = position.currency order_book.append(order) |
As you can see I am storing any potential orders in a list called order_book and don’t actually place them yet. The reason for that is that I want to display them on screen and ask the user to confirm them before sending them off to TWS for execution. I also don’t want this strategy to continue running if there are any open orders still existent. I do this as a precautionary measure:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
if len(order_book) > 0: confirm = input("Do you want to place the above exit orders (yes/no)? ") if confirm == 'yes': open_orders = app.get_open_orders() if open_orders: log.debugv("There are pending orders in the system.") log.debugv(print_open_orders(open_orders)) log.debugv("Not executing. Bye!") app.disconnect() exit() else: for order in order_book: if order.currency is not None: app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency) else: app.order_target_value(symbol=order.symbol, amount=order.amount) else: log.debugv("Decline execution accepted. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Portfolio seems up-to-date. Not executing. Bye!") app.disconnect() exit() |
Once I have exited all my unwanted positions I go ahead and calculate any adjustments that I need to undertake on any current positions and prepare the order for any new positions. Again I am storing any orders in the order_book list and ask the user to confirm them before sending the off to TWS:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
order_book = [] log.debugv("No remaining pending orders.") # Buy/Sell/Adjust for stock in longs: if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0): order = MyOrder(stock.symbol) order.description = "New long position {0} x {1}".format(stock.order_amount, stock.symbol) order.type = "NEW" order.amount = stock.order_amount order_book.append(order) else: position = my_positions[stock.symbol] target_position = stock.order_amount if position.position < 0: stock.order_amount = target_position + abs(position.position) elif position.position > 0: if position.position != target_position: stock.order_amount = target_position - position.position else: stock.order_amount = 0 if stock.order_amount != 0: order = MyOrder(stock.symbol) order.description = "Adjusting existing long position for {0} from {1} -> {2}".format( stock.symbol, my_positions[stock.symbol].position, target_position) order.type = "ADJUSTMENT" order.amount = stock.order_amount order_book.append(order) for stock in shorts: if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0): order = MyOrder(stock.symbol) order.description = "New short position {0} x {1}".format(stock.order_amount, stock.symbol) order.type = "NEW" order.amount = stock.order_amount order_book.append(order) else: position = my_positions[stock.symbol] target_position = stock.order_amount if position.position > 0: stock.order_amount = target_position - position.position elif position.position < 0: if position.position != target_position: stock.order_amount = target_position - position.position else: stock.order_amount = 0 if stock.order_amount != 0: order = MyOrder(stock.symbol) order.description = "Adjusting existing short position for {0} from {1} -> {2}".format( stock.symbol, my_positions[stock.symbol].position, target_position) order.type = "ADJUSTMENT" order.amount = stock.order_amount order_book.append(order) |
In the above code snippet you can see that I am also storing whether a particular trade is caused by a new position or adjustment. It might have been more simple to just exit all positions in my current portfolio and then just go long the top and short the bottom companies but if a position doesn’t change or just needs a minor adjustment this would lead to unnecessary transaction costs. You might even ignore an adjustment if it is too small.
Finally, I again do some checking for any open orders and ask the user to confirm the new portfolio before placing the orders with my broker:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
if len(order_book) > 0: confirm = input("Do you want to place the above orders (yes/no)? ") if confirm == 'yes': open_orders = app.get_open_orders() if open_orders: log.debugv("There are pending orders in the system.") log.debugv(print_open_orders(open_orders)) log.debugv("Not executing. Bye!") app.disconnect() exit() else: for order in order_book: if order.currency is not None: app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency) else: app.order_target_value(symbol=order.symbol, amount=order.amount) open_orders = app.get_open_orders() log.debugv("Waiting for any pending orders to be filled.") while open_orders: sleep(0.1) log.debugv("No remaining pending orders. Exiting. Bye!") app.disconnect() exit() else: log.debugv("Decline execution accepted. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Portfolio seems up-to-date. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Filter resulted in less than %d stocks." % (TARGET_NUMBER_OF_POSITIONS)) |
The Full Strategy Code
Below you will find the full code for the strategy. You simply have to run it using:
|
1 |
python strategy.py |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 |
# Import ibapi deps from ibapi import wrapper from ibapi.client import EClient from ibapi.contract import * from ibapi.common import BarData from ibapi.common import TickerId from ibapi.common import TickAttrib from ibapi.ticktype import TickType from ibapi.ticktype import TickTypeEnum from ibapi.order import * from ibapi.order_state import OrderState from threading import Thread from datetime import datetime from datetime import timedelta from time import time from time import sleep import pandas as pd from fundamentalslib.fundamentals import Fundamentals from fundamentalslib.fundamentals import StockProfile import random from termcolor import colored import math import io from logging import getLoggerClass, addLevelName, setLoggerClass, NOTSET import logging import logging.handlers # use epoch as first orderId at api app startup, and increment it with each order, this way it ensures uniqueness and bigger across multiple clientId and each app invocation next_valid_order_id = 0 MARKET_ID = 19004 TARGET_NUMBER_OF_POSITIONS = 10 MAX_SHORT_POSITION_SIZE = TARGET_NUMBER_OF_POSITIONS // 2 MAX_LONG_POSITION_SIZE = TARGET_NUMBER_OF_POSITIONS // 2 # Create a global DataFrame to insert the data in historical_data = pd.DataFrame(columns=['Date', 'Open', 'Close', 'High', 'Low']) positions = {} position_end = False open_orders = {} open_orders_end = False account_summary_end = False account_summary = {} close_price = None log = None DEBUG_LEVELV_NUM = 9 class MyLogger(getLoggerClass()): def __init__(self, name, level=NOTSET): super().__init__(name, level) addLevelName(DEBUG_LEVELV_NUM, "STRATEGY_DEBUG") def debugv(self, msg, *args, **kwargs): if self.isEnabledFor(DEBUG_LEVELV_NUM): self._log(DEBUG_LEVELV_NUM, msg, args, **kwargs) class Position(object): def __init__(self, symbol=None): self.symbol = symbol self.currency = None self.sec_type = None self.position = 0 def __str__(self): return "Symbol: {0} Currency: {1} Position: {2} Sec Type: {3}".format(self.symbol, self.currency, self.position, self.sec_type) class MyOrder(object): def __init__(self, symbol=None): self.symbol = symbol self.currency = None self.amount = 0 self.description = None self.type = None def __str__(self): return "Symbol: {0} Currency: {1} Amount: {2} Description: {3} Type: {4}".format(self.symbol, self.currency, self.amount, self.description, self.type) class Wrapper(wrapper.EWrapper): def __init__(self): wrapper.EWrapper.__init__(self) def historicalData(self, reqId: int, bar: BarData): global historical_data historical_data = historical_data.append({'Date': str(bar.date), 'Open': float(bar.open), 'Close': float( bar.close), 'High': float(bar.high), 'Low': float(bar.low)}, ignore_index=True) def historicalDataEnd(self, reqId: int, start: str, end: str): log.debugv("Done downloading historical data (request_id=%s)." % (reqId)) def accountSummary(self, reqId: int, account: str, tag: str, value: str, currency: str): super().accountSummary(reqId, account, tag, value, currency) # log.debugv("AccountSummary. ReqId:%s Account:%s Tag:%s Value:%s Currency:%s" % (reqId, account, tag, value, currency)) global account_summary account_summary[tag] = value def accountSummaryEnd(self, reqId: int): super().accountSummaryEnd(reqId) log.debugv("Done fetching account summary (request_id=%s)." % (reqId)) global account_summary_end account_summary_end = True # https://api.lynx.academy/Account&PortfolioData?id=positions def position(self, account: str, contract: Contract, position: float, avgCost: float): super().position(account, contract, position, avgCost) global positions pos = Position(contract.symbol) pos.position = position pos.sec_type = contract.secType pos.currency = contract.currency positions[contract.symbol] = pos def positionEnd(self): super().positionEnd() log.debugv("Done fetching positions.") global position_end position_end = True def tickPrice(self, reqId: TickerId, tickType: TickType, price: float, attrib: TickAttrib): super().tickPrice(reqId, tickType, price, attrib) if tickType == TickTypeEnum.CLOSE: global close_price close_price = price log.debugv("Got close price %s for request_id=%s." % (close_price, reqId)) def openOrder(self, orderId: int, contract: Contract, order: Order, orderState: OrderState): log.debugv("Open %s order with ID=%s Symbol=%s State=%s." % (order.action, orderId, contract.symbol, orderState.status)) global open_orders # https://interactivebrokers.github.io/tws-api/order_submission.html#order_status if orderState.status == 'PreSubmitted': open_orders[orderId] = order elif orderState.status == 'Filled' and orderId in open_orders: del open_orders[orderId] elif orderState.status == 'PendingCancel' and orderId in open_orders: del open_orders[orderId] def openOrderEnd(self): super().openOrderEnd() log.debugv("Done fetching open orders.") global open_orders_end open_orders_end = True def nextValidId(self, orderId: int): """ Receives next valid order id. The LYNXApi.EWrapper.nextValidID callback is commonly used to indicate that the connection is completed and other messages can be sent from the API client to TWS. There is the possibility that function calls made prior to this time could be dropped by TWS. https://api.lynx.academy/Connectivity """ log.debugv("Got next_valid_order_id=%d." % orderId) global next_valid_order_id next_valid_order_id = orderId class Client(EClient): def __init__(self, wrapper): EClient.__init__(self, wrapper) def get_historicalData(self, contract, duration="1 M", barSize="1 hour", reqId=1): log.debugv("Requesting historical data for '%s'." % (contract.symbol)) queryTime = (datetime.today() - timedelta(days=0)).strftime("%Y%m%d %H:%M:%S") # Define the end date of the query # Here we are requesting historical bar data for the the contract self.reqHistoricalData(reqId, contract, queryTime, duration, barSize, "MIDPOINT", 1, 1, False, []) MAX_WAITED_SECONDS = 5 log.debugv("Getting historical data from the server... can take %d second to complete." % MAX_WAITED_SECONDS) sleep(MAX_WAITED_SECONDS) global historical_data return historical_data def get_close_price(self, symbol): contract = Contract() contract.symbol = symbol contract.secType = "STK" contract.currency = "USD" contract.exchange = "SMART" contract.primaryExchange = "NASDAQ" global close_price close_price = None reqId = self.get_next_request_id() self.reqMarketDataType(4) # 1 for live, 4 for delayed-frozen data if live is not available self.reqMktData(reqId, contract, "", False, False, []) log.debugv("Getting close price for '%s' (request_id=%d) from the server." % (contract.symbol, reqId)) while close_price is None: sleep(0.1) self.cancelMktData(reqId) return close_price def get_positions(self): global position_end position_end = False self.reqPositions() while not position_end: sleep(0.1) global positions return positions def get_portfolio_value(self): request_id = int(time()) global account_summary_end account_summary_end = False self.reqAccountSummary(request_id, "All", "$LEDGER:ALL") while not account_summary_end: sleep(0.1) global account_summary return float(account_summary["TotalCashBalance"]) def order_target_value(self, symbol, amount, currency="USD"): if amount == 0: log.debugv("Error: amount cannot be zero.") return contract = Contract() contract.symbol = symbol contract.secType = "STK" contract.currency = currency contract.exchange = "SMART" contract.primaryExchange = "NASDAQ" # Define the order to place order = Order() if (amount > 0): order.action = "BUY" else: order.action = "SELL" order.orderType = "MKT" order.totalQuantity = abs(amount) self.place_order(contract, order) def place_order(self, contract, order): request_id = self.get_valid_order_id() log.debugv("%s (%s) %d %s (Order ID=%s)" % (order.action, order.orderType, order.totalQuantity, contract.symbol, request_id)) self.placeOrder(request_id, contract, order) def get_open_orders(self): log.debugv("Fetching any open orders.") global open_orders_end open_orders_end = False self.reqAllOpenOrders() while not open_orders_end: sleep(0.1) global open_orders return open_orders def get_next_request_id(self): sleep(2) request_id = int(time()) log.debugv("Generated the next request_id=%d." % (request_id)) return request_id def get_valid_order_id(self): log.debugv("Getting valid order id from the server.") global next_valid_order_id next_valid_order_id = None self.reqIds(numIds=1) while next_valid_order_id is None: sleep(0.1) return next_valid_order_id class Strategy(Wrapper, Client): def __init__(self, ipaddress, portid, clientid): Wrapper.__init__(self) Client.__init__(self, wrapper=self) self.connect(ipaddress, portid, clientid) """ API programs always have at least two threads of execution. One thread is used for sending messages to TWS, and another thread is used for reading returned messages. The second thread uses the API EReader class to read from the socket and add messages to a queue. Everytime a new message is added to the message queue, a notification flag is triggered to let other threads now that there is a message waiting to be processed. The thread responsible for the message queue will decode messages and invoke the appropriate functions in EWrapper. In the two-thread design of an API program, the message queue is also processed by the first thread. Once the client is connected, a reader thread will be automatically created to handle incoming messages and put the messages into a message queue for further process. User is required to trigger Client::run() below, where the message queue is processed in an infinite loop and the EWrapper call-back functions are automatically triggered. https://api.lynx.academy/Connectivity """ thread = Thread(target=self.run) thread.start() setattr(self, "_thread", thread) def last_quarter(): now = datetime.now() quarter_no = (now.month - 1) // 3 + 1 return "Q{}".format(quarter_no-1) def year(): return datetime.now().year def filter_stock_profiles(stock_profiles): """[summary] Keep only stock profiles whose industry sector is not 'Public Utilities' or 'Finance'. """ log.debugv("Filtering stocks.") filtered_stock_profiles = list(filter(lambda stock: (stock.price_earnings is not None) and (not math.isnan(stock.price_earnings)) and (stock.price_book is not None) and (not math.isnan(stock.price_book)), stock_profiles)) filtered_stock_profiles = list(filter(lambda stock: (stock.industry is not None) and ( stock.industry != 'Financials') and (stock.industry != 'Utilities'), filtered_stock_profiles)) return filtered_stock_profiles def rank_stocks(stock_profiles): """ Ranks stocks by value. Arguments: stock_profiles -- A list of stock objects to rank. """ return sorted(stock_profiles, key=lambda stock: 1.0 / (float(stock.price_earnings) + float(stock.price_book)), reverse=True) def print_stocks(stock_profiles, title, limit): writer = io.StringIO() writer.write("\n%s\n" % (title)) buf = ('%-4s %-6s %-35s %-25s %25s %25s %25s %25s\n' % ('#', 'Symbol', 'Name', 'Industry', 'PE', 'PB', 'Value', 'Weight')) writer.write(buf) writer.write('-' * len(buf)) writer.write("\n") for i, stock in enumerate(stock_profiles): name = stock.name[: 30] if stock.name is not None else "N/A" industry = stock.industry[: 30] if stock.industry is not None else "N/A" price_earnings = stock.price_earnings if stock.price_earnings is not None else 0.0 price_book = stock.price_book if stock.price_book is not None else 0.0 value = 1.0 / (float(stock.price_earnings) + float(stock.price_book)) if (stock.price_earnings is not None) and (stock.price_book is not None) else 0.0 weight = stock.portfolio_weight writer.write( '%-4d %s %-35s %-25s %25s %25.2f %25.3f %25s\n' % (i + 1, colored("%-6s" % stock.symbol, 'green'), name, industry, price_earnings, price_book, value, ("{:.0%}".format(weight)))) limit -= 1 if limit == 0: break return writer.getvalue() def print_positions(positions): writer = io.StringIO() writer.write("\nCURRENT POSITIONS\n") buf = ('%-6s %-25s %-25s %25s\n' % ('Symbol', 'Position', 'Currency', 'Sec Type')) writer.write(buf) writer.write('-' * len(buf)) writer.write("\n") for symbol in positions: position = positions[symbol] if position.position != 0: s = position.symbol if position.symbol is not None else "N/A" p = position.position c = position.currency if position.currency is not None else "N/A" t = position.sec_type if position.sec_type is not None else "N/A" writer.write('%-6s %-25s %-25s %25s\n' % (s, p, c, t)) return writer.getvalue() def print_order_book(order_book): writer = io.StringIO() writer.write("\nORDER BOOK\n") buf = ('%-25s %-25s %-25s %-65s\n' % ('Type', 'Symbol', 'Amount', 'Description')) writer.write(buf) writer.write('-' * len(buf)) writer.write("\n") for order in order_book: t = order.type if order.type is not None else "N/A" s = order.symbol if order.symbol is not None else "N/A" a = order.amount d = order.description if order.description is not None else "N/A" color = 'green' if t == 'EXIT': color = 'red' elif t == 'ADJUSTMENT': color = 'yellow' writer.write('%-25s %-25s %-25s %-65s\n' % ((colored("%-25s" % t, color), s, a, d))) return writer.getvalue() def print_open_orders(orders): writer = io.StringIO() writer.write("\nOPEN_ORDERS\n") buf = ('%-25s %-25s %-25s %-25s\n' % ('ID', 'Action', 'Quantity', 'Type')) writer.write(buf) writer.write('-' * len(buf)) writer.write("\n") for order_id in orders: order = orders[order_id] a = order.action q = order.totalQuantity t = order.orderType writer.write('%-25s %-25s %-25s %-25s\n' % (order_id, a, q, t)) return writer.getvalue() def main(): setLoggerClass(MyLogger) global log log = logging.getLogger(__name__) log.setLevel(DEBUG_LEVELV_NUM) ch = logging.StreamHandler() fh = logging.handlers.RotatingFileHandler('strategy.log', maxBytes=20000, backupCount=5) formatter = logging.Formatter('[%(asctime)s]: %(message)s') fh.setFormatter(formatter) ch.setFormatter(formatter) log.addHandler(fh) log.addHandler(ch) log.propagate = False app = Strategy(ipaddress="localhost", portid=1234, clientid=0) log.debugv("Server version=%s Connection time=%s." % (app.serverVersion(), app.twsConnectionTime())) if not app.twsConnectionTime(): log.debugv("Could not connect to TWS. Exiting. Bye!") exit() fundamentals = Fundamentals('database.ini') y = year() q = last_quarter() log.debugv("Fetching stocks from database for %s/%s." % (y, q)) stocks = fundamentals.get_stock_profiles(y, q) log.debugv("Fetched %s stocks from database." % (len(stocks))) for stock in stocks: stock.price = app.get_close_price(stock.symbol) log.debugv("Fetched prices for all symbols.") stocks = filter_stock_profiles(stocks) if (len(stocks) >= TARGET_NUMBER_OF_POSITIONS): stocks = rank_stocks(stocks) longs = stocks[:MAX_LONG_POSITION_SIZE] shorts = stocks[-MAX_SHORT_POSITION_SIZE:] # Rebalance portfolio_value = app.get_portfolio_value() amount = portfolio_value // TARGET_NUMBER_OF_POSITIONS # don't overweight position sizes if the pickings are slim my_positions = app.get_positions() amount_available_for_long = (portfolio_value // 2) number_of_long_units = (MAX_LONG_POSITION_SIZE * (MAX_LONG_POSITION_SIZE + 1)) / 2 amount_per_long_unit = amount_available_for_long // number_of_long_units log.debugv("Portfolio value=%s, amount_available_for_long=%s, number_of_long_units=%s, amount_per_long_unit=%s." % (portfolio_value, amount_available_for_long, number_of_long_units, amount_per_long_unit)) for rank, stock in enumerate(longs): # To diminish market impact, I invest less in stocks with very low liquidity. # if data.can_trade(stock): dollar_amount = (MAX_LONG_POSITION_SIZE - rank) * amount_per_long_unit amount = dollar_amount // stock.price stock.portfolio_weight = (amount * stock.price) / portfolio_value stock.order_amount = amount amount_available_for_short = (portfolio_value // 2) number_of_short_units = (MAX_SHORT_POSITION_SIZE * (MAX_SHORT_POSITION_SIZE + 1)) / 2 amount_per_short_unit = amount_available_for_short // number_of_short_units log.debugv("Portfolio value=%s, amount_available_for_short=%s, number_of_short_units=%s, amount_per_short_unit=%s." % (portfolio_value, amount_available_for_short, number_of_short_units, amount_per_short_unit)) for rank, stock in enumerate(shorts): # To diminish market impact, I invest less in stocks with very low liquidity. # if data.can_trade(stock): dollar_amount = (rank + 1) * amount_per_short_unit amount = dollar_amount // stock.price stock.portfolio_weight = (amount * stock.price) / portfolio_value stock.order_amount = -1 * amount log.debugv(print_stocks(longs, "NEW PORTFOLIO - LONGS", MAX_LONG_POSITION_SIZE)) log.debugv(print_stocks(shorts, "NEW PORTFOLIO - SHORTS", MAX_SHORT_POSITION_SIZE)) if any(stock.order_amount == 0 for stock in longs): log.error("Error: some long stocks have zero order amount. Exiting.") app.disconnect() exit() if any(stock.order_amount == 0 for stock in shorts): log.error("Error: some short stocks have zero order amount. Exiting.") app.disconnect() exit() log.debugv(print_positions(my_positions)) order_book = [] # Exit positions that are not ranking any more for symbol in my_positions: position = my_positions[symbol] if position.position > 0 and not any(stock.symbol == position.symbol for stock in longs): order = MyOrder(position.symbol) order.description = "Closing long position." order.type = "EXIT" order.amount = -1 * position.position order.currency = position.currency order_book.append(order) for symbol in my_positions: position = my_positions[symbol] if position.position < 0 and not any(stock.symbol == position.symbol for stock in shorts): order = MyOrder(position.symbol) order.description = "Closing short position." order.type = "EXIT" order.amount = -1 * position.position order.currency = position.currency order_book.append(order) log.debugv(print_order_book(order_book)) if len(order_book) > 0: confirm = input("Do you want to place the above exit orders (yes/no)? ") if confirm == 'yes': open_orders = app.get_open_orders() if open_orders: log.debugv("There are pending orders in the system.") log.debugv(print_open_orders(open_orders)) log.debugv("Not executing. Bye!") app.disconnect() exit() else: for order in order_book: if order.currency is not None: app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency) else: app.order_target_value(symbol=order.symbol, amount=order.amount) else: log.debugv("Decline execution accepted. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Portfolio seems up-to-date. Not executing. Bye!") app.disconnect() exit() open_orders = app.get_open_orders() log.debugv("Waiting for any pending orders to be filled.") while open_orders: sleep(0.1) order_book = [] log.debugv("No remaining pending orders.") # Buy/Sell/Adjust for stock in longs: if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0): order = MyOrder(stock.symbol) order.description = "New long position {0} x {1}".format(stock.order_amount, stock.symbol) order.type = "NEW" order.amount = stock.order_amount order_book.append(order) else: position = my_positions[stock.symbol] target_position = stock.order_amount if position.position < 0: stock.order_amount = target_position + abs(position.position) elif position.position > 0: if position.position != target_position: stock.order_amount = target_position - position.position else: stock.order_amount = 0 if stock.order_amount != 0: order = MyOrder(stock.symbol) order.description = "Adjusting existing long position for {0} from {1} -> {2}".format( stock.symbol, my_positions[stock.symbol].position, target_position) order.type = "ADJUSTMENT" order.amount = stock.order_amount order_book.append(order) for stock in shorts: if (not stock.symbol in my_positions) or (my_positions[stock.symbol].position == 0): order = MyOrder(stock.symbol) order.description = "New short position {0} x {1}".format(stock.order_amount, stock.symbol) order.type = "NEW" order.amount = stock.order_amount order_book.append(order) else: position = my_positions[stock.symbol] target_position = stock.order_amount if position.position > 0: stock.order_amount = target_position - position.position elif position.position < 0: if position.position != target_position: stock.order_amount = target_position - position.position else: stock.order_amount = 0 if stock.order_amount != 0: order = MyOrder(stock.symbol) order.description = "Adjusting existing short position for {0} from {1} -> {2}".format( stock.symbol, my_positions[stock.symbol].position, target_position) order.type = "ADJUSTMENT" order.amount = stock.order_amount order_book.append(order) log.debugv(print_order_book(order_book)) if len(order_book) > 0: confirm = input("Do you want to place the above orders (yes/no)? ") if confirm == 'yes': open_orders = app.get_open_orders() if open_orders: log.debugv("There are pending orders in the system.") log.debugv(print_open_orders(open_orders)) log.debugv("Not executing. Bye!") app.disconnect() exit() else: for order in order_book: if order.currency is not None: app.order_target_value(symbol=order.symbol, amount=order.amount, currency=order.currency) else: app.order_target_value(symbol=order.symbol, amount=order.amount) open_orders = app.get_open_orders() log.debugv("Waiting for any pending orders to be filled.") while open_orders: sleep(0.1) log.debugv("No remaining pending orders. Exiting. Bye!") app.disconnect() exit() else: log.debugv("Decline execution accepted. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Portfolio seems up-to-date. Not executing. Bye!") app.disconnect() exit() else: log.debugv("Filter resulted in less than %d stocks." % (TARGET_NUMBER_OF_POSITIONS)) if __name__ == "__main__": main() |
Disclaimer: this is a very basic algorithm to illustrate how to interact with the IB API using Python in order to build a fully automated trading algorithm. Please do not invest any real money with this algorithm. Algorithmic trading involves a substantial risk of loss and is not appropriate for everyone. No representation is being made that utilizing the algorithmic trading strategy will result in profitable trading or be free of risk of loss.
