Q. The problem I'm having is that ninjatrader lags pretty bad when there is significant movement in the markets. By lag I meant the NT dom lags (can't move or place orders, price doesn't update for 2-3 seconds) as well as NT charts not updating promptly. Bookmap seems to be running fine with no lag while this is happening.
If I remove the bookmap indicators from the three charts in ninjatrader then the lag no longer happens.
Please make sure that both Bookmap and NT are set with equal priority as shown below. It’s highly recommended to set both with Normal priority. Based on your description it’s likely that Bookmap has higher priority than NT. Bookmap itself never sets or changes those settings. The default priority is Normal, set by Windows OS. However it happened in the past that Bookmap users change this settings manually and then forgot about it. If this is not the case and you still have this problem, please contact us and our tech support will help.
Q. I installed bookmap on a new ssd drive separate from NT and the OS to see if it helps but the problem still occurs.
Indeed, if the problem is as suspected above, this wouldn’t help.
Q. t is set to "Normal" priority. Ninjatrader is also set to Normal, I never changed these values. I am running Windows 10 btw. I still experience the lag. Again, this lag is usually apparent under heavier market activity. At this time I'm only running CL and ES on bookmap.
Hi. Thank you for clarifications. You will see bookmap.exe instead of java.exe with new version Bookmap 7. It's available on the website. Although it has some nice features (see here: https://bookmap.com/release-notes/#7_0_10), it doesn't guarantee solving the problem that you described.
The next candidate for the problem source is your Antivirus (specifically host intrusion prevention system, sandbox) that may cause the lag. Try to disable your antivirus, and then completely reinstall Bookmap.
If this doesn't help, update NT7 to NT8. NT8 seems to run indicators in a separate thread, not the UI (user interface) thread. So, even if communication between NT and Bookmap isn't fast enough to handle spikes in data stream, the UI of NT8 will not freeze. This would be at least a workaround.
In general, please note that NT doesn't have a standard API exposed to other processes. It's only intended to indicators (NT scripts), running inside NT process. But because Bookmap is itself a standalone program, it cannot be rewritten as NT indicator. Instead, when you add a Bookmap indicator to NT, this indicator communicates with Bookmap through Windows interprocess communication protocol. While it's pretty fast and we've done a lot to optimize it, it's not as fast as direct connections as with the rest of platforms and API's that Bookmap supports. Also, this interprocess communication can be blocked or slowed down by Antiviruses as false positive detection. So, please try the above steps and we'll decide how to proceed based on the results. You can also write us and request remote desktop support, just mention this post as a reference.
Q. I did notice that the bookmap\cache folder grows to about 2.5 GB by end of day. Not sure this is normal.
Yes, this is normal. Market data typically consists of 90-97% of market depth updates (order book) and the rest is trades data. This is why observing market depth is so important. But that also implies technological challenges. Note that most trading charts don’t need to handle market depth data and even the trades data is aggregated into candlesticks. Even if those charts display DoM, they need to handle only the current snapshot of market depth.
Bookmap chart may contain millions of market data events. The technological challenges are to allow smooth real-time chart display like HD video experience and also to allow quick and seamless chart manipulation. This required fast access to the market data collected so far. Handling all this data in RAM is not an option due to the large amounts of the data which can reach many gigabytes before compression. To solve this we have developed a proprietary database and a data structure that effectively use both RAM and disk drive. The Bookmap\Cache folder contains the latter. It can easily grow to several gigabytes and it’s normal. The proprietary data structure is optimized for fastest access with both SSD and HDD, but SSD is obviously faster.
Q. I installed Bookmap on a new ssd drive...
There are two main folders which Bookmap creates during installation:
Installation folder. By default: C:\Program Files\Bookmap\
User folder. By default: C:\Bookmap
From the performance point of view it doesn't really matter on which drive you have the Installation folder. But if you have enough space on your SSD drive, it’s recommended to put the User folder on it because this is where market data is maintained by Bookmap and SSD is faster than HDD.
Q. Tell me about graphic/video cards and GPU
GPU’s are Graphics Processing Unit - the core of graphic/video cards. Unlike CPU that are optimized for computations, GPU’s are optimized for quick and smooth rendering on display, thus reducing the load on CPU’s and RAM. Bookmap utilizes modern GPU technologies using OpenGL 3.0 interface which is available since 2008, so most computers are supporting it. Users with older graphic cards can disable it via Settings->GPU acceleration. Another feature is Settings->Enable smoothing. This allows smooth transition of chart display on sub-pixel level. It’s mostly noticeable on low resolution monitors.
Today many computers have both integrated and dedicated (typically more powerful) graphic cards. You can can let Bookmap use the more powerful graphic card as following. Right click on your desktop, and open NVIDIA Control Panel. Select “Manage 3D settings”, find Bookmap in the list of programs, and define which graphic card it should use. See example below:
Q. Tell me about memory/RAM consumption of Bookmap
See the question above about Bookmap/Cache folder. Bookmap maintains large amounts of market data. While most of the data is located in Bookmap/Cache folder, some part of it is managed in RAM for even quicker access. A typical memory consumption that appears in Windows’ Task Manager is 1.5 GB. That doesn’t mean that all this memory is being used by Bookmap, but this is how much is allocated for Bookmap. Technically Bookmap uses less memory, but reducing the allocation limit would make Java Garbage Collector (Bookmap is a Java program) to optimize memory size more frequently, thus slightly interrupting smooth operation of the program. Bookmap 7 (released on 16-Jan-2018) can automatically set RAM allocation based on computer hardware. Users can adjust the settings manually via Settings->Memory allocation:
Q. Tell me about CPU load
Typically Bookmap concumes 2-10% of CPU depending on computer hardware.
It’s highly recommended to keep hardware drivers updated or to make sure they are configured for automatic updates. This is especially important for newer hardware. It’s also important to keep Windows OS up to date. For instance recently Desktop Window Manager (dwm.exe) was causing a significant memory leak, but then it was fixed with one of Windows updates.
CPU consumption is expected to be higher during
- Chart manipulations such as shifting, zooming, adding indicators or changing their settings, etc.
- Starting from Bookmap 7 -- loading historical market depth data after subscription to an instrument.
- Loading data in Replay mode (see progress bar in Replay controls panel).
If you experience higher CPU load by Bookmap in other situations, please contact our support for assistance.
Additional factors that can increase CPU load:
- Not using GPU acceleration or old and low memory graphic card.
- Very high resolution displays
- Number of Bookmap charts detached for a separate view
- Number of subscribed instruments / tabs.
- High speed Replay mode
- Increased chart refresh rate. Set larger refresh period (means lower FPS - frames per second) via Settings->Chart refresh rate.
- Zooming out. The more price levels are displayed on the chart and the longer period - the more computations are required.
- Number of indicators displayed on the chart. Currently Correlation Tracker takes a lot of resources when aligned to the left edge of the chart - this will be fixed soon.