kdb+ by Kx Systems

kdb+ is a columnar time-series database developed by Kx Systems, designed from the ground up for speed on ordered, timestamped data. It stores data in column-major format, meaning queries that aggregate or filter across a single field read only the relevant column from disk or memory rather than scanning full rows. This architecture delivers query performance that general-purpose relational databases and most NoSQL stores cannot match for time-series workloads at scale.

The database ships with q, a terse array-based language that handles data manipulation, query execution, and inter-process communication in a unified environment. q is a steep learning curve but a high return: experienced q developers can express in a single line what would require dozens of lines in SQL or Python, and execute it against hundreds of millions of rows in milliseconds.

kdb+ is the dominant database technology in capital markets. Most large investment banks and financial services firms with real-time trading infrastructure run kdb+ as their tick database, historical data store, or both.

q is both a query language and a general-purpose programming language. It handles table operations, mathematical aggregations, string manipulation, file I/O, and socket communication using the same concise syntax, which means a kdb+ system can ingest data, store it, query it, and distribute results to downstream consumers without requiring any additional runtime.

The language is array-oriented: most operations apply implicitly to entire columns or lists rather than requiring explicit iteration. This eliminates the overhead of row-by-row processing that affects interpreted languages like Python when working at scale. For FinTech engineering teams building real-time analytics pipelines, this means kdb+ can process millions of ticks per second on commodity hardware with single-digit millisecond query response times.

q's terse syntax makes it appear inaccessible at first, but Kx Systems maintains extensive documentation and a structured learning path at code.kx.com that takes practitioners from basics to production patterns systematically.

Tick database (tickerplant): The canonical kdb+ architecture captures market data from exchanges and data vendors in real time, writes it to an intraday in-memory table, and flushes to a compressed on-disk historical database at end of day. The entire tick-to-history pipeline is a standard kdb+ pattern with well-established reference implementations.

Real-time risk calculations: Trading desks run position and P&L calculations against live market data stored in kdb+, querying intraday tick data and joining it against position tables to produce real-time risk views with sub-second latency.

Backtesting infrastructure: Quantitative researchers query years of historical tick data stored in kdb+ to test trading strategies at the tick level, exploiting the database's columnar architecture for fast retrieval of specific symbols, time windows, and price fields.

Surveillance and compliance: Market surveillance systems ingest full order book data into kdb+ and run pattern-detection queries to flag manipulative trading behaviour, a requirement for regulated trading venues. Teams building this infrastructure as part of a broader financial data platform commonly anchor the time-series layer on kdb+ even when other components use different technologies.

kdb+ processes run as single-threaded event loops by default, with parallelism achieved by running multiple q processes and routing queries across them via a load balancer or a gateway process. This architecture keeps the concurrency model simple and predictable, which is important when operating systems where latency spikes have direct commercial consequences.

Data is stored in a binary columnar format on disk using memory-mapped files. Queries against historical data read directly from mapped memory rather than copying data into a buffer, which eliminates a significant source of latency on large reads. Compression is applied at the column level, reducing storage requirements without adding decompression overhead on typical queries.

kdb+ is available under a commercial license from Kx Systems for production use. A free 32-bit non-commercial version is available for personal use and learning. Kx also makes educational resources, sample code, and the kdb+ Cookbook available through code.kx.com, and the Kx Community forums provide practitioner support for common patterns and troubleshooting.

For organisations evaluating kdb+ as part of a financial data platform build, the licensing cost needs to be assessed against the engineering time saved by using a purpose-built time-series store rather than adapting a general-purpose database to the same workload. In most capital markets contexts, the performance and operational maturity of kdb+ justify the commercial license. For guidance on building the surrounding infrastructure, starting a conversation with an engineering team experienced in financial services data platforms can accelerate the evaluation.