Investment banks, being data-rich organisations, have long had the infrastructure and processes to deal with large volumes of data. However, the volume, velocity and variety of data flowing through banks, in 2016, challenges traditional data management systems, which are limited by inelastic processing power and complex data structures. Big Data technology is increasingly adopted by capital markets’ firms to address these concerns, but before launching into or, indeed, dismissing Big Data technology, business managers must be certain of the benefits to be gleaned from it.
Fundamentally, Big Data has characteristics that make it unsuitable for storage, processing and analysis on a traditional data platform; it could be rapidly streamed, of poor quality or be data that is presented in a variety of formats, such as text and images – or all of the above. Big Data technology is designed to handle such data. That a dataset is large does not, in itself, identify it as ‘Big Data’. Big Data platforms can ingest, store and process structured data – tabulated and formulated – and more problematic unstructured data – for example, textual data. The technology is fault tolerant and can incorporate data that may otherwise be rejected. Additionally, Big Data analysis includes the whole data universe, rather than just a sample as in more traditional platforms, and the analysis is frequently designed to look for correlations and is less fixated on determining causality.
To fully understand the capabilities that Big Data technology can deliver, a business manager and the technical team need to build a bridge of understanding. A step towards achieving this is for the business to grasp the basic concepts of the technology, so that managers can understand what Big Data technology can offer.
