Software system for investment risk forecasting using Big Data and artificial intelligence

Abstract

The article is devoted to the development, detailed architectural design, and comprehensive experimental study of a high-performance software system for automated investment risk forecasting in the stock market using Big Data analytics technologies and modern artificial intelligence models. The high relevance of automating analytical processes is substantiated, driven by the extreme dynamism of stock price quotes, substantial volumes of unstructured news streams and social media content, and significant human physical limitations in manually monitoring them in real time. Software requirements specifications have been formulated in accordance with the international standard IEEE Std 830-1998. Based on two independent mathematical effort estimation models – the Constructive Cost Model and Use Case Points – an optimal development timeframe for a minimum viable product is substantiated, ranging from 5 to 6 months of pure individual development. A modular system architecture based on modern design patterns is proposed. An asynchronous automated pipeline for harvesting raw information arrays from stock exchange platforms (Nasdaq, Finnhub API) and the Reddit social network has been developed. Storage of relational structured data and semantic vectors is implemented using the PostgreSQL database management system with the PGVector extension and the optimized columnar Apache Arrow Parquet format. An analytical core has been established, featuring text vectorization using natural language processing methods based on the jinaai_v3 model suite (with an 8,192-token context window) and two predictive neural network models: a conventional stochastic classifier with a logistic loss function and a recurrent neural network with memory using a sliding window. A web user interface has been deployed based on the asynchronous FastAPI framework, HTMX reactive technology, Jinja2 template engine, and Chart.js visualization library, enabling asynchronous dashboard updates without full page reloads. Automated unit testing confirmed dataset integrity and connection pool stability. Experimental validation of artificial intelligence models confirmed a significant advantage of the deep recurrent neural network with memory over the baseline classifier: the values of key validation metrics for model comparison demonstrated substantial changes when evaluating the performance shift between the models. Operational testing demonstrated a reduction in financial asset analysis time from several hours down to a few minutes, as well as full compliance of the system with regulated non-functional requirements for performance, reliability, and user convenience under real market conditions.