A major challenge in AI projects, such as Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and copilot implementations, is the availability of data. Companies often spend up to 80% of the AI project time on tasks such as inputting, cleaning, curating, semantically tagging, and converting (indexing and vectorizing) data. Data Stream addresses these challenges by automatically integrating data pipelines into the underlying AI architecture that connects storage and GPUs directly.
Data Stream is an integral part of the Pure Storage Data Platform and is optimized for enterprise inference use cases based on the NVIDIA AI Data Platform reference design. Some key technical capabilities of Data Stream include:
- Automated, real-time data ingestion and structuring: Data Stream processes raw data from various sources, including text, PDFs, images, and structured tables, intelligently segmenting and transforming it to maintain context and provide precise access control. The solution supports multiprotocol access (NFS, S3, SMB), can handle billions of files and objects, and can be seamlessly integrated with vector databases on Pure Storage FlashBlade//S.
- Seamless NVIDIA NeMo integration: Data Stream orchestrates end-to-end workflows with NVIDIA NeMo Retriever. This allows organizations to rapidly convert raw data into meaningful digital representations (vectors), enabling AI systems to better understand context and relationships. These vectors enable advanced meaning-based searches, allowing AI systems in RAG pipelines to quickly and accurately retrieve the most relevant information. The integration with NVIDIA NIM enables optimized inference and seamless scalability in on-premise and cloud environments, via standardized APIs.
- Optimized pipelines: by leveraging the computing power of the NVIDIA RTX PRO 6000 Blackwell Server Edition GPU and various software libraries, such as NVIDIA Spark Rapids and NVIDIA cuVS, organizations can use GPU-optimized pipelines for synchronized and efficient data processing. This architecture utilizes NVIDIA ConnectX-7 NICs for networked storage access (central data storage accessible via the network to multiple systems) with low latency. Combined with FlashBlade//S, this synchronization prevents compute bottlenecks in the RTX PRO server and provides performance improvements in vector ingestion.
- Transformation and enrichment at the storage layer: Data Stream processes data enrichment directly on FlashBlade DirectFlash Modules, using NVRAM for fast metadata management. This reduces the need for data movement and improves efficiency. The output is stored in formats such as JSON, Apache Parquet, or Arrow, enabling scalable vector storage and RAG datasets at petabyte scale.
"AI requires a data platform that can convert vast amounts of unstructured information into real-time insights. Pure Storage Data Stream leverages the reference design of the NVIDIA AI Data Platform to boost AI reasoning and agents with an AI-ready storage infrastructure with computing, networking, and AI software that is fully accelerated by NVIDIA," says Justin Boitano, Vice President of Enterprise AI at NVIDIA.
