Time series with rich payloads (TSRP) are sequences of irregularly sampled, high-dimensional observations in which temporal alignment is informative but sample sizes are often limited. Such data arise in numerous domains, yet existing classification methods are ill-suited for these challenges. This setting falls outside the assumptions of most existing time-series classifiers, which rely on dense numerical observations or large training samples. We present a framework for classifying TSRP that integrates representation learning with sparse Dynamic Time Warping (DTW) and support vector machines (SVMs). High-dimensional payloads are first embedded into lower-dimensional representations, after which sparse DTW aligns sequences for SVM-based classification. Our approach is evaluated on six datasets derived from social media, capturing medically relevant transitions such as shifts in mental health status. Results demonstrate that the proposed framework substantially outperforms a baseline based of average pooling across five of six datasets. We further analyze performance relative to dataset characteristics, provide insights into model behavior, and introduce an interpretable mechanism linking temporal alignments to input features. Overall, this work establishes a general, interpretable, and effective approach for classifying sparse time series with rich payloads, providing a foundation for applications in healthcare and other high-stakes domains.