Early crop yield prediction is crucial in smart agriculture for making administrative plans to post-harvest management and distribution of a crop. Crop yield prediction is crucial for global food security yet challenging due to multitudinous factors that jointly determine the yield, including genotype, environment, management, and their complex interactions. The proposed model is a multi-crop yield prediction. Initially, data was collected from CropNet dataset for Contextual data and image based on crop yield prediction. The data Pre-process will be done using atmospheric correction, cloud masking, and temporal harmonization. Contextual data preprocessing will be done using z-score normalization, missing value imputation. Following that, feature extraction model for spatial feature extraction for multi-scale dilated Convolutional Flatten Spatial attention based GhostNet (MDC-FSG) and temporal feature extraction for Progressive Temporal Convolutional Relative Transformer Network model (PTC-RTN) are performed. After that, the feature fusion model for coordinate Convo block attention-guided feature fusion module (CCA-GFF) integrated with the contextual feature with satellite derived representation. Furthermore, the optimization algorithm namely Cellular Automata based Successive Optimizer (CA-SO) is employed to optimize network hyper parameters and fusion weights in both extractors. Finally, a regression head based deep network (RHDN) predicts crop yield that provides better accurate results. Extensive experiments have been carried out on CropNet and attained better performance enhancement including, MSE of 0.11743, MAE of 0.3263, RMSE of 0.3426, RAE of 0.0009 and Theil_U analysis for 0.00033 of precision on the CropNet dataset, and it outperformed all other current MMA frameworks.