There are eight roads of a route in Dalat city (Vietnam) used to collect lifelog data for our experiment. The length of the route is about 6.6 km. Some different sensors are used to record both environmental images and numerical data. The AQI factor of numerical data, i.e., PM2.5, and images are recorded along the route in the meantime. We invested 33 days for data acquisition with both visual and numerical data from Oct. 15 to Nov. 17, 2019. In addition, we also used 43.824 collected PM2.5 public data days from 2010 to 2014 years for the single LSTM training stage.

The lifelog images were processed to extract spatial features, while PM2.5 data was included as an additional input feature to study the association relationship between environmental visual features and AQI levels. CNN is an efficient deep learning method to detect and extract features of visual big data (i.e., images)[5]. Recently, there have been some CNN-based recognition algorithms. Deep Residual Networks (ResNet) [6] was proposed by Microsoft Research as the best model for ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [7] in 2015, and it became possible to construct deep neural networks of up to 1000 CNN layers or more. Each layer is considered to extract more sophisticated and complex features. Place-CNN[8] fine-tuned ResNet for scene classification. Attention-CNN [9] also adopted the ResNet as a CNN seed for image captioning and visual question answering. In our experiment, Attention-CNN is the best feature extractor of the above-mentioned methods that is utilized to extract feature vectors of our lifelog visual data. Then, the combined one-dimensional vectors of extracted features are used as the input data for the proposed CNN-LSTM model training to predict PM2.5 concentrations.

The traditional LSTM model[10] was trained to capture temporal dependencies in the PM2.5 data. The LSTM model was designed to enable effective prediction of PM2.5 concentrations over a specific period. In our experimental data, PM2.5 values of each route recorded within the same period are strongly different for various days. The PM2.5 concentrations of each route can be fairly predicted using the traditional LSTM model with untrained data of the remaining days.

The proposed deep learning model combined the CNN Extraction model in Step 2 and the traditional LSTM model in Step 3 above to recognize some abnormal changes of PM2.5 data (e.g., traffic congestion, garbage, …etc.). Different from the traditional pure LSTM architecture, the first stage of the proposed combined CNN-LSTM uses CNN-Extractor, and the latter stage is LSTM forecasting, which is used to analyze the features extracted by CNN-Extractor and then to estimate the PM2.5 concentration. The training process involved optimizing the model’s parameters to minimize the prediction error. The training was performed until the model reached a satisfactory level of accuracy.

In our combined CNN-LSTM model, the loss function is “categorical_crossentropy” and the optimizer is SGD (Stochastic Gradient Descent) with a learning rate of 0.01.

In our experiment, a feature extractor called CNN-Extractor[9], which adopted the ResNet as a CNN seed for image captioning and visual question answering, is the best feature extractor of the aforementioned methods that is utilized to extract feature vectors of environmental images. The output of the last fully connected layer of the CNN-Extractor is collected as the input feature vector for the combined CNN-LSTM model training stage.

The training data of Road 1 are selected randomly with five cases of training data ratios containing 65%, 70%, 75%, 80%, and 85%. The results of data training using the proposed CNN-LSTM model are achieved in terms of mean squared error (MSE) of 5.416, 5.336, 4.927, 4.826, and 4.683, respectively. The decrease in the MSE of the evaluation stage with more trained data shows that the proposed model is probably stable and robust

1 n 2

enough to cope with a wide range of different visual data. MSE is formulated as MSE = ∑i=1(oi − pi) .

where oi is the ith original value, pi is the corresponding predicted value for oi, and n is the number of observations. The Σ indicates that a summation is performed over all values of i.

Table 1 lists experimental results with the same training data ratio of 85% for both traditional LSTM and combined CNN-LSTM models and shows clear evidence of better forecast accuracy of the proposed combined method compared to the original LSTM model. MSE for lifelog data is much smaller with the proposed combined method than with the traditional LSTM.

As we know, the LSTM model outperformed traditional statistical models commonly used for time-series prediction. However, the results demonstrated in Fig. 2a show that the traditional LSTM model captured only temporal information and was not accurate in some abnormal changes in PM2.5 concentration (e.g., traffic congestion, garbage, …etc.).

As illustrated in Fig. 2b, the proposed combined CNN-LSTM model which used both spatial environmental images and temporal PM2.5 information achieved superior PM2.5 concentration prediction accuracy compared to the traditional LSTM approach. In the proposed model, the CNN component effectively extracted spatial features from environmental images, capturing the spatial features associated with PM2.5 concentrations. Then, the LSTM component captured the temporal dependencies and extracted spatial features in the data, allowing accurate predictions over time, especially for abnormal PM2.5 cases. Finally, the experimental results show the effectiveness of the proposed combined CNN-LSTM approach for PM2.5 prediction in Vietnam.