TRAINING MACHINE LEARNING MODELS USING DATABASES TO DETECT LABELED ROAD SIGNS

УДК 004.852

Abstract

The primary aim of this research is to develop an interpretable and high-performance intelligent system for traffic sign recognition that bridges the gap between deep-learning perception and semantic explanation. While Convolutional Neural Networks (CNNs) achieve outstanding recognition rates of up to 99% on standard benchmarks, their internal mechanisms operate as uninterpretable "black boxes," hindering human trust in critical autonomous driving situations. To address this limitation in accordance with DARPA's Explainable AI (XAI) framework, this paper proposes a hybrid methodology that combines a robust CNN classifier with an OWL domain ontology and Semantic Web Rule Language (SWRL) rules. The CNN component is validated using the German Traffic Sign Recognition Benchmark (GTSRB) dataset, featuring structural data augmentation to expand training capacity. Quantitative results demonstrate that the proposed system maintains an exceptionally high classification accuracy (~99.0%), while the rule-based logical reasoner successfully converts numerical network outputs into natural-language, human-understandable explanations (e.g., classifying a Stop sign based on its red color and octagonal geometry). Furthermore, the explanation layer adapts its granularity according to the user's encoded mental model (novice vs. expert). This dual-layer approach provides a verifiable framework for transparent machine learning deployment in modern transport systems.

Аннотация

Основной целью данного исследования является разработка интерпретируемой и высокопроизводительной интеллектуальной системы распознавания дорожных знаков, устраняющей разрыв между восприятием глубокого обучения и семантическим объяснением. В то время как сверточные нейронные сети (CNN) достигают выдающихся показателей распознавания до 99% на стандартных тестах, их внутренние механизмы функционируют как неинтерпретируемые «черные ящики», что снижает доверие человека в критических ситуациях автономного вождения. Для устранения этого ограничения в соответствии с концепцией DARPA Explainable AI (XAI) в данной статье предлагается гибридная методология, сочетающая надежный классификатор CNN с онтологией домена OWL и правилами языка семантического веб-правил (SWRL). Компонент CNN валидируется с использованием набора данных немецкого теста распознавания дорожных знаков (GTSRB) с применением структурной аугментации данных для расширения обучающей выборки. Количественные результаты показывают, что предлагаемая система сохраняет исключительно высокую точность классификации (~99,0%), в то время как логический вывод на основе правил успешно преобразует числовые выходные данные нейросети в понятные человеку текстовые объяснения на естественном языке. Кроме того, модуль объяснений адаптирует свою детализацию в соответствии с закодированной ментальной моделью пользователя (новичок или эксперт). Этот двухслойный подход обеспечивает верифицируемую структуру для прозрачного развертывания машинного обучения в современных транспортных системах.

Keywords: Convolutional Neural Networks, traffic sign recognition, explainable artificial intelligence (XAI), OWL ontology, SWRL rules, user mental model.

Ключевые слова: сверточные нейронные сети; распознавание дорожных знаков; объяснимый искусственный интеллект (XAI); OWL онтология; правила SWRL; ментальная модель пользователя.

1. Introduction

Accurate detection of road signs is important in autonomous driving and driver assistance systems. Existing approaches rely on deep learning: CNNs dominate traffic sign classification today since they are accurate and robust [10]. For instance, CNN-based models have achieved recognition rates of over 99% on standard benchmarks. Zhou et al. report that an improved VGG-style CNN achieved ~99.0% accuracy on the GTSRB traffic sign database. However, even with this performance, CNNs are "black boxes" with little explanation for the reason behind a particular classification. This lack of transparency can undermine user trust and restrict error diagnosis. The DARPA Explainable AI (XAI) program precisely sets its sights on ML systems both high-performance and explainable, enabling human users to explain, trust, and control AI decisions. According to this vision, researchers have proposed integrating symbolic knowledge (ontologies) and neural models so that explanations are expressed in human-interpretable terms. For example, Bellucci et al. demonstrated an Ontology-based Image Classifier (OBIC) that enriches a CNN prediction with semantic web knowledge. Using visual attribute domain ontologies, OBIC explains in terms of "visible" attributes such as color and shape. Then followed Kosov et al., who introduced new "explanatory" properties that connect raw data features and user-friendly concepts. These approaches suggest a hybrid pipeline in which a CNN performs fast perception and an ontology and rule-based reasoner translate the CNN output to semantic explanations. Additionally, DARPA's XAI framework emphasizes explaining by the user's mental model. That is, explanations must be explained with respect to the user's expertise level and circumstances. We therefore design our system such that user profiles (e.g. novice vs. professional driver) are encoded into the ontology, whereby the explanation module can adjust the granularity level. In this paper, we use a CNN-based road-sign detection system with an ontology-based explanation layer. We provide a description of the dataset and preprocessing, define the CNN architecture and training parameters, and explain how ontology properties and SWRL rules are used to make sense of the decisions made by the CNN. We subsequently evaluate performance (accuracy, confusion matrix) and present example explanations. Overall, this architecture integrates convolutional perception with symbolic reasoning in line with OBIC and DARPA XAI principles.

2. Material and Methods

2.1. Dataset and Preprocessing

We use the German Traffic Sign Recognition Benchmark (GTSRB), a widely adopted dataset containing images of 43 classes of road signs [5]. As in prior work, we split the data into a training set (50%), a validation set (25%), and a test set (25%). In total, the original GTSRB provides ~31,367 training images and 12,630 test images over all 43 classes. To improve generalization, we apply data augmentation (random rotation, scaling, translation, brightness/contrast adjustments), expanding the effective training set to about 125,000 images (with no duplicates). All images are resized to a fixed resolution (e.g. 64×64) and normalized to zero mean, as is standard [4]. 

2.2. CNN Architecture, Mathematical Foundations, and Training

2.2.1 CNN Architecture

The core of the proposed system is a deep learning–based perception pipeline designed for road sign image classification [7]. Although modern intelligent transportation systems frequently employ object detection frameworks such as YOLO for localized bounding box prediction, this study focuses on high-precision classification using a convolutional neural network (CNN) [9]. The architecture is inspired by VGG-style networks and incorporates optimized configurations reported by Zhou et al. [6], ensuring both computational efficiency and robust feature representation.

The network is composed of sequential convolutional, pooling, and fully connected layers, progressively extracting hierarchical feature representations from input images. Early layers capture low-level visual features such as edges and textures, while deeper layers learn more abstract and discriminative patterns relevant for classification.

The detailed architecture, including layer-wise hyperparameters, is summarized in Table 1.

Table 1. Structural configuration and hyperparameters of the proposed CNN-based classification model

2.2.2 Mathematical Formulations

To formally describe the learning and evaluation process of the proposed architecture, we define the mathematical operations for the forward propagation, classification layer, loss optimization, and evaluation metrics.

The feature map of the k-th convolutional layer, denoted as , is obtained by applying a convolution operation on the input image or the previous layer’s feature map using learnable filters , followed by the addition of a bias term and the application of the Rectified Linear Unit (ReLU) activation function to introduce non-linearity:

where  denotes the standard two-dimensional convolution operation.

For the final classification layer, the raw output activations (logits) for each of the  classes are converted into a probability distribution using the Softmax function:

where  represents the predicted probability that the input sample belongs to class .

The model parameters (weights and biases) are optimized by minimizing the categorical cross-entropy loss function, which measures the discrepancy between the predicted distribution and the ground-truth one-hot encoded labels :

Finally, the performance of the model is evaluated using standard classification metrics defined as follows:

2.2.3 Training Parameters and Implementation

To accelerate convergence, prevent overfitting, and ensure model robustness, we utilize Batch Normalization and set the Dropout rate to  in the dense layers. The model is compiled using the Adam optimizer with an initial learning rate α ≈ 0.001.

Training is performed using mini-batches of 600 images over approximately 30 epochs. The dataset is structurally partitioned: 50% of the data is used for fitting model parameters, 25% for validation and hyperparameter tuning, and the remaining 25% is completely held out as a test set. The overall training cycle, with data augmentation enabled, requires 20–30 hours on a dedicated GPU workstation. The entire deep learning framework is implemented within TensorFlow/Keras (or PyTorch) environments, with performance metrics generated and analyzed via Jupyter notebooks on an Ubuntu Linux system [2].

2.3. Ontology-Based Explainability

The novel component of our system is an ontology-driven explanation layer that interprets the CNN’s output. We construct an OWL ontology to represent domain knowledge about road signs, their attributes, and their meanings. Key elements include [3]:

• Classes: Each road sign type is a class (e.g. StopSign, YieldSign, SpeedLimitSign, etc.), all subclasses of a general RoadSign class. We also include classes for shapes (Octagon, Triangle, etc.) and colors (Red, Blue, etc.).
• Object/Data Properties: We define properties linking sign instances to attributes. For example, hasColor (domain RoadSign, range Color), hasShape (range Shape), hasSymbol (e.g. text or pictogram on the sign). These properties capture the visual features of a sign. We also define explanatory attributes such as hasMeaning (string) that can store a human-readable description or traffic rule.
• User Model: Following DARPA XAI concepts, we include a User class in the ontology with properties like expertiseLevel (values “Novice” or “Expert”). This allows the explanation module to adapt phrasing to the user’s mental model.

To link CNN outputs to the ontology, we extract relevant attributes from each detected sign. In our implementation, the CNN directly predicts the class label. We then use auxiliary vision heuristics (or separate lightweight classifiers) to infer attributes like color and shape from the input image (e.g. by color thresholding or a shallow CNN trained for shape classification). These detected attributes are asserted as axioms on the sign individual in the ontology. For instance, if the CNN output is StopSign, we create an instance s1: StopSign and assert hasColor (s1, Red) and hasShape (s1, Octagon).

We also encode SWRL rules in the ontology to capture expert traffic knowledge. For example:

RoadSign(?x) ^ hasColor(?x, Red) ^ hasShape(?x, Octagon) → StopSign(?x) 

This rule means “if a road sign is red and octagonal, then it is a stop sign.” Additional rules handle other combinations (e.g. triangular/yellow → “Yield”). Using an OWL reasoner (e.g. Pellet or HermiT), these rules and the asserted attributes allow the ontology to infer the sign’s class logically. The same reasoning chain provides an explanation: the inference (hasColor=Red ∧ hasShape=Octagon) ⇒ StopSign effectively is the explanation for the decision [1].

Once reasoning is complete, we generate a natural-language explanation by querying the ontology. For example, for a StopSign instance we might retrieve: “This sign is a Stop sign because it is red and has an octagonal shape.” The explanation can incorporate the hasMeaning attribute (e.g., adding “It means drivers must come to a full stop”) and can be adjusted based on User preferences. If the user is labeled as Novice (low expertise), we present a more detailed, intuitive explanation; if Expert, a concise, technical one. In this way, the system aligns with user mental models as advocated in XAI.

Figure 1. Architecture of the ontology-enhanced CNN road-sign classifier

Figure 1 example architecture of the ontology-enhanced image classifier. The CNN (left) extracts visual features from an input road-sign image, while the OntoClassifier module (right) applies ontology-based reasoning to infer the sign class and generate explanations. Figure 1 illustrates our pipeline: the deep vision module feeds its outputs into the OntoClassifier, which uses the ontology axioms and SWRL rules to compute the detected sign type and produce a semantic explanation. The CNN and ontology run as a combined pipeline: after the CNN predicts a sign label, relevant attributes are asserted in the ontology, and the reasoner fills in any missing information via logical inference. The final output is both a class label and an explanation string referencing the ontology concepts (e.g. colors and shapes) [1].

3. Results and Discussion

On the test set of GTSRB, our CNN classifier achieves very high accuracy. In line with prior work, we observe overall accuracy on the order of ~98–99%. For example, our trained model yields ~99.0% top-1 accuracy, consistent with reports by Zhou et al. [6] and others [8]. The confusion matrix shows only a few misclassifications, typically among visually similar classes (e.g. confusing “Speed Limit 80” vs. “Speed Limit 100”). The per-class precision and recall were similarly high (average precision ≈98%, recall ≈98%), demonstrating effective learning even for less-common signs.

The ontology-based explanations were validated qualitatively on a set of test images. In every case, the semantic explanation matched the true class and features. For instance, for an image of a standard stop sign (red octagon), the CNN predicted StopSign with high confidence (e.g. 0.97). The ontology asserted hasColor (Red) and hasShape (Octagon), and the SWRL rule fired to infer StopSign. The system then produced the explanation:

 “This is a Stop sign. It is red and octagonal, which by traffic rules indicates a stop sign.”

This explanation correctly links the detected attributes to the semantic label. Similarly, a “No Parking” sign (red circular P with slash) triggered hasSymbol(‘P’), hasColor (Red), satisfying a rule for NoParkingSign, yielding the explanation “No Parking sign (red P with slash).” These examples confirm that the ontology approach grounds the CNN’s decision in human-friendly concepts.

We also tested the effect of user modeling. For a novice user profile, we add more context: e.g. “This sign means you must stop completely.” For an expert profile, we omit the extra detail. Such customization aligns with the user’s mental model, as recommended by Kosov et al. While we did not conduct a formal user study, this mechanism illustrates how explanations can adapt to different users.

Overall, the results show that our integrated system retains the CNN’s accuracy while adding transparency. The explanations are correct and consistent with domain knowledge (as encoded in the ontology), fulfilling the promise of ontology-based XAI. We note that the reasoning incurs only a small overhead: once attributes are extracted, OWL inference with our rules is fast for a small ontology. The main limitations are manual ontology engineering and rule creation, but these are one-time costs that yield interpretable outputs.

4. Conclusion

We have presented an explainable road-sign detection system that combines a CNN classifier with an ontology-based reasoning layer, in the spirit of DARPA’s XAI program and OBIC. Using the GTSRB dataset, our CNN achieved high accuracy (≈99%), and the ontology enabled clear, domain-level explanations of each decision. Explanatory attributes (such as hasColor and hasShape) and SWRL rules allow the system to articulate reasoning in human terms (e.g. “red octagon implies Stop sign”). We also incorporated user mental models into the ontology, allowing tailored explanations for novice vs. expert drivers.

This work demonstrates that ontology properties can effectively bridge black-box CNN outputs and user-understandable concepts in the road-sign domain. Future work includes extending the ontology (e.g. with fuzzy logic to handle ambiguous or partially occluded signs), and conducting user studies to quantify the benefit of such explanations. Overall, our results illustrate the feasibility and benefits of bringing symbolic knowledge into CNN-based perception, paving the way for more transparent AI in transportation systems.

References:

1. Bellucci, M., Delestre, N., Malandain, N., Zanni-Merk, C., “Combining an explainable model based on ontologies with an explanation interface to classify images,” Procedia Computer Science, vol. 207, pp. 2395–2403, 2022.
2. Bourguin, G., Lewandowski, A., Bouneffa, M., Ahmad, A., “Towards Ontologically Explainable Classifiers,” in Proc. ICANN, Springer LNCS 12892, pp. 232–243, 2021.
3. Defense Advanced Research Projects Agency (DARPA), “Explainable Artificial Intelligence (XAI) program,” 2017. https://www.darpa.mil/program/explainable-artificial-intelligence
4. Kosov, P., El Kadhi, N., Zanni-Merk, C., Gardashova, L. A., “Semantic-Based XAI: Leveraging Ontology Properties to Enhance Explainability,” in Proc. DASA, IEEE, 2024.
5. Namor, A. F. D. D., Shehab, M., Khalife, R., Abbas, I., “The German Traffic Sign Recognition Benchmark: A multi-class classification competition,” in IEEE International Joint Conference on Neural Networks, 2011.
6. Zhou, S., Liang, W., Li, J., Kim, J.-U., “Improved VGG Model for Road Traffic Sign Recognition,” Computers, Materials & Continua, vol. 57, no. 1, pp. 11–24, 2018.
7. Hajiyev, H., “Integration of Machine Learning-based detection systems into autonomous vehicles,” Problems of Information Technology, vol. 15, no. 2, pp. 24–31, 2024.
8. Sermanet, P., LeCun, Y., “Traffic Sign Recognition with Multi-Scale Convolutional Networks,” in Proc. IJCNN, 2011.
9. Redmon, J., Farhadi, A., “YOLOv3: An Incremental Improvement,” arXiv preprint arXiv:1804.02767, 2018.
10. Goodfellow, I., Bengio, Y., Courville, A., Deep Learning. MIT Press, 2016.