With clean data in hand, we turned our focus to modeling. But we didn’t want to rely on a single one. Instead, we used a comparative approach, testing six different machine learning classifiers; each with distinct strengths, weaknesses.
We began with Logistic Regression, our baseline model. It was fast, interpretable, and surprisingly competitive: achieving a training accuracy of 66.74% and a testing accuracy of 65.91%. Its simplicity made it a great diagnostic tool for spotting linearly separable relationships.
Next came the Decision Tree Classifier, which learned nonlinear patterns by recursively splitting features. It scored slightly better in training (68.95%) but dipped in testing (64.59%), a sign of mild overfitting. To address this, we tried the Random Forest Classifier, an ensemble of decision trees that averaged their outputs to reduce variance. This model achieved the highest training accuracy (79.33%) but saw only modest gains in testing (65.64%).
We also tested the k-Nearest Neighbors (k-NN) algorithm. While intuitive and simple, it struggled with the randomness in our data—overfitting to training samples (75.28%) but generalizing poorly (62.21%).
Interestingly, Gaussian Naive Bayes became the strongest model in our project. It assumed feature independence (which was not strictly true), but its probabilistic nature and low complexity gave it consistent performance: 67.09% training accuracy and 66.11% testing accuracy.
Finally, the Support Vector Machine (SVM) showed disappointing results, peaking at 64.88% training and 60.77% testing. Its sensitivity to hyperparameters like C and gamma, coupled with the high dimensionality of our feature space, likely hindered its performance.
In summary, every model taught us something about the data, about our assumptions, and about the tradeoff between complexity and generalization.
| Model |
Training Accuracy |
Testing Accuracy |
| Logistic Regression |
66.74% |
65.91% |
| Decision Tree Classifier |
68.95% |
64.59% |
| Random Forest Classifier |
79.33% |
65.64% |
| K-Neighbors Classifier |
75.28% |
62.21% |
| Gaussian Naive-Bayes |
67.09% |
66.11% |
| Support Vector Machine |
64.88% |
60.77% |
Table 1: Training and testing accuracy of each machine learning model.
While our modeling pipeline was sound, the results left room for improvement. Our top performing model (Gaussian Naive Bayes) topped out at 66.11% testing accuracy; respectable, but far from industry grade. Understanding this plateau became a project of its own.
One issue was data separability. Even after scaling, the difference between “healthy” and “faulty” classes was often subtle. Many samples had overlapping sensor profiles, suggesting either labeling inconsistencies or intrinsic ambiguity in real world engine faults. This made it hard for any classifier to draw clean boundaries.
Another challenge was feature dependency. Most models assumed feature independence or simple additive relationships, which may not capture the true causal mechanisms of engine behavior. Sequential models like LSTMs might perform better in future iterations by modeling temporal patterns.
In conclusion, the dataset itself was likely a limiting factor. Without access to richer sensor modalities (e.g., vibration, audio, or temporal sequences), our models were predicting health from snapshots rather than trends. This is a tough task, even for neural nets.
