1 of 16

Evaluating and Comparing Machine Learning Algorithms in WEKA

Model Evaluation and Statistical Significance Testing

dr. Jamolbek Mattiev

2 of 16

Objective of the Exercise

  • Evaluate multiple machine learning algorithms
  • Compare performance on different datasets
  • Apply statistical significance testing
  • Make evidence-based model selection

3 of 16

Datasets in WEKA

  • Load multiple datasets (ARFF or CSV)
  • Different domains: medical, text, numerical
  • Ensure correct class attribute is set
  • Apply same preprocessing for fair comparison

4 of 16

Algorithms for Comparison

  • Decision Trees (J48)
  • Naive Bayes
  • Support Vector Machines (SMO)
  • Random Forest
  • k-Nearest Neighbors (IBk)

5 of 16

Evaluation Methods

  • 10-fold Cross-Validation (recommended)
  • Percentage Split (e.g., 70/30)
  • Use Training Set (not recommended for comparison)

6 of 16

Performance Metrics

  • Accuracy
  • Precision, Recall, F1-score
  • ROC Area (AUC)
  • Confusion Matrix

7 of 16

Running Experiments in WEKA

  • Explorer → Classify tab
  • Select classifier and evaluation method
  • Run experiments on each dataset
  • Record results for each algorithm

8 of 16

Why Statistical Significance Testing?

  • Differences in accuracy may be due to chance
  • Statistical tests assess real performance differences
  • Essential for scientific comparison

9 of 16

WEKA Experimenter

  • Open WEKA → Experimenter
  • Allows automated algorithm comparison
  • Supports multiple datasets and runs
  • Stores results for statistical analysis

10 of 16

Setting Up an Experiment

  • Add datasets
  • Select classifiers
  • Choose evaluation method (cross-validation)
  • Set number of runs (e.g., 10)

11 of 16

Statistical Tests in WEKA

  • Paired t-test (default)
  • Corrected resampled t-test
  • Non-parametric tests (via plugins)

12 of 16

Result Analysis

  • Use 'Analyse' tab in Experimenter
  • Select reference algorithm
  • Significant wins/losses highlighted
  • Confidence level usually 0.05

13 of 16

Interpreting Results

  • W = statistically significant win
  • L = statistically significant loss
  • No mark = no significant difference
  • Prefer simpler models if performance is similar

14 of 16

Exercise Workflow

  • Load datasets
  • Run classifiers using Experimenter
  • Analyze results with paired t-test
  • Identify best-performing algorithm

15 of 16

Best Practices

  • Use same preprocessing for all models
  • Use multiple datasets
  • Report both performance and significance
  • Avoid overfitting

16 of 16

Conclusion

  • WEKA supports systematic model comparison
  • Statistical testing improves reliability
  • Essential for research and publications