10-Times Random Split Method in WEKA

• Course: Databases and Data Mining
• Practical Session
• Instructor: Jamolbek Mattiev

Learning Objectives

• • Understand repeated holdout validation
• • Perform 10 random splits in WEKA
• • Calculate mean and standard deviation
• • Compare with 10-fold cross-validation

What is Random Split?

• • Dataset divided into training and test sets
• • Example: 70% train / 30% test
• • Depends on random seed
• • Results may vary each run

What is 10-Times Random Split?

• • Repeat percentage split 10 times
• • Use different random seeds
• • Compute average performance
• • Reduces randomness effect

Software: WEKA Explorer

• • Open WEKA
• • Go to Explorer
• • Load dataset (.arff)
• • Choose Classify tab

Step 1: Choose Classifier

• • Click Choose
• • Select trees → J48
• • Ensure class attribute is correct

Step 2: Set Percentage Split

• • Select Percentage split
• • Set 70% training
• • Set random seed = 1
• • Click Start

Step 3: Repeat 10 Times

• • Change seed from 1 to 10
• • Run experiment each time
• • Record accuracy results
• • Fill results table

Recording Results

• Run | Seed | Accuracy (%)
• 1 | 1 |
• 2 | 2 |
• ...
• 10 | 10 |

Mean Accuracy Formula

• x̄ = (1/10) Σ xi
• Average of 10 runs
• Represents overall performance

Standard Deviation Formula

• σ = sqrt((1/10) Σ (xi − x̄)²)
• Measures stability
• Lower value = more stable model

Interpretation

• • Large variation → unstable model
• • Small variation → stable model
• • Small datasets → higher variance

Comparison with 10-Fold CV

• • 10-Fold uses all samples for testing
• • More robust estimation
• • Higher computational cost
• • Usually lower variance

Advantages & Limitations

• Advantages:
• • Simple
• • Easy to implement
• Limitations:
• • Still random
• • Not as robust as K-fold

Discussion Questions

• • Why do results change with seed?
• • Which method is more reliable?
• • When should we use repeated holdout?

Example Accuracy Variation (10 Random Splits)