Why do I get smaller accuracy when I use 80% of training sets using HMAX model?

Question

I am trying to compute the accuracy of the HMAX model. I am using the Face category (containing 435 images) from the Caltech101 database. I split it into $x$ training and $y$ testing. At each time, when $x$ increases, the accuracy also increases. Furthermore, I heard that the number of training should be equal to 80% by comparing it to the tests. So when I split my data into 348 positive training and the rest for positive testing, I got an accuracy that it is smaller than the other smaller splits (when $x<348)$!

By the way, I also used the Background category and I split it into 50 negative training and 50 negative testing.

Why do I get smaller accuracy?

Answer 1

I got the answer for this question:

Because the dataset becomes imbalanced. That is why the performance of our classifier decreases instead to decrease. Usually you should have a balance between out positive and negative training.