Detecting brown spot in rice leaf is an urgent complication in the agricultural field as Brown Spot disease lessen the rice yield remarkably. Several segmentation techniques have been applied to identify and extract the infected portion of the rice-leaf and machine learning algorithms such as decision trees, support vector machines are applied to detect this infection. In particular, a combination of Convolution Neural Networks with these algorithms has also tried to resolve this problem. Although this attempt has achieved success in providing accuracy (96.8%), these kinds of approaches raise issues regarding the size and interpretability of feature space and interpretability of the decision model. Indeed, Deep learning networks automatically create a feature space that usually contains a massive number of features (numerous of them are not necessarily appropriate). This vast number of features extends the non-interpretability of the machine learning model. Furthermore, training the model with these many features is computationally expensive. To resolve these issues, we propose a method to extract a few interpretable features from rice-leaf images and construct a low-dimensional feature space; however, interpretation shows that they deserve significant credit for the decent accuracy of our classification model.