Pattern Recognition

A Single Feature Bayesian Classifier

Bayesian decision theory is a fundamental statistical approach to the problem of pattern classification. Bayesian formula can be expressed in English by saying that

posterior = likelihood * prior / evidence

Notice that it is the product of the likelihood and the prior probability that is most important in determining the posterior probability. The evidence factor can be viewed as merely a scale factor that guarantees that the posterior probabilities sum to one, as all good probabilities must.

We consider x to be a continuous random variable whose distribution depends on the state of nature and is expressed as p(x|ω). This is the class-conditional probability density function, the probability density function for x given that the state of nature is ω.

Suppose that we know the prior probabilities P(ωj) – P denotes a probability mass function – and the conditional densities p(x|ωj) ∀ j=1, 2, 3, …, c where ∀ means for all and c is the number of states of nature ( categories ). We note the (joint) probability density of finding a pattern that is in category ωj and has feature value x can be written:

p(ωj, x) = P(ωj|x)p(x) = p(x|ωj)P(ωj)

Rearranging these leads us to Bayes formula:

P(ωj|x) = p(x|ωj)P(ωj) / p(x)

where in the case of c categories

p(x) = ∑p(x|ωj)P(ωj) ∀ j=1, 2, 3, …, c

We would naturally be inclined to decide that the true state of nature is ωk, if we have an observation x for which P(ωk|x) is the maximum P(ωj|x) ∀ j=1, 2, 3, …, c.

Bayes Decision Rule : Decide ωk if P(ωk|x) = max { P(ωj|x) ∀ j=1, 2, 3, …, c }

Now that we’ve laid down the basic mathematical theory, let’s build our first Bayesian Classifier. The example implemented here is copied from my university Pattern Recognition class assignment. We suppose that all likelihood probabilities follow a normal (Gaussian) distribution characterized by its mean (μ) and standard deviation (σ) given as inputs for each state of nature (category).

Here is the function for opening the image file:

        public Stream open_file_dialog(Form parent_form, TextBox text_box_file_name)
            OpenFileDialog file_dialog = new OpenFileDialog();
            file_dialog.Filter = "JPEG Files (*.jpeg)|*.jpeg|PNG Files (*.png)|*.png|JPG Files (*.jpg)|*.jpg|GIF Files (*.gif)|*.gif|BMP Files (*.bmp)|*.bmp";
            file_dialog.FilterIndex = 1;
            DialogResult clicked_result = file_dialog.ShowDialog(parent_form);
            if (clicked_result == DialogResult.OK)
                update_text_box_with_file_path(text_box_file_name, file_dialog.FileName);
                return file_dialog.OpenFile();
                return null;

Then, the file stream is loaded into a bitmap object as follows:

        public Bitmap load_image_file(Stream image_file_stream)
            Image obj_img = Image.FromStream(image_file_stream);
            return new Bitmap(obj_img);

The goal is to classify each pixel of a digital image into one of the states of nature ( categories ). The categories are identified by mouse clicks over an input image file, as each clicked pixel value characterizes the mean (μ) of a specific class ωj.

Here is the function for capturing the mouse clicks:

        public void mouse_clicks_handler(MouseEventArgs e)
            switch (comboBox_input_image_source.SelectedIndex)
                case 0:
                    int x = e.X;
                    int y = e.Y;
                    Color c = mvc.fill_gridview_from_mouse_clicks(dataGridView_inputs, x, y, pictureBox_classified);
                    MessageBox.Show(String.Format("Click at Point := ({0}, {1}) \n\n Color (R,G,B) := {2}, {3}, {4}", x, y, c.R, c.G, c.B));
                    tabControl_task_2.SelectedTab = tabPage_outputs;
                case 1:
                    MessageBox.Show("If you want to use this option, please select file first!");

Then, it fills the DataGridView with pixel value at the clicked mouse location:

        public Color fill_gridview_from_mouse_clicks(DataGridView dgrv, int x, int y, PictureBox pictureBox_classified)
            if (null == grey_image || x < 0 || x >= grey_image.Width || y < 0 || y >= grey_image.Height)
                return Color.FromArgb(0);
            Color c = grey_image.GetPixel(x, y);
            dgrv.Rows.Add(c.R, 0.5, c.G, 0.5, c.B, 0.5, 1);
            propagata_class_regions_array(dgrv, grey_image.Width, grey_image.Height);
            pictureBox_classified.Image = apply_bayesian_inference(grey_image, dgrv);
            return c;

In this blog post, a single feature x is observed, hence the pixel value is converted to its grey scale equivalent by averaging its RGB components:

x = (PixelValue.Red + PixelValue.Green + PixelValue.Blue) / 3

In a future blog post, a multi-feature Bayesian classifier should be implemented.

Here is the function for getting the grey scale image:

        public Bitmap get_grey_scale(Bitmap bmp)
            if (bmp == null)
                return null;
            Bitmap ret = new Bitmap(bmp.Width, bmp.Height);
            for (int x = 0; x < bmp.Width; x++ )
                for (int y = 0; y < bmp.Height; y++)
                    Color original_color = bmp.GetPixel(x, y);
                    int grey_level = (original_color.R + original_color.G + original_color.B) / 3;
                    ret.SetPixel(x, y, Color.FromArgb(grey_level, grey_level, grey_level));
            return ret;

Finally, let’s apply the Bayesian inference to each pixel of the grey scale image as follows:

        public Bitmap apply_bayesian_inference(Bitmap grey_image, DataGridView dgrv_meus_sigmas)
            Bitmap ret = new Bitmap(grey_image.Width, grey_image.Height);
            if (array_class_regions == null)
                return ret;
            for (int x = 0; x < grey_image.Width; x++)
                for (int y = 0; y < grey_image.Height; y++)
                    Color original_color = grey_image.GetPixel(x, y);
                    double averaged_color_value = (original_color.R + original_color.G + original_color.B) / 3;
                    int class_index = new BayesianInferenceEngine().classify(array_class_regions, averaged_color_value);
                    ret.SetPixel(x, y, array_class_regions[class_index].color);
            return ret;

The classification itself is based on Bayes Decision Rule mentioned above and is implemented simply as follows:

            public int classify(ClassRegion[] classes, double x){
                if (classes == null)
                    return -1;
                double meu, sigma;
                composition_object_normal_distribution = new NormalDistribution();
                double maximum_likelihood = double.NegativeInfinity;
                int class_index = -1;
                for (int i = 0; i < classes.Length; i++) { 
                    if (classes[i] == null)
                    meu = classes[i].get_averaged_meu();
                    sigma = classes[i].get_averaged_sigma();
                    double likelihood = composition_object_normal_distribution.my_normal_function(x , meu, sigma);
                    double prior = classes[i].prior;
                    if (likelihood * prior > maximum_likelihood)
                        class_index = i;
                        maximum_likelihood = likelihood * prior;
                return class_index;

For the sake of completeness, let’s include the function to calculate the normal (Gaussian) distribution using the formula:

p(x|μ,σ) = (1 / σ√2π) e^-((x-μ)² / 2σ² )

        public double my_normal_function(double x, double meu, double sigma)
            return (1 / (Math.Sqrt(2 * Math.PI) * sigma)) * Math.Exp(-1 * Math.Pow(x - meu, 2) / (2 * Math.Pow(sigma, 2)));

As in the screen-shot, our simple single feature Bayesian classifier successfully classified the regions numbered and painted them with the same color.

bayesian inference single featurebayesian inference single feature


Special thanks to my teaching assistant for diligently helping me debugging the code for more than two hours.



2 thoughts on “A Single Feature Bayesian Classifier

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s