$ tesseract -v
tesseract 4.0.0-beta.3
 leptonica-1.76.0
  libjpeg 9c : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11
 Found AVX512BW
 Found AVX512F
 Found AVX2
 Found AVX
 Found SSE

pip install pillow
pip install pytesseract
pip install imutils

python
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import pytesseract
>>> import imutils
>>>

tesseract --help-oem
OCR Engine modes:
  0    Legacy engine only.
  1    Neural nets LSTM engine only.
  2    Legacy + LSTM engines.
  3    Default, based on what is available.

tesseract --help-psm
Page segmentation modes:
  0    Orientation and script detection (OSD) only.
  1    Automatic page segmentation with OSD.
  2    Automatic page segmentation, but no OSD, or OCR.
  3    Fully automatic page segmentation, but no OSD. (Default)
  4    Assume a single column of text of variable sizes.
  5    Assume a single uniform block of vertically aligned text.
  6    Assume a single uniform block of text.
  7    Treat the image as a single text line.
  8    Treat the image as a single word.
  9    Treat the image as a single word in a circle.
 10    Treat the image as a single character.
 11    Sparse text. Find as much text as possible in no particular order.
 12    Sparse text with OSD.
 13    Raw line. Treat the image as a single text line,
       bypassing hacks that are Tesseract-specific.

tree --dirsfirst
.
├── images
│   ├── example_01.jpg
│   ├── example_02.jpg
│   ├── example_03.jpg
│   ├── example_04.jpg
│   └── example_05.jpg
├── frozen_east_text_detection.pb
└── text_recognition.py
 
1 directory, 7 files

# import the necessary packages
from imutils.object_detection import non_max_suppression
import numpy as np
import pytesseract
import argparse
import cv2

def decode_predictions(scores, geometry):
  # grab the number of rows and columns from the scores volume, then
  # initialize our set of bounding box rectangles and corresponding
  # confidence scores
  (numRows, numCols) = scores.shape[2:4]
  rects = []
  confidences = []
 
  # loop over the number of rows
  for y in range(0, numRows):
    # extract the scores (probabilities), followed by the
    # geometrical data used to derive potential bounding box
    # coordinates that surround text
    scoresData = scores[0, 0, y]
    xData0 = geometry[0, 0, y]
    xData1 = geometry[0, 1, y]
    xData2 = geometry[0, 2, y]
    xData3 = geometry[0, 3, y]
    anglesData = geometry[0, 4, y]
 
    # loop over the number of columns
    for x in range(0, numCols):
      # if our score does not have sufficient probability,
      # ignore it
      if scoresData[x] < args["min_confidence"]:
        continue
 
      # compute the offset factor as our resulting feature
      # maps will be 4x smaller than the input image
      (offsetX, offsetY) = (x * 4.0, y * 4.0)
 
      # extract the rotation angle for the prediction and
      # then compute the sin and cosine
      angle = anglesData[x]
      cos = np.cos(angle)
      sin = np.sin(angle)
 
      # use the geometry volume to derive the width and height
      # of the bounding box
      h = xData0[x] + xData2[x]
      w = xData1[x] + xData3[x]
 
      # compute both the starting and ending (x, y)-coordinates
      # for the text prediction bounding box
      endX = int(offsetX + (cos * xData1[x]) + (sin * xData2[x]))
      endY = int(offsetY - (sin * xData1[x]) + (cos * xData2[x]))
      startX = int(endX - w)
      startY = int(endY - h)
 
      # add the bounding box coordinates and probability score
      # to our respective lists
      rects.append((startX, startY, endX, endY))
      confidences.append(scoresData[x])
 
  # return a tuple of the bounding boxes and associated confidences
  return (rects, confidences)

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", type=str,
  help="path to input image")
ap.add_argument("-east", "--east", type=str,
  help="path to input EAST text detector")
ap.add_argument("-c", "--min-confidence", type=float, default=0.5,
  help="minimum probability required to inspect a region")
ap.add_argument("-w", "--width", type=int, default=320,
  help="nearest multiple of 32 for resized width")
ap.add_argument("-e", "--height", type=int, default=320,
  help="nearest multiple of 32 for resized height")
ap.add_argument("-p", "--padding", type=float, default=0.0,
  help="amount of padding to add to each border of ROI")
args = vars(ap.parse_args())

# load the input image and grab the image dimensions
image = cv2.imread(args["image"])
orig = image.copy()
(origH, origW) = image.shape[:2]
 
# set the new width and height and then determine the ratio in change
# for both the width and height
(newW, newH) = (args["width"], args["height"])
rW = origW / float(newW)
rH = origH / float(newH)
 
# resize the image and grab the new image dimensions
image = cv2.resize(image, (newW, newH))
(H, W) = image.shape[:2]

# define the two output layer names for the EAST detector model that
# we are interested in -- the first is the output probabilities and the
# second can be used to derive the bounding box coordinates of text
layerNames = [
  "feature_fusion/Conv_7/Sigmoid",
  "feature_fusion/concat_3"]
 
# load the pre-trained EAST text detector
print("[INFO] loading EAST text detector...")
net = cv2.dnn.readNet(args["east"])

Our two output layer names are put into list form on Lines 99-101. To learn why these two output names are important, you’ll want to refer to my original EAST text detection tutorial.

Then, our pre-trained EAST neural network is loaded into memory (Line 105).

I cannot emphasize this enough: you need OpenCV 3.4.2 at a minimum to have the  cv2.dnn.readNet  implementation.

The first bit of “magic” occurs next:

# construct a blob from the image and then perform a forward pass of
# the model to obtain the two output layer sets
blob = cv2.dnn.blobFromImage(image, 1.0, (W, H),
  (123.68, 116.78, 103.94), swapRB=True, crop=False)
net.setInput(blob)
(scores, geometry) = net.forward(layerNames)
 
# decode the predictions, then  apply non-maxima suppression to
# suppress weak, overlapping bounding boxes
(rects, confidences) = decode_predictions(scores, geometry)
boxes = non_max_suppression(np.array(rects), probs=confidences)

To determine text locations we:

• Construct a blob  on Lines 109 and 110. Read more about the process here.
• Pass the blob  through the neural network, obtaining scores  and geometry  (Lines 111 and 112).
• Decode the predictions with the previously defined decode_predictions  function (Line 116).
• Apply non-maxima suppression via my imutils method (Line 117). NMS effectively takes the most likely text regions, eliminating other overlapping regions.

Now that we know where the text regions are, we need to take steps to recognize the text! We begin to loop over the bounding boxes and process the results, preparing the stage for actual text recognition:

# initialize the list of results
results = []
 
# loop over the bounding boxes
for (startX, startY, endX, endY) in boxes:
  # scale the bounding box coordinates based on the respective
  # ratios
  startX = int(startX * rW)
  startY = int(startY * rH)
  endX = int(endX * rW)
  endY = int(endY * rH)
 
  # in order to obtain a better OCR of the text we can potentially
  # apply a bit of padding surrounding the bounding box -- here we
  # are computing the deltas in both the x and y directions
  dX = int((endX - startX) * args["padding"])
  dY = int((endY - startY) * args["padding"])
 
  # apply padding to each side of the bounding box, respectively
  startX = max(0, startX - dX)
  startY = max(0, startY - dY)
  endX = min(origW, endX + (dX * 2))
  endY = min(origH, endY + (dY * 2))
 
  # extract the actual padded ROI
  roi = orig[startY:endY, startX:endX]

We initialize the results  list to contain our OCR bounding boxes and text on Line 120.

Then we begin looping over the boxes  (Line 123) where we:

• Scale the bounding boxes based on the previously computed ratios (Lines 126-129).
• Pad the bounding boxes (Lines 134-141).
• And finally, extract the padded roi  (Line 144).

Our OpenCV OCR pipeline can be completed by using a bit of Tesseract v4 “magic”:

# in order to apply Tesseract v4 to OCR text we must supply
# (1) a language, (2) an OEM flag of 4, indicating that the we
# wish to use the LSTM neural net model for OCR, and finally
# (3) an OEM value, in this case, 7 which implies that we are
# treating the ROI as a single line of text
config = ("-l eng --oem 1 --psm 7")
text = pytesseract.image_to_string(roi, config=config)
 
# add the bounding box coordinates and OCR'd text to the list
# of results
results.append(((startX, startY, endX, endY), text))

Taking note of the comment in the code block, we set our Tesseract config  parameters on Line 151 (English language, LSTM neural network, and single-line of text).

Note: You may need to configure the —psm  value using my instructions at the top of this tutorial if you find yourself obtaining incorrect OCR results.

The pytesseract  library takes care of the rest on Line 152 where we call pytesseract.image_to_string , passing our roi  and config string .

? Boom! In two lines of code, you have used Tesseract v4 to recognize a text ROI in an image. Just remember, there is a lot happening under the hood.

Our result (the bounding box values and actual text  string) are appended to the results  list (Line 156).

Then we continue this process for other ROIs at the top of the loop.

Now let’s display/print the results to see if it actually worked:

# sort the results bounding box coordinates from top to bottom
results = sorted(results, key=lambda r:r[0][1])
 
# loop over the results
for ((startX, startY, endX, endY), text) in results:
  # display the text OCR'd by Tesseract
  print("OCR TEXT")
  print("========")
  print("{}\n".format(text))
 
  # strip out non-ASCII text so we can draw the text on the image
  # using OpenCV, then draw the text and a bounding box surrounding
  # the text region of the input image
  text = "".join([c if ord(c) < 128 else "" for c in text]).strip()
  output = orig.copy()
  cv2.rectangle(output, (startX, startY), (endX, endY),
    (0, 0, 255), 2)
  cv2.putText(output, text, (startX, startY - 20),
    cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 0, 255), 3)
 
  # show the output image
  cv2.imshow("Text Detection", output)
  cv2.waitKey(0)

Our results are sorted  from top to bottom on Line 159 based on the y-coordinate of the bounding box (though you may wish to sort them differently).

From there, looping over the results , we:

• Print the OCR’d text  to the terminal (Lines 164-166).
• Strip out non-ASCII characters from text  as OpenCV does not support non-ASCII characters in the   cv2.putText  function (Line 171).
• Draw (1) a bounding box surrounding the ROI and (2) the result text  above the ROI (Lines 173-176).
• Display the output and wait for any key to be pressed (Lines 179 and 180).

OpenCV text recognition results

Now that we’ve implemented our OpenCV OCR pipeline, let’s see it in action.

Be sure to use the “Downloads” section of this blog post to download the source code, OpenCV EAST text detector model, and the example images.

From there, open up a command line, navigate to where you downloaded + extracted the zip, and execute the following command:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_01.jpg
[INFO] loading EAST text detector...
OCR TEXT
========
OH OK

We’re starting with a simple example.

Notice how our OpenCV OCR system was able to correctly (1) detect the text in the image and then (2) recognize the text as well.

The next example is more representative of text we would see in a real- world image:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_02.jpg
[INFO] loading EAST text detector...
OCR TEXT
========
® MIDDLEBOROUGH

Again, notice how our OpenCV OCR pipeline was able to correctly localize and recognize the text; however, in our terminal output we see a registered trademark Unicode symbol — Tesseract was likely confused here as the bounding box reported by OpenCV’s EAST text detector bled into the grassy shrubs/plants behind the sign.

Let’s look at another OpenCV OCR and text recognition example:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_03.jpg
[INFO] loading EAST text detector...
OCR TEXT
========
ESTATE
 
OCR TEXT
========
AGENTS
 
OCR TEXT
========
SAXONS

In this case, there are three separate text regions.

OpenCV’s text detector is able to localize each of them — we then apply OCR to correctly recognize each text region as well.

Our next example shows the importance of adding padding in certain circumstances:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_04.jpg 
[INFO] loading EAST text detector...
OCR TEXT
========
CAPTITO
 
OCR TEXT
========
SHOP
 
OCR TEXT
========
|.

In the first attempt of OCR’ing this bake shop storefront, we see that “SHOP” is correctly OCR’d, but:

1. The “U” in “CAPUTO” is incorrectly recognized as “TI”.
2. The apostrophe and “S” is missing from “CAPUTO’S’.
3. And finally, “BAKE” is incorrectly recognized as a vertical bar/pipe (“|”) with a period (“.”).

By adding a bit of padding we can expand the bounding box coordinates of the ROI and correctly recognize the text:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_04.jpg --padding 0.05
[INFO] loading EAST text detector...
OCR TEXT
========
CAPUTO'S
 
OCR TEXT
========
SHOP
 
OCR TEXT
========
BAKE

Just by adding 5% of padding surrounding each corner of the bounding box we’re not only able to correctly OCR the “BAKE” text but we’re also able to recognize the “U” and “’S” in “CAPUTO’S”.

Of course, there are examples where OpenCV flat out fails:

python text_recognition.py --east frozen_east_text_detection.pb \
  --image images/example_05.jpg --padding 0.25
[INFO] loading EAST text detector...
OCR TEXT
========
Designer
 
OCR TEXT
========
a

I increased the padding to 25% to accommodate the angle/perspective of the words in this sign. This allowed for “Designer” to be properly OCR’d with EAST and Tesseract v4. But the smaller words are a lost cause likely due to the similar color of the letters to the background.

In these situations there’s not much we can do, but I would suggest referring to the limitations and drawbacks section below for suggestions on how to improve your OpenCV text recognition pipeline when confronted with incorrect OCR results.

Limitations and Drawbacks

It’s important to understand that no OCR system is perfect!

There is no such thing as a perfect OCR engine, especially in real-world conditions.

And furthermore, expecting 100% accurate Optical Character Recognition is simply unrealistic.

As we found out, our OpenCV OCR system worked in well in some images, it failed in others.

There are two primary reasons we will see our text recognition pipeline fail:

1. The text is skewed/rotated.
2. The font of the text itself is not similar to what the Tesseract model was trained on.

Even though Tesseract v4 is significantly more powerful and accurate than Tesseract v3, the deep learning model is still limited by the data it was trained on — if your text contains embellished fonts or fonts that Tesseract was not trained on, it’s unlikely that Tesseract will be able to OCR the text.

Secondly, keep in mind that Tesseract still assumes that your input image/ROI has been relatively cleaned.

Since we are performing text detection in natural scene images, this assumption does not always hold.

In general, you will find that our OpenCV OCR pipeline works best on text that is (1) captured at a 90-degree angle (i.e., top-down, birds-eye-view) of the image and (2) relatively easy to segment from the background.

If this is not the case, you may be able to apply a perspective transform to correct the view, but keep in mind that the Python + EAST text detector reviewed today does not provide rotated bounding boxes (as discussed in my previous post), so you will still likely be a bit limited.

Tesseract will always work best with clean, preprocessed images, so keep that in mind whenever you are building an OpenCV OCR pipeline.

If you have a need for higher accuracy and your system will have an internet connection, I suggest you try one of the “big 3” computer vision API services:

• Google Vision API OCR Engine
• Amazon Rekognition
• Microsoft Cognitive Services

…each of which uses even more advanced OCR approaches running on powerful machines in the cloud.

Summary

In today’s tutorial you learned how to apply OpenCV OCR to perform both:

1. Text detection
2. Text recognition

To accomplish this task we:

1. Utilized OpenCV’s EAST text detector, enabling us to apply deep learning to localize regions of text in an image
2. From there, we extracted each of the text ROIs and then applied text recognition using OpenCV and Tesseract v4.

We also looked at Python code to perform both text detection and text recognition in a single script.

Our OpenCV OCR pipeline worked well in some cases but also failed in others. For the best OpenCV text recognition results I would suggest you ensure:

1. Your input ROIs are cleaned and preprocessed as much as possible. In an ideal world your text would be perfectly segmented from the rest of the image, but in reality, that won’t always be possible.
2. Your text have been captured at a 90 degree angle from the camera, similar to a top-down, birds-eye-view. In the case this is not the case, a perspective transform may help you obtain better results.

I hope you enjoyed today’s blog post on OpenCV OCR and text recognition!

To be notified when future blog posts are published here on PyImageSearch (including text recognition tutorials), be sure to enter your email address in the form below!

https://www.pyimagesearch.com/2018/09/17/opencv-ocr-and-text-recognition-with-tesseract/