| OpenCV 4.3.0 Open Source Computer Vision

Upscaling images: multi-output

In this tutorial you will learn how to use the 'dnn_superres' interface to upscale an image via a multi-output pre-trained neural network. OpenCVs dnn module supports accessing multiple nodes in one inference, if the names of the nodes are given. Currently there is one model included that is capable of giving more output in one inference run, that is the LapSRN model. LapSRN supports multiple outputs with one forward pass. It can now support 2x, 4x, 8x, and (2x, 4x) and (2x, 4x, 8x) super-resolution. The uploaded trained model files have the following output node names:

- 2x model: NCHW_output
- 4x model: NCHW_output_2x, NCHW_output_4x
- 8x model: NCHW_output_2x, NCHW_output_4x, NCHW_output_8x

## Building

When building OpenCV, run the following command to build all the contrib module:

cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules/

Or only build the dnn_superres module:

cmake -D OPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules/dnn_superres

Or make sure you check the dnn_superres module in the GUI version of CMake: cmake-gui.

## Source Code of the sample

Run the sample code with the following command

./bin/example_dnn_superres_dnn_superres_multioutput path/to/image.png 2,4 NCHW_output_2x,NCHW_output_4x \
path/to/opencv_contrib/modules/dnn_superres/models/LapSRN_x4.pb
1 // This file is part of OpenCV project.
2 // It is subject to the license terms in the LICENSE file found in the top-level directory
3 // of this distribution and at http://opencv.org/license.html.
4
5 #include <iostream>
6 #include <sstream>
7 #include <opencv2/dnn_superres.hpp>
8
9 #include <opencv2/imgproc.hpp>
10 #include <opencv2/highgui.hpp>
11
12 using namespace std;
13 using namespace cv;
14 using namespace dnn_superres;
15
16 int main(int argc, char *argv[])
17 {
18 // Check for valid command line arguments, print usage
19 // if insufficient arguments were given.
20 if (argc < 4) {
21 cout << "usage: Arg 1: image | Path to image" << endl;
22 cout << "\t Arg 2: scales in a format of 2,4,8\n";
23 cout << "\t Arg 3: output node names in a format of nchw_output_0,nchw_output_1\n";
24 cout << "\t Arg 4: path to model file \n";
25 return -1;
26 }
27
28 string img_path = string(argv[1]);
29 string scales_str = string(argv[2]);
30 string output_names_str = string(argv[3]);
31 std::string path = string(argv[4]);
32
33 //Parse the scaling factors
34 std::vector<int> scales;
35 char delim = ',';
36 {
37 std::stringstream ss(scales_str);
38 std::string token;
39 while (std::getline(ss, token, delim)) {
40 scales.push_back(atoi(token.c_str()));
41 }
42 }
43
44 //Parse the output node names
45 std::vector<String> node_names;
46 {
47 std::stringstream ss(output_names_str);
48 std::string token;
49 while (std::getline(ss, token, delim)) {
50 node_names.push_back(token);
51 }
52 }
53
54 // Load the image
55 Mat img = cv::imread(img_path);
56 Mat original_img(img);
57 if (img.empty())
58 {
59 std::cerr << "Couldn't load image: " << img << "\n";
60 return -2;
61 }
62
63 //Make dnn super resolution instance
64 DnnSuperResImpl sr;
65 int scale = *max_element(scales.begin(), scales.end());
66 std::vector<Mat> outputs;
67 sr.readModel(path);
68 sr.setModel("lapsrn", scale);
69
70 sr.upsampleMultioutput(img, outputs, scales, node_names);
71
72 for(unsigned int i = 0; i < outputs.size(); i++)
73 {
74 cv::namedWindow("Upsampled image", WINDOW_AUTOSIZE);
75 cv::imshow("Upsampled image", outputs[i]);
76 //cv::imwrite("./saved.jpg", img_new);
77 cv::waitKey(0);
78 }
79
80 return 0;
81 }
cv::imread
Mat imread(const String &filename, int flags=IMREAD_COLOR)
Loads an image from a file.
cv::WINDOW_AUTOSIZE
the user cannot resize the window, the size is constrainted by the image displayed.
Definition: highgui.hpp:184
std
STL namespace.
highgui.hpp
cv::imshow
void imshow(const String &winname, InputArray mat)
Displays an image in the specified window.
cv
"black box" representation of the file storage associated with a file on disk.
Definition: affine.hpp:51
cv::namedWindow
void namedWindow(const String &winname, int flags=WINDOW_AUTOSIZE)
Creates a window.
dnn_superres.hpp
imgproc.hpp
cv::Mat
n-dimensional dense array class
Definition: mat.hpp:791
cv::Mat::empty
bool empty() const
Returns true if the array has no elements.
cv::waitKey
int waitKey(int delay=0)
Waits for a pressed key.

## Explanation

1. Set header and namespaces
   #include <opencv2/dnn_superres.hpp>
   using namespace std;
   using namespace cv;
   using namespace dnn_superres;

2. Create the Dnn Superres object

   DnnSuperResImpl sr;

   Instantiate a dnn super-resolution object.

3. Read the model

   path = "models/LapSRN_x8.pb"
   sr.readModel(path);

   Read the model from the given path.

4. Set the model

   sr.setModel("lapsrn", 8);

   Sets the algorithm and scaling factor. The last (largest) scaling factor should be given here.

5. Give the node names and scaling factors

   std::vector<int> scales{2, 4, 8}
   std::vector<int> node_names{'NCHW_output_2x','NCHW_output_4x','NCHW_output_8x'}

   Set the scaling factors, and the output node names in the model.

6. Upscale an image

   Mat img = cv::imread(img_path);
   std::vector<Mat> outputs;
   sr.upsampleMultioutput(img, outputs, scales, node_names);

   Run the inference. The output images will be stored in a Mat vector.