1.

Introduction

Despite rapid developments in electronic media, most people still prefer reading text printed on paper rather than reproduced on electronic displays.¹ Printed media remain more suitable for delivering local news than electronic media, and the packaging industry increasingly relies on the production of visually pleasing and personalized packages by using digital printing. In addition, an increasing number of images are captured each year, and despite the fact that the digitization has created novel ways to share and distribute images, printed images still have their users. For example, the amount of bound photobooks has grown rapidly during the recent years.² These, among other reasons, are why paper and other fiber-based products still play an important role in communication, and printed products, such as books, newspapers, and packages, are an important part of daily life. When a customer decides to purchase a printed book or magazine, one of the key factors is print and image quality. Rather than using technical measurements, humans do not evaluate the quality of print and images based on physical parameters, but rather based on personal preferences and what they see as pleasurable.³

The problem of how humans perceive the quality of a reproduced image is of interest to researchers in many fields, including optics and material physics, image processing (compression and transfer), printing and media technology, and psychology. The problem is particularly difficult for printed media, since solving it requires understanding the paper and ink physics, viewing parameters, optics, and elements of human visual perception. No measure of visual print quality can be defined without ambiguity, because it is ultimately a subjective opinion of an “end-user” observing the result. As a consequence, visual evaluations have been traditionally conducted using groups of human observers, but recent developments in perceptual models and machine vision have made it possible to develop automatic methods of print quality evaluation. The use of machine vision founded on reliable perceptual and print models promises to make it possible to replace humans in laborious off-line evaluations. In addition, such computational methods suggest the potential for quality-optimized on-line measurements during printing.

Image quality assessment (IQA) models can be divided into three categories: full-reference (FR), reduced-reference (RR), and no-reference (NR) methods. In FR methods, a reference image with presumed ideal quality is available, whereas in RR methods only a small amount of information describing the reference image is given as input. NR methods operate in the absence of any reference image. Currently, FR methods are the main approach for evaluating and comparing the quality of digital images, especially compressed ones. The digital representations of the original and compressed images are in correspondence, i.e., there exist no-spatial transformations between the images, and the compression should retain at least photometric equivalence. Therefore, FR measures can be computed in a straightforward manner by computing “distance metrics,” and the actual problem is to define an appropriate metric for the task. NR-IQA is the most difficult task, and the majority of the proposed methods are designed for a single-distortion type and can be considered as domain specific.

The FR-IQA has been shown to be a good approach to predict the image quality when the reference image exists.⁴ With a carefully designed measurement framework, it is possible to apply an FR approach to printed images and relatively high correlations with subjective evaluation results can be achieved, as was shown by Eerola et al.⁵ However, several problems exist when the quality of printed images is evaluated by FR methods.

The first obvious weakness is the fact that the FR methods are suitable only when a digital reference image exists. This is not always the case with printed images. Even more notable problems arise from the basic assumption of the FR approach that the reference image is of ideal quality and can be used as a basis for quality evaluation. For quality assessment (QA) of compressed images, this assumption is perhaps justified; a good image compression method reduces the size of the image in such a manner that the visual appearance of the image changes as little as possible, i.e., the evaluated (compressed) image is visually similar to the reference (original) image. For printed images, however, it is not clear that such an assumption applies. First of all, the original image is in a very different form than the printed image that is being evaluated, making its use as a reference image not only difficult, but also rather questionable.⁶ It is not clear how the difference between a printed photograph and a digital image should be measured. Second, visual quality may be impaired when the original image is transferred onto paper even if no printing artifacts appear, since different aspects of image quality take different degrees of importance on different media. For example, gloss is not a property of a digital image, but has a remarkable effect on the perceived quality of a printed image. Even in the hypothetical ideal situation, where the original image is of “perfect quality” (whatever that is) and the printer or paper do not cause any visible artifacts, quality may still be compromised after transferring onto paper due to the different natures of the media. Third, while making subjective evaluations of printed samples, showing a digital reference to human observers is not a simple matter, since simultaneous viewing of digital and printed images does not allow an observer to adapt both white points, whereas the memory viewing technique does not allow one to directly compare the images.⁷ Often a digital reference image is not shown to the observers, and they are forced to make decisions without knowing what the printed image was supposed to look like.⁵ RR-QA algorithms suffer from similar problems since a reference image is still needed.

For the aforementioned reasons, NR-QA methods are of high interest. However, NR-QA is a much more difficult task than FR-QA, and until recently, there did not exist any general NR-QA methods. All methods were either application specific or measured only a specific kind of distortion such as blur or noise. While these methods have a role in QA, no such method alone can predict the perceived quality of an image. During the last few years, significant developments have led to the creation of generic NR-QA algorithms. Thus, the objective of this study is to determine the efficacy of these new NR models for predicting the subjective quality of printed images by statistically evaluating their performances against subjective mean opinion scores (MOSs) obtained from psychometric experiments on printed samples. Since NR models have been developed for images of natural scenes, this study focuses only on the important application of printed photograph quality analysis while the quality of printed text and graphics is not considered.

The paper is organized as follows. Section 2 introduces the existing generic NR-IQA algorithms that are statistically evaluated in this study. Section 3 presents the data, a method to apply NR-IQA algorithms to printed images, and the results. The results are discussed in Sec. 4, and the conclusion is drawn in Sec. 5.

2.

NR-IQA

Most existing NR-IQA algorithms fall in one of the following categories: (1) distortion-specific IQA algorithms, (2) training-based IQA algorithms, and (3) natural scene statistics (NSS)-based IQA algorithms. The first category is composed of methods that try to model the distortion such as blur^8–10 or blockiness.¹¹ These methods are application specific and are not in the scope of this work. However, it should be mentioned that the distortion-specific methods have also been developed for printed images.¹² The second category contains methods and models that use image-based features and require training on appropriate data.^13–15 These methods are highly dependent on the quality of the selected features and often require a large amount of training data, i.e., distorted images with subjective data that is laborious to collect. Generic image features to describe the image quality are difficult to establish, which limits the application domain. The third category contains methods that are based on the assumption that the pristine natural images form a subset of images that have different statistical properties than the distorted images. NSS methods have turned out to be a very promising approach, and most existing NR-IQA algorithms with good performance more or less rely on NSS. These methods may also require training, but the amount of training is greatly reduced relative to training-based IQA algorithms.

Ideally, NSS features are invariant to image content, but are sensitive to distortions. Such features can be used to assess the image quality by estimating the degree of distortion or the distance of the distorted image to the pristine (natural) image, regardless of the content of the image. Several promising NSS approaches have been proposed in the literature. A typical NSS-based IQA algorithm starts with a multiscale image transform, such as the discrete cosine¹⁶ or wavelet transform,¹⁷ but spatial NSS methods also exist.¹⁸ The computed transform coefficients have statistical properties that vary based on the presence of distortions. For example, the distribution of wavelet coefficients computed from natural images usually has a sharp peak near zero and long and smooth tails. Most image distortions break this regularity, which makes the shape of the coefficient distribution a good feature for IQA.

The coefficient distributions are often parametrized using the (zero mean) general Gaussian distribution (GGD),¹⁹ which has been found to capture the broad spectrum of possible distribution shapes. The GGD is defined as

The related asymmetric GGD²⁰ is also used

The distribution parameters ($\alpha$, $\beta$) or ($\gamma$, ${\beta }_l$, ${\beta }_r$) are then used as features to either (1) classify images based on the decided distortion, then apply a distortion-specific IQA algorithm or (2) estimate the quality directly using regression techniques. Most of the methods are trained on data containing MOSs or difference MOSs (DMOSs) of the images.

The NR-IQA algorithms selected for this study and their basic information are listed in Table 1. All the selected methods are based on NSS. There are also promising training-based methods, such as the learning-based blind image quality (LBIQ)¹³ measure and the visual codebook-based image quality (CBIQ) measure,¹⁵ but these were excluded from the study due to their strong dependence on training data. As discussed later in Sec. 3.3, due to the laborious nature of preparing and subjectively evaluating printed samples, data volume remains a problem and limits the selection of the NR-IQA algorithms that can be used.

Table 1

NR-IQA algorithms used in this study.

The Blind Image Quality Index (BIQI)¹⁷ makes use of NSS features based on wavelet coefficients to first classify an image between different distortion types and to estimate distortion-specific quality scores. A support vector machine (SVM) is used for classification, and a support vector regression (SVR) is used for a distortion-specific quality score estimation. The final quality score is computed as a weighted sum of the distortion-specific quality scores. The weights are the probability estimates provided by the SVM used in the distortion classification stage.

The Distortion Identification-based Image Verity and Integrity Evaluation (DIIVINE) index²³ is an extension of BIQI with a richer set of NSS-based features. Instead of the wavelet transform, a scale-space-orientation decomposition is used.

BLIINDS-II^21,22 is an extension of the Blind Image Integrity Notator (BLIINDS).¹⁶ It uses NSS features based on the local discrete cosine transform coefficients and a simple probabilistic model to predict the quality.

Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE)^25,26 differs from the methods above because it uses only spatial features, albeit over multiple scales. There is no need to map the data to a different coordinate domain, such as the wavelet or DCT domain. The NSS used is based on locally normalized luminances [mean subtracted contrast normalized (MSCN) coefficients].¹⁸

BRISQUE was further developed leading to the very generic, training-free Natural Image Quality Evaluator (NIQE).²⁷ It uses similar NSS features, but instead of establishing the quality value directly from features using an SVR, the NSS features are modeled as multivariate Gaussian. The main advantage of NIQE is that, unlike other methods, it does not require training data with subjective human evaluations. Instead, the model is constructed from features drawn from a corpus of undistorted natural images, making it the first truly distortion-independent NR-IQA algorithm.

The hybrid NR (HNR) model²⁴ is based on curvelet, wavelet, and discrete cosine transform coefficient statistics, and the quality is predicted using the peak coordinates of the logarithmic probability distribution of the coefficient magnitudes (LPMC).

3.

Experiments

In this section, we introduce the data used in this study, i.e., the particular types of paper used, the (printed) natural images, the psychometric subjective evaluation results (subjective scores), and the methods used to process the raw data.

3.1.

Test Sets

The original objective guiding the data collection was to evaluate the effect of paper grade on the overall perceived visual quality of printed images. Therefore, our test sets consist of several paper grades at the cost of image contents. The first set of test samples (Test Set A) consisted of natural images printed with a prepress proofing inkjet printer on 16 different paper grades. The paper grades and the printing process were selected according to current practices, as described in detail in previous publications.^29–31 The natural images used in the study are presented in Fig. 1. The image contents were selected based on current practices and previous experience in media technology, and typical content types such as objects with details (cactus), a human portrait (man), and a landscape (landscape) were included. The fourth image content combined all the types (studio).

Fig. 1

Image contents used in this study: (a) man; (b) lake; (c) cactus; and (d) studio.

The second set of samples (Test Set B) consisted of images printed with a production-scale electrophotographic (EPG) printer on 21 different paper grades. The same image contents were used excluding studio [Fig. 1(d)]. The subjective evaluations, described below, were performed separately for both sets and image contents resulting in seven separate tests of 16 or 21 samples, respectively.

3.4.

Results

The descreening was performed using six different cut-off wavelengths: $0.05,0.10,\dots ,0.30\mathrm{mm}$, and seven scale factors: $0.08,0.10,\dots ,0.20$ corresponding to $100,125,\dots ,250\mathrm{dpi}$. The NR-IQA algorithms were applied with every cut-off wavelength-scale pair to find the optimal parameter values for each method.

Figure 2 presents the examples of coefficient histograms that were used to compute NSS features. For visualization purposes, the histograms are presented for four samples: low-quality inkjet sample (the sample with lowest MOS in Test Set A), low-quality EPG sample, high-quality EPG sample, and high-quality inkjet sample. Although the two test sets were never combined, the quality variation between the selected samples is high. Most people would prefer the high-quality inkjet sample as the best one, followed by the high-quality EPG sample and low-quality EPG sample, the low-quality inkjet sample being the worst one. As can be seen from Fig. 2(a), the wavelet coefficient distributions used by BIQI seem to have a lower-standard deviation for high-quality samples and a higher one for low-quality samples, which indicates their potential in print QA. The same also applies for the MSCN coefficients used in BRISQUE and NIQE. However, the scale-space-orientation coefficients used in DIIVINE seem to be more suitable for distinguishing printing methods from each other than predicting the quality of printed images.

Fig. 2

Examples of coefficient histograms used to compute NSS features: (a) wavelet coefficient distribution used in BIQI; (b) MSCN coefficient distribution used in BRISQUE and NIQE; and (c) scale-space-orientation coefficient distribution used in DIIVINE.

Figures 3 and 4 present the results of the best NR-IQA algorithm with the optimal cut-off wavelength and scale factor for each image content. As can be seen, the correlations are very high. However, selecting the optimal parameter values for each method produces overly optimistic results. Since the image content is often unknown beforehand in a practical application, the parameters should be fixed in such a manner that the NR-IQA algorithm is not sensitive to image content. Figure 5 shows the optimal parameter values for each method for different image contents, while Table 2 presents the corresponding correlations [linear correlation coefficient (LCC) and Spearman’s rank correlation coefficient (SRCC)] between the MOS and algorithm scores. Also, the parameter values that maximize the mean correlation over all image contents and the corresponding correlation coefficients are presented.

Fig. 3

The best method, cut-off wavelength, and scale factor selected for each image content of Test Set A: (a) man; (b) lake; (c) cactus; and (d) studio.

Fig. 4

The best method, cut-off wavelength, and scale factor selected for each image content of Test Set B: (a) man; (b) lake; and (d) cactus.

Fig. 5

Optimal parameter values for different image contents in Test Set A (red) and Test Set B (blue): (a) BIQI; (b) BLIINDS-II; (c) BRISQUE; (d) DIIVINE; (e) HNR-noise; and (f) NIQE. If for two or more image contents of the optimal parameter values were the same, then the markers were slightly shifted to make the figures more readable.

Table 2

Correlations with optimal parameter values for each image content. The first number in each entry is LCC value, and the second is SRCC. The numbers shown in bold indicate the best results.

	Image content	Method
		BIQI	BLIINDS-II	BRISQUE	DIIVINE	HNR-noise	NIQE
Test Set A	Man	0.97/0.92	0.87/0.77	0.79/0.82	0.74/0.64	0.98/0.95	0.97/0.92
	Lake	0.98/0.91	0.69/0.56	0.88/0.60	0.95/0.83	0.98/0.91	0.98/0.92
	Cactus	0.97/0.93	0.72/0.62	0.93/0.85	0.92/0.86	0.98/0.95	0.97/0.92
	Studio	0.94/0.93	0.76/0.75	0.94/0.77	0.91/0.91	0.98/0.94	0.97/0.84
	Mean	0.97/0.89	0.33/0.32	0.84/0.69	0.85/0.77	0.97/0.90	0.97/0.85
Test Set A (multipurpose papers only)	Man	0.68/0.78	0.56/0.54	0.79/0.75	0.23/0.27	0.64/0.81	0.57/0.71
	Lake	0.87/0.88	0.60/0.54	0.69/0.70	0.39/0.40	0.76/0.76	0.59/0.69
	Cactus	0.77/0.87	0.85/0.86	0.76/0.82	0.62/0.68	0.91/0.87	0.78/0.78
	Studio	0.83/0.79	0.58/0.55	0.42/0.35	0.81/0.70	0.75/0.80	0.43/0.38
	Mean	0.64/0.72	0.39/0.39	0.50/0.43	0.27/0.25	0.64/0.66	0.34/0.44
Test Set B	Man	0.86/0.88	0.82/0.81	0.87/0.87	0.57/0.56	0.89/0.85	0.92/0.93
	Lake	0.89/0.81	0.65/0.60	0.77/0.70	0.88/0.89	0.70/0.54	0.85/0.73
	Cactus	0.85/0.76	0.75/0.75	0.87/0.83	0.71/0.72	0.78/0.67	0.70/0.67
	Mean	0.82/0.79	0.67/0.64	0.80/0.76	0.59/0.64	0.67/0.53	0.73/0.66

As it can be seen from Fig. 3, Test Set A contains two relatively distinct clusters of different qualities: high-quality photopapers and lower-quality multipurpose papers. Any NR-IQA algorithm that distinguishes the two clusters and places them into the right order gets a high LCC value, while a method that fails to correct select the better cluster gets a high negative correlation coefficient. Moreover, the overall correlation coefficient increases if an algorithm places the cluster further away even if the correlations inside the clusters do not change. Therefore, based on the LCCs obtained using Test Set A, one can only determine whether a method works adequately at a coarse level or not, but it does not reveal the relative performances of the method studied in detail. Hence, when selecting the optimal combination parameter values, SRCC appears to be a more suitable measure.

To further study the performances of the various NR-IQA models on Test Set A using LCC, the samples representing the two clusters were divided into two and an additional test was carried out using the challenging lower-quality multipurpose papers. The result is shown in Fig. 6 and Table 2, and as expected, the correlations are much lower than on the full Test Set A. However, although the quality variation inside the set is very low, clear correlations between the MOS values and the algorithm scores can be observed.

Fig. 6

The best method, cut-off wavelength, and scale factor selected for each image content of Test Set A (only multipurpose papers): (a) man; (b) lake; (c) cactus; and (d) studio.

Figures 7 and 8 show the results with optimal parameter values (mean SRCC over all image contents). Based on the figures, it is clear that the image content significantly affects the algorithm scores. Although the within-content correlations are high, different image quality score values for different contents cause a low overall correlation over all contents. However, it should be noted that the subjective evaluation results were separately scaled to the interval 1 to 5 for each content, and the MOS values are not directly comparable between the image contents. Therefore, combining the plots without preprocessing is not a well-grounded approach.

Fig. 7

Unaligned algorithm scores with the best cut-off wavelengths and scale factors selected for each method on Test Set A: (a) BIQI; (b) BLIINDS-II; (c) BRISQUE; (d) DIIVINE; (e) HNR-noise; and (f) NIQE.

Fig. 8

Unaligned algorithm scores with the best cut-off wavelengths and scale factors selected for each method on Test Set B: (a) BIQI; (b) BLIINDS-II; (c) BRISQUE; (d) DIIVINE; (e) HNR-noise; and (f) NIQE.

One option to avoid the above problem is to scale the IQA algorithm scores for each image content, as was done in Ref. 5. From a practical viewpoint, it is more interesting to put the paper grades in a proper order than to find the overall quality of a single-printed image on some abstract quality scale. Therefore, the subjective evaluation as well as the IQA algorithm scores should be similar over different image contents for the same paper grade. The subjective evaluation results were always scaled to the interval 1 to 5, but the IQA algorithm scores may differ significantly across image contents. Therefore, either the IQA algorithm scores need to be scaled to a common scale or the analysis needs to be done separately for different image contents. We selected the first option, since the number of samples (16 or 21) was not enough to find statistically significant differences between the IQA algorithms. Therefore, the different image contents were combined to form a larger test set by scaling the IQA algorithm scores. Here, the scaling was performed linearly. Let ${\mathit{x}}_{\mathit{n}}=(x_{n,1},\dots ,x_{n,M})$ represent the IQA algorithm scores of one assessment for all samples ($1,\dots ,M$) within a single-image content $n$. Then, in the linear model, we have

Eq. (4)

$${\widehat{x}}_{n,i}={\widehat{\mathit{b}}}_{\mathit{n}}\left(\begin{array}{c}1\\ x_{n,i}\end{array}\right),$$

where ${\widehat{\mathit{b}}}_{\mathit{n}}=(b_{n,1},b_{n,2})$ are selected by minimizing the errors between the image contents as follows:

Eq. (5)

$${\widehat{\mathit{b}}}_{\mathit{n}}=\underset{{\widehat{\mathit{b}}}_{\mathit{n}}}{\mathrm{argmin}}\sum _i{[x_{1,i}-(b_{n,1}+b_{n,2}{x}_{n,i})]}^2.$$

For the first image content ${\widehat{\mathit{b}}}_{\mathbf{1}}=(\mathrm{0,1})$ and for the remaining image contents ${\widehat{\mathit{b}}}_{\mathit{n}}$ is such that the IQA algorithm scores are converted to values similar to the values of the first image content on the same paper grade. The above-mentioned method does not allow combining Test Sets A and B, since there were no samples with comparable quality in different sets that could have been used to scale the MOS values to the same scale. Figures 9 and 10 and Table 3 present the results with aligned algorithm scores. Table 3 also lists the results for selected FR-IQA algorithms obtained using similar procedures:⁵ peak signal-to-noise ratio (PSNR), structural similarity metric (SSIM),³⁵ multiscale SSIM (MS-SSIM),³⁶ and visual information fidelity (VIF).³⁷

Fig. 9

Aligned algorithm scores with the best cut-off wavelengths and scale factors selected for each method on Test Set A: (a) BIQI; (b) BLIINDS-II; (c) BRISQUE; (d) DIIVINE; (e) HNR-noise; and (f) NIQE.

Fig. 10

Aligned algorithm scores with the best cut-off wavelengths and scale factors selected for each method on Test Set B: (a) BIQI; (b) BLIINDS-II; (c) BRISQUE; (d) DIIVINE; (e) HNR-noise; and (f) NIQE.

Table 3

Correlations between the MOS and the aligned algorithm scores from NR-IQA algorithms and from selected FR-IQA algorithms.5 The numbers shown in bold indicate the best results.

IQA algorithm	Test Set A		Test Set B
	LCC	SRCC	LCC	SRCC
PSNR (FR)	0.44	0.29	0.54	0.43
SSIM (FR)	0.98	0.92	0.80	0.70
MS-SSIM (FR)	0.98	0.91	0.89	0.79
VIF (FR)	0.98	0.90	0.82	0.74
BIQI (NR)	0.96	0.88	0.74	0.77
BLIINDS-II (NR)	0.75	0.66	0.68	0.68
BRISQUE (NR)	0.82	0.73	0.79	0.78
DIIVINE (NR)	0.82	0.75	0.52	0.59
HNR-noise (NR)	0.93	0.86	0.67	0.58
NIQE (NR)	0.97	0.84	0.73	0.67

The statistical significance of the previous results was studied using the variance test. It expresses the trust in the superiority or inferiority of one QA algorithm over another based on performance measures. The test is based on the assumption that the residuals (difference between MOS and the IQA algorithm score linearly fitted to MOS) are normally distributed. The normality of the residuals was tested using the Lilliefors test³⁸ at a 5% significance level, and the residuals were shown to follow a normal distribution for all methods in Test Set A and for all methods except DIIVINE in Test Set B. Moreover, since DIIVINE had the lowest LCC in Test Set B, the non-normality has no significant effect on our conclusions. The F-test was used to test whether the variances of the residuals of two QA algorithms are identical, i.e., the QA algorithm residuals are randomly drawn from the same distribution. The null hypothesis is that the residuals of both QA algorithms come from the same distribution and are statistically indistinguishable with 90% confidence. The significance test results for the aligned algorithm scores are shown in Tables 4 and 5 for both test sets and for all possible pairings of QA algorithms.

Table 4

F-test results for Test Set A: 0 means that the QA algorithms are statistically indistinguishable from each other, 1 means that the IQA algorithm for the row is statistically better than the IQA algorithm in the column, and −1 means that the IQA algorithm in the row is statistically worse than the IQA algorithm for the column.

	BIQI	BLIINDS-II	BRISQUE	DIIVINE	HNR-noise	NIQE
BIQI	—	1	1	1	1	0
BLIINDS-II	$-1$	—	0	0	$-1$	$-1$
BRISQUE	$-1$	0	—	0	$-1$	$-1$
DIIVINE	$-1$	0	0	—	$-1$	$-1$
HNR-noise	$-1$	1	1	1	—	$-1$
NIQE	0	1	1	1	1	—

Table 5

F-test results for Test Set B.

	BIQI	BLIINDS-II	BRISQUE	DIIVINE	HNR-noise	NIQE
BIQI	—	0	0	1	0	0
BLIINDS-II	0	—	0	0	0	0
BRISQUE	0	0	—	1	0	0
DIIVINE	$-1$	0	$-1$	—	0	$-1$
HNR-noise	0	0	0	0	—	0
NIQE	0	0	0	1	0	—

4.

Discussion

As described earlier, the training of IQA algorithms was performed using a separate set of distorted or pristine digital images, and the printed and scanned images were used only for testing. Therefore, the training and testing data were very different from each other and the training data did not contain the printing distortions. It is understandable that this reduced the performance of the learned models and might favor certain IQA algorithms. However, as the results showed, most IQA algorithms were still able to achieve high correlations with MOS, suggesting that the distortions in the training data were close to the distortions in the testing data. Moreover, since the ultimate goal is to develop an NR-IQA algorithm that needs to be trained only once, after which it should be possible to apply it to any appropriate image quality application, it is worthwhile to also evalate the algorithms’ dependency on the quality of the training data. This justifies the use of different training and testing data as it reveals not only the performance of the methods, but also their sensitivity to imperfect training data.

On both test sets, most of the NR methods outperformed PSNR and the best methods were shown to produce almost as good results as state-of-the-art FR-IQA algorithms. As mentioned earlier, Test Set A contains two relatively distinct clusters, making SRCC a more reliable metric for comparing the performance of IQA algorithms. Based on the SRCC results, the best methods are BIQI, NIQE, and HNR-noise and these are also statistically significantly better than the rest of the methods. Test Set B does not contain similar distinguishable clusters, and therefore, LCC also provides useful information. Unlike SRCC, it also measures the linear dependence, i.e., whether a change in the quality at the high-quality end of the scale corresponds to a similar change at the low-quality end. Based on the LCC results, the best methods are BIQI, BRISQUE, and NIQE. The SRCC results support this conclusion. However, it should be noted that no statistically significant differences in performance were found between these methods and BLIINDS-II or HNR-noise.

The notably lower performance of DIIVINE compared with BIQI on both test sets is a surprising result, since DIIVINE is an extension of BIQI and it outperformed BIQI in experiments made using the LIVE database. The results shown in Fig. 2 suggest that the scale-space-orientation decomposition used by DIIVINE is more sensitive to the different printing methods and, therefore, a less suitable approach for the QAs of printed images. Moreover, since it is the more complex of the two (88 NSS features in DIIVINE compared with 18 in BIQI), there is a higher chance of over-tuning, making it more vulnerable to the large differences between training and testing data.

As can be seen in Figs. 7 and 8, image content has a noticeable effect on the algorithm scores. The main reasons for this are the fact that the NR-IQA algorithms were trained using a separate database containing different image contents, and the scaling of the MOS values for each type of image content was performed separately. Scaling the algorithm scores similarly to the MOS values does not really solve the problem, since we do not know if the effect of image content will be eliminated even if the above problems did not exist. Therefore, the results presented in Figs. 9 and 10 can be seen as an upper limit of method performance and to achieve such results, a manual effort is needed. However, if the parameters $a$ and $b$ that define the scaling of the algorithm scores could be predicted based on the image content, then the results presented in Figs. 9 and 10 could be achieved using a fully automatic method.

BIQI and NIQE perform well on both the test sets. While BIQI achieves slightly higher correlations, NIQE contains one significant benefit: it requires only pristine images to be trained. This enables training with a much larger number of images, making it less vulnerable to new image content. The results show that the NIQE is less sensitive to different image contents than the other tested NR-IQA algorithms [see Fig. 7(f)].

5.

Conclusion

We applied the leading general-purpose NR-IQA algorithms to printed images and evaluated the performance of several state-of-the-art NR-IQA algorithms on an extensive set of printed photographs. We found that the BIQI and NIQE algorithms outperformed the other QA algorithms. Of these two, NIQE was less sensitive to image content, making it the most promising current method for NR-printed IQA.