Learning Spatial Relationships between Samples of Patent Image - - PowerPoint PPT Presentation

Managed by Triad National Security, LLC for the U.S. Department of Energy’s NNSA

Learning Spatial Relationships between Samples of Patent Image Shapes

• J. Castorena, M. Bhattarai, D. Oyen

June, 2020

Introduction

• USPTO includes in their guidelines for patent

application design that ”the drawing disclosure is the most important element

• f

the application.

• Most

widely available methods for classification and retrieval are text based and a patent agents reviews results. If we can cut times and make the process more efficient by automatically classifying and retrieving patent documents based on images it would be highly beneficial.

• Challenges:

Image generation mechanism variances.

• Lack of texture or color: binary images
• Presence of multiple components
• View-point
• Drawing style of artisan.

Figure 1: Examples of patent images.

This work:

Focus:

• Learning based methods to classify/retrieve patent images while being more

effective in the binary image domain and at disentangling the learned functionals from some of the variations in the image generation mechanisms.

Idea:

• Analyze shape instead of image texture and learn geometrical features

characterizing shapes. Overall method:

• Extract samples from image shapes and learn spatial relationships between

the points that characterize shapes [Castorena and Oyen, 2020].

Figure 2: Learning spatial relationships characterizing image shapes

Method

• Image is sampled by clustering pixel

shape locations and computing representative point.

• Algorithm:

Mini-batch K-means [Sculley, 2010] .

• Orders of magnitude more computationally

efficient than standard k-means with only small penalties in performance.

• Learning

node and spatial relationship features characterizing shape samples.

• Dynamic graph CNN’s [Wang et al.,

2019]

Figure 4: Learning spatial relationships characterizing image shapes

Experiments I

• Classification against the effect of data transformations

plausibly influencing the patent image generation mechanisms.

• First evaluation: on MNIST because it allows simple

transformation manipulations

• We want to demonstrate the inability of standard CNN’s
• perating in the Euclidean domain to cope with data

transformations in the absence

• f

sufficient data characterizing these. A matter of crucial importance to build effective classification/retrieval solutions in the patent application realm.

• Two experimental cases:
• (1)

training examples include images subjected to the corresponding transformations.

• (2) training does not include examples of the transformed data.

Transformations are drawn uniformly from : ±90◦ rotation, ± 9 pixels translations and [0.2,1] scaling.

• Results show that our proposed framework remains more
• r less invariant to the aforementioned transformations in

comparison to LeNet-5 and that it outperforms it in most cases in its own intended benchmark dataset. Figure 5: 1st row: the MNIST dataset and 2nd row sampling of the MNIST digits.

Experiments II

• Classification evaluated against the CLEF-IP 9-

class dataset, image shapes from an unknown and varying viewpoint, presence of cross-class components and style.

• Empirical results reflect that CNN’s operating in

the Euclidean domain are not powerful enough to disentangle performance from generation mechanisms in binary patent imagery.

• Method

based

• n

the Fisher vector representation

• utperforms

standard CNN methods.

• Our proposed method seems to better cope with

the plausible variations in the image generation mechanism and

• utperforms

all compared methods which includes to the best of our knowledge state of the art.

Figure 6: Examples of the CLEF-IP 9-class images.

References

• [Castorena and Doyen, 2020] Castorena, J., & Oyen, D. Learning Shapes on Image Sampled Points

with Dynamic Graph CNNs. In 2020 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI) (pp. 1-4).

• [Csurka et al. 2011]. Csurka, G., Renders, J. M., & Jacquet, G. (2011). XRCE's Participation at Patent

Image Classification and Image-based Patent Retrieval Tasks of the Clef-IP 2011. In CLEF (Notebook Papers/Labs/Workshop) (Vol. 2, No. 4).

• [He et al., 2016] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image
• recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp.

770-778).

• [Lecun et al., 1998 ] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, "Gradient-based learning applied to

document recognition," in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.

• [Piroi et al., 2011] F. Piroi, et al. "CLEF-IP 2011: Retrieval in the Intellectual Property Domain." CLEF

(notebook papers/labs/workshop). 2011.

• [Sculley, 2010] D. Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th

international conference on World wide web (WWW ’10). Association for Computing Machinery, New York, NY, USA, 1177–1178.

• [Wang et al., 2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and

Justin M. Solomon. 2019. Dynamic Graph CNN for Learning on Point Clouds. ACM Trans. Graph. 38, 5, Article 146 (October 2019).