A Docker-Based Replicability Study of a Neural Information - - PowerPoint PPT Presentation

▶

Apr 06, 2024 335 likes •416 views

A Docker-Based Replicability Study of a Neural Information Retrieval Model Nicola Ferro, Stefano Marchesin, Alberto Purpura , Gianmaria Silvello University of Padua, Italy The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019)

SLIDE 1

A Docker-Based Replicability Study of a Neural Information Retrieval Model

Nicola Ferro, Stefano Marchesin, Alberto Purpura, Gianmaria Silvello  University of Padua, Italy The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

SLIDE 2

The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

Neural Vector Space Model

Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2018. Neural Vector Spaces for Unsupervised Information Retrieval. ACM Trans. Inf. Syst. 36, 4, Article 38 (June 2018), 25 pages. DOI: https://doi.org/10.1145/3196826.

wk, . . . , wk+n ∈ dp

Minimize the distance Maximize  the distance Word n-gram

dp dn

SLIDE 3

The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

Results

Retrieval results on the Robust04 (T) collection computed with the two shared Docker images of NVSM.

R e p l i c a t e d

MAP nDCG@100 P@10 Recall Original 0.150 0.287 0.298 – OSIRRC run 0.142 0.276 0.288 0.616 CPU (run 0) 0.138 0.271 0.285 0.608 GPU (run 0) 0.137 0.265 0.277 0.610 GPU (run 1) 0.138 0.270 0.277 0.607 GPU (run 2) 0.137 0.268 0.270 0.611

<latexit sha1_base64="ScYDaTIPx7Te3S+eIYKxeM4GIxc=">AF3ictVTPb9MwFM62Bkb41cGRi0U1B1W2WX9wa1QpMFhUDraTVqynHd1prjRI6DVIVcuHAIa78W9z4Q7jdGlX1i5DmnhR4qf3nv1937Njx+csUBD+WlvfyJk3bm7esm7fuXvfn7rQTfwQkloh3jck8cODihngnYU5we+5Ji1+H0yDltJvmjD1QGzBPv1cSnPRePBsygpUO9bc2ftsOHTERKeyEHMs4kh/TJ7bASnsCDp639Fe8bO43EITabelRD21KMOcLpbYN7HFCzdp+0z08AMW3kmkwzHfma9luyBXTQkJXRCiBjyNYQhUYX5os12sZyWf1JDlfH6EVEs5zQgtqPtPjHZ3F1APX7fbTSBDcTneXjljyVo1i8wUD5aqHqudAbdbHVAMQGO1cIflrPIoCyCFRSArC+0Or9f0fO6ma1kWrNpMOVyGj62nOgFzZFhbhVy+nuaraSWakUZOd3vJLJuKwfwv7+cLsASnBpYdlDoFI7VWP/THngkdKlQhOMgOEHQV70IS02I09iyw4D6mJziET3RrsAuDXrR9H6KwbaODMDQk/oVCkyjizMi7AbBxHV0pYvVOLiYS4KrciehGtZ7ERN+qKgZ0DkAPlgeSyAwMmKVF8oh1MJNcARljiYnSV6Klm4AuSl52umW9OaXyu71C40Xajk3jkfHYKBrIqBkN45XRMjoGydm5T7kvua8mNj+b38zvZ6Xra+mch8ZfZv74A1vGzec=</latexit>

SLIDE 4

The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

Runs Similarity

Kendall's correlation coefficient values between the runs we computed with the NVSM GPU and CPU Docker images considering the top 100 ranked documents in each run.

τ

GPU (run 0) GPU (run 1) GPU (run 2) CPU GPU (run 0) 1.0 0.025 0.025 0.018 GPU (run 1) 0.025 1.0 0.089 0.014 GPU (run 2) 0.025 0.089 1.0 0.009 CPU 0.018 0.014 0.009 1.0

<latexit sha1_base64="betvrDmKt1kp0hCev74OFC2a9ds=">ADfXicjVJPT9swFHeTMaBjWxnHXaxVIqmKMlAlBsaB3bspBWQmqpy3NfWwnEi20GqQj/Fvhk3vgqXzWnTgdtN2rMsv3+/3t+enHGmdK+/1hz3Fcbrze3tutvdt6+e9/Y/XCl0lxS6NKUp/ImJgo4E9DVTHO4ySQJOZwHd9elPHrO5CKpeKHnmbQT8hYsBGjRBvXYLf2ExuJYhgzUWgS5zIWSHvqzOrY1sO8GWniw9lLrDfemkFlhW1oWxoialL3ZPDYHxoHnz7l9zw9P7DdoL0HrVHbxJcia59VNMf/QRO27NpzsEXn/2bpvzu85gWrS9rL8HPdCsMEYjhn/kPGs0yvRS8rgSV0kSVdAaNh2iY0jwBoSknSvUCP9P9gkjNKIdZPcoVZITekjH0jCpIAqpfzLdnhveNZ4hHqTRXaDz3vkQUJFqmsQmMyF6olZjpfNvsV6uR+1+wUSWaxB0UWiUc6xTXK4iHjIJVPOpUQiVzPSK6YRIQrVZ2LoZQrD65XlKvSCL174/bh5/rUaxb6iD6hQxSgU3SOvqEO6iJae3Kw03KOnF/uvZ9RapTq3C7CFL3NPfSk/kPg=</latexit>

SLIDE 5

The SIGIR 2019 Open-Source IR Replicability Challenge (OSIRRC 2019) July 25, Paris, France

Final Remarks

CPU GPU (run 0) GPU (run 1) GPU (run 2) FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS4-41991 FBIS4-7811 FBIS4-7811 FBIS4-7811 FBIS4-45469 FBIS4-43965 FBIS4-41991 FBIS4-41991 FBIS3-54945 FBIS3-23986 FBIS3-23986 FBIS3-23986 FBIS4-7811 FBIS4-41991 FBIS4-65446 FBIS4-65446

<latexit sha1_base64="tr0RPZ4kVOdIXYPbH/vZI3kDgx0=">AAAECXicpVNNb9MwGPYSPkb56tiRAxYVaDu0itska27TJvFxK4Juk5qqclyntZY4ke2gVaFXLvwVLhxAiCv/gBv/BqcLVZuOXXijWM/7vB9+XssO0ohJZVm/twzzxs1bt7fv1O7eu//gYX3n0YlMMkFonyRRIs4CLGnEOO0rpiJ6lgqK4yCip8H5cRE/fU+FZAl/p2YpHcZ4wlnICFaaGu0YT6A2P6ATxnOFgyzCYp6LD4tvXoOlHff68Dl8qdc9kXFo7a96aM1r70Pfh/60ULSs9xW9UEGYvzh6/bbTdJw28ua66P9p31/uUdB200aehyrJdvOgi1C1xbXsSt/1qO3Y7oYaTXc81ynodRUrXkWpHsD2bGdjrHbH67qb015L/1Ps3wGvEuU6tu2ue2Ufn/Lx8i6M6g2rZS0MbgJUggYorTeq//LHCcliyhWJsJQDZKVqmGOhGInovOZnkqaYnOMJHWjIcUzlMF/c5Dl8ppkxDBOhf67ggl2tyHEs5SwOdGaM1VRWYwV5VWyQqbA7zBlPM0U5udwozCKoElg8CzhmghIVzTTARDCtFZIpFpgo/Xhq+hBQdeRNcNJuoU6r/cZuHB6Vx7ENHoOnYA8gcAAOwSvQA31AjI/GZ+Or8c38ZH4xv5s/LlONrbJmF6yZ+fMPnswh/A==</latexit>

Thank you!

Alberto Purpura, purpuraa@dei.unipd.it

A Docker-Based Replicability Study of a Neural Information Retrieval Model

Neural Vector Space Model

wk, . . . , wk+n ∈ dp

dp dn

Results

Retrieval results on the Robust04 (T) collection computed with the two shared Docker images of NVSM.

R e p l i c a t e d

MAP nDCG@100 P@10 Recall Original 0.150 0.287 0.298 – OSIRRC run 0.142 0.276 0.288 0.616 CPU (run 0) 0.138 0.271 0.285 0.608 GPU (run 0) 0.137 0.265 0.277 0.610 GPU (run 1) 0.138 0.270 0.277 0.607 GPU (run 2) 0.137 0.268 0.270 0.611

Runs Similarity

Kendall's correlation coefficient values between the runs we computed with the NVSM GPU and CPU Docker images considering the top 100 ranked documents in each run.

τ

GPU (run 0) GPU (run 1) GPU (run 2) CPU GPU (run 0) 1.0 0.025 0.025 0.018 GPU (run 1) 0.025 1.0 0.089 0.014 GPU (run 2) 0.025 0.089 1.0 0.009 CPU 0.018 0.014 0.009 1.0

Final Remarks

CPU GPU (run 0) GPU (run 1) GPU (run 2) FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS3-55219 FBIS4-41991 FBIS4-7811 FBIS4-7811 FBIS4-7811 FBIS4-45469 FBIS4-43965 FBIS4-41991 FBIS4-41991 FBIS3-54945 FBIS3-23986 FBIS3-23986 FBIS3-23986 FBIS4-7811 FBIS4-41991 FBIS4-65446 FBIS4-65446

Top 5 documents in the runs computed with the CPU and the

Thank you!