Awesome-360-processing

Awesome works and resources relevant to 360 processing. Insipred by awesome-360-vision.

Welcome to pull.

• Awesome-360-processing
  • 360 saliency prediction
    • Image
      • Scanpath prediction
      • Salient regions prediction
      • Dataset or Tools
    • Video
      • Scanpath prediction
      • Salient regions prediction
      • Dataset or Tools
  • 360 visual quality assessment
    • 2D-Plane-based
    • Shpere-based
    • Viewport-based
  • 360 sickness
  • 360 transmission

360 saliency prediction

Image

Scanpath prediction

• SaltiNet: Scan-path prediction on 360 degree images using saliency volumes
  Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto, Noel E. O'Connor
  IEEE International Conference on Computer Vision Workshops (ICCVW), 2017.
  [pdf] [code]

• The prediction of head and eye movement for 360 degree images
  Zhu, Yucheng, Guangtao Zhai, and Xiongkuo Min
  Signal Processing: Image Communication (SPIC), 2018.
  [pdf]

• PathGAN: Visual scanpath prediction with generative adversarial networks
  Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, and Noel E. O’Connor
  European Conference on Computer Vision (ECCV), 2018.
  [pdf] [code]

• ScanGAN360: A generative model of realistic scanpaths for 360° images
  Daniel Martin, Ana Serrano, Alexander W. Bergman, Gordon Wetzstein, Belen Masia
  IEEE Transactions on Visualization and Computer Graphics (TVCG), 2022.
  [project]

• ScanDMM: A deep Markov model of scanpath prediction for 360° Images
  Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, Zhou Wang
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  [pdf][project]

Salient regions prediction

• Saliency in VR: How do people explore virtual environments?
  Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, Gordon Wetzstein
  IEEE Transactions on Visualization and Computer Graphics (TVCG), 2018.
  [pdf][code]

• SalGAN360: Visual saliency prediction on 360 degree images with generative adversarial networks
  Fang-Yi Chao, Lu Zhang, Wassim Hamidouche, and Olivier Deforges
  IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2018.
  [pdf] [code]

• SalNet360: Saliency maps for omnidirectional images with CNN
  Rafael Monroy, Sebastian Lutz, Tejo Chalasani, and Aljosa Smolic
  Signal Processing: Image Communication (SPIC), 2018.
  [pdf] [code]

• A novel superpixel-based saliency detection model for 360-degree images
  Fang, Yuming, Xiaoqiang Zhang, and Nevrez Imamoglu
  Signal Processing: Image Communication (SPIC), 2018.
  [pdf] [code]

• SalGCN: Saliency prediction for 360-degree images based on spherical graph convolutional networks
  Haoran Lv, Qin Yang, Chenglin Li, Wenrui Dai, Junni Zou, Hongkai Xiong
  ACM International Conference on Multimedia (ACMM), 2020.
  [pdf]

• Stage-wise salient object detection in 360° omnidirectional image via object-level semantical saliency ranking
  Guangxiao Ma, Shuai Li, Chenglizhao Chen, Aimin Hao, Hong Qin
  IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
  [pdf][code + dataset]

• A multi-FoV viewport-based visual saliency model using adaptive weighting losses for 360◦ images
  Fang-Yi Chao, Lu Zhang, Wassim Hamidouche, Olivier Déforges
  IEEE Transactions on Multimedia (TMM), 2020.
  [pdf] [code]

• SalBiNet360: Saliency prediction on 360° images with local-global bifurcated deep network
  Dongwen Chen, Chunmei Qing, Xiangmin Xu, Huansheng Zhu
  IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2020.
  [pdf]

Dataset or Tools

• A dataset of head and eye movements for 360 degree images [salient360!]
  Yashas Rai, Jesús Gutiérrez, and Patrick Le Callet
  ACM Multimedia Systems Conference (ACMMSys), 2017.
  [pdf] [project]

• A fixation-based 360° benchmark dataset for salient object detection
  Yi Zhang, Lu Zhang, Wassim Hamidouche, Olivier Deforges
  IEEE International Conference on Image Processing (ICIP), 2020.
  [pdf][dataset]

Video

Scanpath prediction

• Pano2vid: Automatic cinematography for watching 360° videos
  Yu-Chuan Su, Dinesh Jayaraman, and Kristen Grauman
  Asian Conference on Computer Vision (ACCV), 2016.
  [pdf] [code] [dataset]

• Making 360° video watchable in 2D: learning videography for click free viewing
  Yu-Chuan Su, Kristen Grauman
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  [pdf] [code]

• Deep 360 pilot: Learning a deep agent for piloting through 360° sports video
  Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, Min Sun
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  [pdf] [code + dataset]

• Tell me where to look: Investigating ways for assisting focus in 360° video
  Yen-Chen Lin, Yung-Ju Chang, Hou-Ning Hu, Hsien-Tzu Cheng, Chi-Wen Huang, Min Sun
  Conference on Human Factors in Computing Systems (CHI), 2017.
  [pdf] [project]

• Your attention is unique: Detecting 360-degree video saliency in head-mounted display for head movement prediction
  Nguyen, Anh, Zhisheng Yan, and Klara Nahrstedt
  ACM international conference on Multimedia (ACMM), 2018.
  [pdf] [code]

• Predicting head movement in panoramic video: A deep reinforcement learning approach
  Mai Xu, Yuhang Song, Jianyi Wang, Minglang Qiao, Liangyu Huo, and Zulin Wang
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
  [pdf] [code]

• Interactive and automatic navigation for 360° video playback
  Kyoungkook Kang and Sunghyun Cho
  ACM Transactions on Graphics (TOG), 2019.
  [pdf] [project]

• Attention-based deep reinforcement learning for virtual cinematography of 360° videos
  Jianyi Wang, Mai Xu, Lai Jiang, Yuhang Song
  IEEE Transactions on Multimedia (TMM), 2020.
  [pdf]

• DGaze: CNN-based gaze prediction in dynamic scenes
  Zhiming Hu, Sheng Li, Congyi Zhang, Kangrui Yi, Guoping Wang, Dinesh Manocha
  IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
  [pdf] [code] [project]

• A spherical convolution approach for learning long term viewport prediction in 360 immersive video
  Chenglei Wu, Rui-Xiao Zhang, Zhi Wang, Lifeng Sun
  AAAI Conference on Artificial Intelligence (AAAI), 2020.
  [pdf] [code]

• Transitioning360: Content-aware NFoV virtual camera paths for 360° video playback
  Miao Wang, Yi-Jun Li, Wen-Xuan Zhang
  International Symposium on Mixed and Augmented Reality (ISMAR), 2020.
  [pdf]

• Scanpath prediction in panoramic videos via expected code length minimization
  Mu Li, Kanglong Fan, and Kede Ma
  arXiv:2305.02536, 2023.
  [pdf]

Salient regions prediction

• Cube padding for weakly-supervised saliency prediction in 360° videos
  Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, Min Sun
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  [pdf] [code]

• V-BMS360: A video extention to the BMS360 image saliency model
  Lebreton, Pierre, Stephan Fremerey, and Alexander Raake
  IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2018.
  [pdf] [code]

• Saliency detection in 360° videos
  Ziheng Zhang, Yanyu Xu, Jingyi Yu
  European Conference on Computer Vision (ECCV), 2018.
  [pdf] [code]

• Video Saliency Prediction Based on Spatial Temporal Two-Stream Network
  Kao Zhang and Zhenzhong Chen
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
  [pdf] [code]

• Viewport-dependent saliency prediction in 360° video
  Minglang Qiao, Mai Xu, Zulin Wang, Ali Borji
  IEEE Transactions on Multimedia (TMM), 2020.
  [pdf]

• ATSal: An attention based architecture for saliency prediction in 360° videos
  Yasser Dahou, Marouane Tliba, Kevin McGuinness, Noel O'Connor
  arxiv, 2020.
  [pdf] [code]

Dataset or Tools

• A saliency dataset for 360-degree videos
  Anh Nguyen and Zhisheng Yan
  ACM Multimedia Systems Conference (ACMMSys), 2019.
  [pdf] [project]

360 visual quality assessment

2D-Plane-based

• [WS-PSNR] Weighted-to-spherically-uniform quality evaluation for omnidirectional video
  Yule Sun, Ang Lu, Lu Yu
  IEEE Signal Processing Letters (SPL), 2017.
  [pdf]

• [W-SSIM/WMS-SSIM] Subjective and objective quality assessment of omnidirectional video
  Francisco Lopes, João Ascenso, António Rodrigues, Maria Paula Queluz
  Applications of Digital Image Processing, 2018.
  [pdf]

• [CPP-PSNR] Quality metric for spherical panoramic video
  Vladyslav Zakharchenko, Kwang Pyo Choi, Jeong Hoon Park
  Optics and Photonics for Information Processing, 2016.
  [pdf]

• Deep virtual reality image quality assessment with human perception guider for omnidirectional image
  Hak Gu Kim, Heoun-Taek Lim, Yong Man Ro
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
  [pdf]

Shpere-based

• [S-PSNR] A framework to evaluate omnidirectional video coding schemes
  Matt Yu, Haricharan Lakshman, Bernd Girod
  IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2015.
  [pdf]

• [S-SSIM] Spherical structural similarity index for objective omnidirectional video quality assessment
  Sijia Chen, Yingxue Zhang, Yiming Li, Zhenzhong Chen, Zhou Wang
  IEEE International Conference on Multimedia and Expo (ICME), 2018.
  [pdf]

Viewport-based

• Saliency-driven omnidirectional imaging adaptive coding: Modeling and assessment
  Guilherme Luz, João Ascenso, Catarina Brites, Fernando Pereira
  International Workshop on Multimedia Signal Processing (MMSP), 2017.
  [pdf]

• Assessing visual quality of omnidirectional videos
  Mai Xu, Chen Li, Zhenzhong Chen, Zulin Wang, Zhenyu Guan
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2019.
  [pdf]

• Quality assessment of stereoscopic 360-degree images from multi-viewports
  Jiahua Xu, Ziyuan Luo, Wei Zhou, Wenyuan Zhang, Zhibo Chen
  IEEE Picture Coding Symposium (PCS), 2019.
  [pdf]

• Bridge the gap between VQA and human behavior on omnidirectional video: A large-scale dataset and a deep learning model
  Chen Li, Mai Xu, Xinzhe Du, Zulin Wang
  ACM international conference on Multimedia (ACMM), 2018.
  [pdf] [dataset]

• Viewport proposal CNN for 360° video quality assessment
  Chen Li, Mai Xu, Lai Jiang, Shanyi Zhang, Xiaoming Tao
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  [pdf] [code]

• MC360IQA: A multi-channel CNN for blind 360-degree image quality assessment
  Wei Sun, Weike Luo, Xiongkuo Min, Guangtao Zhai, Xiaokang Yang, Ke Gu, Siwei Ma
  IEEE Journal of Selected Topics in Signal Processing (JSTSP), 2019.
  [pdf]

• Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks
  Jiahua Xu, Wei Zhou, Zhibo Chen
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2020.
  [pdf]

• Perceptual quality assessment of omnidirectional images as moving camera videos
  Xiangjie Sui, Kede Ma, Yiru Yao, Yuming Fang
  IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
  [pdf] [code]

• Viewport-based omnidirectional video quality assessment: Database, modeling and inference
  Yu Meng, Zhan Ma
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021.
  [pdf] [project]

• Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network
  Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2022. [pdf]

• Perceptual quality assessment of omnidirectional images
  Yuming Fang, Liping Huang, Jiebin Yan1, Xuelin Liu, Yang Liu
  AAAI Conference on Artificial Intelligence (AAAI), 2022.
  [pdf]

• Subjective quality assessment of user-generated videos
  Yuming Fang, Yiru Yao, Xiangjie Sui, and Kede Ma
  IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 2023.
  [pdf][Database]

• Perceptual quality assessment of virtual reality videos in the wild
  Wen Wen, Mu Li, Yiru Yao, Xiangjie Sui, Yabin Zhang, Long Lan, Yuming Fang, Kede Ma
  arXiv:2206.08751, 2022.
  [pdf]

• Assessor360: Multi-sequence network for blind omnidirectional image quality assessment
  Tianhe Wu, Shuwei Shi, Haoming Cai, Mingdeng Cao, Jing Xiao, Yinqiang Zheng and Yujiu Yang
  arXiv:2305.10983, 2023.
  [pdf][code]

• Perceptual quality assessment of 360° images based on generative scanpath representation
  Xiangjie Sui, Hanwei Zhu, Xuelin Liu, Yuming Fang, Shiqi Wang, Zhou Wang
  arXiv:2309.03472, 2023.
  [pdf][code]

360 sickness

• VRSA net: VR sickness assessment considering exceptional motion for 360° VR video
  Hak Gu Kim, Heoun-Taek Lim, Sangmin Lee, Yong Man Ro
  IEEE Transactions on Image Processing (TIP), 2018.
  [pdf]

• VR sickness versus VR presence: A statistical prediction model
  Woojae Kim, Sanghoon Lee, Alan Conrad Bovik
  IEEE Transactions on Image Processing (TIP), 2020.
  [pdf]

• Assessing individual VR sickness through deep feature fusion of VR video and physiological response
  Sangmin Lee, Seongyeop Kim, Hak Gu Kim, Yong Man Ro
  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2021.
  [pdf]

360 transmission

• Graph learning based head movement prediction for interactive 360 video streaming
  Xue Zhang, Gene Cheung, Yao Zhao, Patrick Le Callet, Chunyu Lin, and Jack Z. G. Tan
  IEEE Transactions on Image Processing (TIP), 2021.
  [pdf]

• QoE evaluation methods for 360-degree VR video transmission
  Zesong Fei, Fei Wang, Jing Wang, Xiang Xie
  IEEE Journal of Selected Topics in Signal Processing (JSTSP), 2020.
  [pdf]

• A Log-Rectilinear transformation for foveated 360-degree video streaming
  David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney
  IEEE Transactions on Visualization and Computer Graphics (TVCG Honorable Mentions), 2021.
  [pdf] [project] [code]