Sequence-to-Segment Networks For Segment Detection

Authors:
Zijun Wei Stony Brook University
Boyu Wang Stony Brook University
Minh Hoai Nguyen Stony Brook University
Jianming Zhang Adobe Research
Zhe Lin Adobe Research
Xiaohui Shen ByteDance AI Lab
Radomir Mech Adobe Systems Incorporated
Dimitris Samaras Stony Brook University

Introduction:

Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments.To address this problem, the authors propose the Sequence-to-Segment Network (S$^2$N), a novel end-to-end sequential encoder-decoder architecture.

Abstract:

Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments. To address this problem, we propose the Sequence-to-Segment Network (S$^2$N), a novel end-to-end sequential encoder-decoder architecture. S$^2$N first encodes the input into a sequence of hidden states that progressively capture both local and holistic information. It then employs a novel decoding architecture, called Segment Detection Unit (SDU), that integrates the decoder state and encoder hidden states to detect segments sequentially. During training, we formulate the assignment of predicted segments to ground truth as bipartite matching and use the Earth Mover's Distance to calculate the localization errors. We experiment with S$^2$N on temporal action proposal generation and video summarization and show that S$^2$N achieves state-of-the-art performance on both tasks.

You may want to know: