当前位置: 首页  科学研究  学术交流  学术报告

计算机科学与技术学科机器学习与视觉研究所系列学术报告(张力 复旦大学)

发布者:戴 情   发布时间:2021-11-28  浏览次数:206

复旦大学-浙江师范大学“机器学习与视觉前沿论坛”系列报告六



报告题目:SOFT: Softmax-free Transformer with Linear Complexity

报告专家:张力(复旦大学)

报告时间11月29日16:00-16:45

报告地点:腾讯会议号766-826-483(复旦浙师大MLV前沿论坛)

报告摘要ViTs have pushed the state-of-the-art for various visual recognition tasks. However, the employment of self-attention modules results in a quadratic complexity. An in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations are rooted in keeping the softmax self-attention during approximations. Keeping this softmax operation challenges any subsequent linearization efforts. Based on this insight, a softmax-free transformer or SOFT is proposed. To remove softmax in self-attention, Gaussian kernel function is used to replace the dot-product similarity without further normalization. This enables a full self-attention to be approximated via a low-rank matrix decomposition. The robustness of the approximation is achieved by calculating its Moore-Penrose inverse using a Newton-Raphson method. Extensive experiments on ImageNet show that SOFT significantly improves the computational efficiency of existing ViT variants, resulting in superior trade-off between accuracy and complexity.


报告专家简介

张力(复旦大学)

张力,复旦大学大数据学院青年研究员,获得国家自然科学基金青年项目,上海科技青年35人引领计划(35U35),上海海外高层次人才计划支持;伦敦玛丽女王大学电子工程与计算机科学系博士,牛津大学工程科学系博士后研究员,英国剑桥三星人工智能研究院研究科学家。其研究成果多次发表在人工智能与计算机视觉顶级期刊与会议上,其发表的论文据 Google Scholar 统计总引用次数超过 4700 次。




邀请人:郑忠龙