Paper Image

Compressing recommendation models using knowledge distillation

Published on:

8 November 2023

Primary Category:

Information Retrieval

Paper Authors:

Zhangchi Zhu,

Wei Zhang

Bullets

Key Details

Decomposes recommendation models into input, intermediate, and output layers

Distills knowledge in each layer to address deficiencies of prior work

Proposes techniques like neighbor-based input, preference consistency, and soft labels

Enables 10x smaller student models to match/exceed teacher performance

AI generated summary

Compressing recommendation models using knowledge distillation

This paper proposes a multi-layer knowledge distillation framework to compress large recommendation models into small, efficient ones without sacrificing accuracy. It decomposes models into input, intermediate, and output layers, then improves distillation in each: utilizing teacher knowledge of similar entities in the input layer, reducing inconsistency in the intermediate layer, and constructing soft labels using teacher and student predictions in the output layer. Extensive experiments demonstrate it enables students to match or exceed teacher performance with just 10% of parameters and 1/3 the latency.

Answers from this paper

Comments

No comments yet, be the first to start the conversation...

Sign up to comment on this paper

Sign Up