From Watching to Cooking: Interactive Assistance for Short-Form Video Recipes

Liu, X., Zhang, T., Au Yeung, C., Zhu, B., Li, F., Tang, A., and Ma, D. (2026). From Watching to Cooking: Interactive Assistance for Short-Form Video Recipes. In CCF Transactions on Pervasive Computing and Interaction.

Abstract

With the surge of short-form video platforms, short-form video recipes have become an increasingly popular medium for home cooking, owing to their rich visual demonstrations. However, the short-form format poses practical challenges for in-situ cooking assistance, particularly for novice cooks, including difficulty locating specific steps, mismatched pacing, and a lack of contextual support during execution. To address these issues, we introduce CHEF, a Multimodal Large Language Model (MLLM)-powered assistant designed to enhance the short-form video-guided cooking experience through intelligent, real-time support. CHEF integrates three key capabilities: (1) video recipe understanding, which segments instructional videos, generates structured text-based recipes, and produces a step dependency graph; (2) automatic video navigation, which dynamically adjusts video playback based on the user's real-time progress; and (3) contextual assistance, which lets users ask questions and proactively offers useful hints and tips based on the current cooking context. We evaluate CHEF through a technical analysis on the YouCook2 dataset and an in-kitchen user experience study with real cooking tasks. The results indicate that CHEF effectively generates accurate, well-structured text-based recipes, while participants generally perceive its contextual assistance as useful during cooking, demonstrating its potential for practical cooking assistance.

Materials

DOI (https://doi.org/10.1007/s42486-026-00274-8)

Keywords

Short-form video recipes, Cooking support, Multimodal large language models, Video recipe understanding, Video navigation

BibTeX

@article{liu2026chef,
  type = {journal},
  videourl = {},
  pdfurl = {},
  abstract = {With the surge of short-form video platforms, short-form video recipes have become an increasingly popular medium for home cooking, owing to their rich visual demonstrations. However, the short-form format poses practical challenges for in-situ cooking assistance, particularly for novice cooks, including difficulty locating specific steps, mismatched pacing, and a lack of contextual support during execution. To address these issues, we introduce CHEF, a Multimodal Large Language Model (MLLM)-powered assistant designed to enhance the short-form video-guided cooking experience through intelligent, real-time support. CHEF integrates three key capabilities: (1) video recipe understanding, which segments instructional videos, generates structured text-based recipes, and produces a step dependency graph; (2) automatic video navigation, which dynamically adjusts video playback based on the user's real-time progress; and (3) contextual assistance, which lets users ask questions and proactively offers useful hints and tips based on the current cooking context. We evaluate CHEF through a technical analysis on the YouCook2 dataset and an in-kitchen user experience study with real cooking tasks. The results indicate that CHEF effectively generates accurate, well-structured text-based recipes, while participants generally perceive its contextual assistance as useful during cooking, demonstrating its potential for practical cooking assistance.},
  keywords = {Short-form video recipes, Cooking support, Multimodal large language models, Video recipe understanding, Video navigation},
  doi = {10.1007/s42486-026-00274-8},
  publisher = {Springer},
  year = {2026},
  journal = {CCF Transactions on Pervasive Computing and Interaction},
  title = {From Watching to Cooking: Interactive Assistance for Short-Form Video Recipes},
  author = {Liu, Xiaochen and Zhang, Tianyi and Au Yeung, Colin M. and Zhu, Bin and Li, Fan and Tang, Anthony and Ma, Dong},
}