• Patrick Olivier
  • (Newcastle University)

  • Jesse Hoey
  • (University of Waterloo)
Multi-modal Recognition of Food Preparation Activities for Situational Support Systems

In aging societies the ratio of people needing personal care to those able to provide care steadily increases. Therefore, the role of technology to develop new assistive solutions gains in importance. We investigate new methods for activity recognition based on accelerometers and computer vision and their applicability to situational support systems. Our target domain - food preparation activities - is one of the most challenging as it involves a large number of different objects, gestures and cognitive capabilities. In this context the use of computer vision techniques is particularly promising because food cannot be equipped with RFID tags or accelerometers and cameras may identify important properties of complex interactions.

As we find vision and accelerometer data complement each other we combine optical sensors with three-axis accelerometers embedded in kitchen utensils. With combined 2D and depth cameras becoming increasingly available at a reasonable price we incorporate this type of sensor in our experiments in order to investigate the potential of depth information for action recognition in the kitchen scenario. In summary, the central goal of this project is to investigate methods of computer vision and fusion with accelerometer data in the context of recognizing food preparation activities for situational support systems.

This research receives funding from the RCUK Digital Economy Research Hub EP/G066019/1 and SIDE: Social Inclusion through the Digitial Economy.


50 Salads: Activity recognition from RGBD-video and accelerometers attached to kitchen objects.
Accelerometer Localization: Quantitative evaluation of accelerometer localization algorithms from RGBD-video and accelerometers attached to kitchen utensils; includes ground-truth locations of all accelerometers in each frame.

[4] S. Stein, S. J. McKenna,
"User-adaptive Models for Recognizing Food Preparation Activities", in Proceedings of the 21st ACM International Conference on Multimedia, 5th International Workshop on Multimedia for Cooking & Eating Activities (CEA'13), Barcelona, Spain, ACM. [bib] [pdf]
[3] S. Stein, S. J. McKenna,
"Combining Embedded Accelerometers with Computer Vision for Recognizing Food Preparation Activities", in Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013), Zurich, Switzerland (In Press). [bib] [pdf]
[2] S. Stein, S. J. McKenna,
"Towards recognizing food preparation activities in situational support systems", in Digital Futures 2012 (3rd Annual Digital Economy All Hands Conference), Aberdeen. [bib] [pdf]
[1] S. Stein, S. J. McKenna,
"Accelerometer Localization in the View of A Stationary Camera", in Proceedings of the Ninth Conference on Computer and Robot Vision (CRV'12), Toronto, Ontario, Canada, IEEE, pp. 109 - 116. [bib] [pdf] [doi]
Powered by bibtexbrowser