• Dr. Timothy Smith

Secrets Hidden in Videos Revealed with Mathematics

Photo Source: Pixnio

Digital video, cheap and available on every smartphone in the world, makes it easy to capture all of life’s moments from birthdays and anniversaries to DIY instructions on how to build a fence or bake a cake. Obviously, video can capture the smiles and special moments of life, but mathematics applied to video has recently demonstrated our little digital movies contain much, much more than meets the eye. Video holds many truths not visible through simple playback—secrets unavailable to the naked eye even with painstaking slow motion. A mathematical technique called Eulerian Video Magnification (EVM) amplifies subtle variations in video images to reveal invisible variations in color and motion. EVM exposes truths such as someone’s heartbeat through changes in face color or the deformations of glass bending in response to sound waves.

Researchers at MIT’s Computer Science & Artificial Intelligence Laboratory (CSAIL), Hao-Yu Wu, Michael Rubenstein, and others, published a paper titled, “Eulerian Video Magnification for Revealing Subtle Changes in the World,” published by the Association for Computing Machinery. The paper details the techniques for video magnification. Although a video looks like a smooth image with curved lines and sharp transitions between light and shadow, close examination shows that video pictures consist of little squares of color called pixels. A video consists of many pictures shown rapidly one after the other. Even a video of something that looks steady will have variations at the pixel level. A video of a person, a house, or anything else at the pixel level will subtly change color. Such variations capture small but real variations in the subject of the video not detectable by the human eye. In short, Eulerian video magnification takes advantage of small changes in the color of the tiny dots to allow others to find things that they wouldn’t have by simply zooming in.

Using mathematics, researchers can break down videos to the individual pixel and then amplify the subtle variations over each frame of the video. After amplifying the variations, the video gets reassembled with the subtle changes in color exaggerated at different frequencies or rates. Essentially, if a pixel is flipping between two colors such as red and blue every second, the video magnification will make the variation appear much more clearly.

Although people cannot see it, the human face actually changes color with every heartbeat. When the heart pushes blood through the arteries close to the surface of the skin, the skin will briefly turn red. The change happens too quickly and subtly for detection with regular vision, but with Eulerian video magnification, the periodic reddening of the face appears crystal clear and reveals someone’s pulse rate without physical contact. Using EVM, one can find someone’s heart rate through EVM and can use it to detect diseases and disorders early.

EVM does not require expensive equipment. Italian researchers including Ennio Gambi and Susanna Spinsante demonstrated the application Eulerian video magnification for heart rate detection using a commonly available system. In “Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to Wearable Devices,” published in Sensors (Basel) in 2017. Using a smartphone and a computer, Stefan Williams and others in an article titled, ”A Smartphone Camera Reveals an ‘Invisible’ Parkinsonian tremor: a Potential Pre-Motor Biomarker” in the Journal of Neurology demonstrated a powerful use of EVM in early Parkinson’s diagnosis.

The universal availability of inexpensive video cameras on smartphones gives anyone with a phone the ability to capture and share videos wherever they are. Unbeknown to most, all these videos contain information hidden to the naked eye. Slight variations in color and motion get captured in every video, and these small variations become visible to the naked eye with a mathematical amplification called Eulerian video magnification. Eulerian video magnification can detect someone’s pulse without touching them and reveal invisible tremors, indicating early Parkinson’s Disease or epilepsy. In a way, a collection of family videos over the years can be a valuable resource for identifying early signs of degenerative diseases and possibly prompt early intervention with low-cost cameras and computation.

Dr. Smith’s career in scientific and information research spans the areas of bioinformatics, artificial intelligence, toxicology, and chemistry. He has published a number of peer-reviewed scientific papers. He has worked over the past seventeen years developing advanced analytics, machine learning, and knowledge management tools to enable research and support high level decision making. Tim completed his Ph.D. in Toxicology at Cornell University and a Bachelor of Science in chemistry from the University of Washington.

You can buy his book on Amazon in paperback and in kindle format here.