Research News

Augmented reality (AR) technology has long fascinated both the scientific community and the general public, remaining a staple of modern science fiction for decades.

In the pursuit of advanced AR assistants – ones that can guide people through intricate surgeries or everyday food preparation, for example – a research team from NYU Tandon School of Engineering has introduced Augmented Reality Guidance and User-Modeling System, or ARGUS. 

An interactive visual analytics tool, ARGUS is engineered to support the development of intelligent AR assistants that can run on devices like Microsoft HoloLens 2 or MagicLeap. It enables developers to collect and analyze data, model how people perform tasks, and find and fix problems in the AR assistants they are building.

Claudio Silva, NYU Tandon Institute Professor of Computer Science and Engineering and Professor of Data Science at the NYU Center for Data Science, leads the research team that will present its paper on ARGUS at IEEE VIS 2023 on October 26, 2023, in Melbourne Australia.  The paper received Honorable Mention in that event’s Best Paper Awards.  

“Imagine you’re developing an AR AI assistant to help home cooks prepare meals,” said Silva. “Using ARGUS, a developer can monitor a cook working with the ingredients, so they can assess how well the AI is performing in understanding the environment and user actions. Also, how the system is providing relevant instructions and feedback to the user.  It is meant to be used by developers of such AR systems.”

ARGUS works in two modes: online and offline. 

The online mode is for real-time monitoring and debugging while an AR system is in use. It lets developers see what the AR system sees and how it's interpreting the environment and user actions. They can also adjust settings and record data for later analysis. 

The offline mode is for analyzing historical data generated by the AR system. It provides tools to explore and visualize this data, helping developers understand how the system behaved in the past. 

ARGUS’ offline mode comprises three key components: the Data Manager, which helps users organize and filter AR session data; the Spatial View, providing a 3D visualization of spatial interactions in the AR environment; and the Temporal View, which focuses on the temporal progression of actions and objects during AR sessions. These components collectively facilitate comprehensive data analysis and debugging.

“ARGUS is unique in its ability to provide comprehensive real-time monitoring and retrospective analysis of complex multimodal data in the development of systems,” said Silva. “Its integration of spatial and temporal visualization tools sets it apart as a solution for improving intelligent assistive AR systems, offering capabilities not found together in other tools.”

ARGUS is open source and available on GitHub under VIDA-NYU. The work is supported by the DARPA Perceptually-enabled Task Guidance (PTG) program.

ARGUS: Visualization of AI-Assisted Task Guidance in AR

Peripheral arterial disease (PAD) is a vascular disease that is caused by clogging of the arteries due to plaque. The lower legs and feet are often impacted, and symptoms include pain, numbness, difficulty walking, and non-healing wounds.

For many patients, wounds continue not to heal even after they have undergone a surgical revascularization procedure designed to unclog the affected arteries, necessitating another intervention. Determining whether wounds will heal must be done as soon as possible after the first intervention to reduce the duration of a patient’s pain and increase the likelihood of a good outcome.

Currently, the most common methods for monitoring PAD progression (and related wound healing) are the ankle-brachial index (ABI) and ultrasound imaging. The ABI uses the ratio of systolic blood pressure measurements from arteries in the lower extremities to the systolic blood pressure measurement from the brachial artery in the arm. PAD patients generally have pressure ratios that are below a certain threshold (often 0.9). The ABI has low accuracy when monitoring vasculature in diabetic patients and ultrasound imaging has low accuracy when monitoring smaller arteries such as those in the feet. Unfortunately, PAD is often comorbid with diabetes and the affected arteries are commonly in the feet.

To address the existing limitations of the current technology, a team from NYU Tandon School of Engineering’s department of Biomedical Engineering, including lead author Nisha Maheshwari from Andreas Hielscher's Clinical Biophotonics Laboratory, developed optical imaging technology to assist physicians with monitoring the healing of lower limb ulcers after a surgical intervention.

Dynamic vascular optical spectroscopy (DVOS) is an optical imaging technology that uses light in the red and near-infrared ranges to determine characteristics of blood flow through arteries. In the paper “Postintervention monitoring of peripheral arterial disease wound healing using dynamic vascular optical spectroscopy,” published in the Journal of Biomedical Optics, the team used their DVOS system to monitor a set of 14 patients with PAD that underwent a surgical revascularization procedure. Of these patients, five needed a second intervention due to the persistence of non-healing wounds.

The team was able to correctly categorize the long-term healing and non-healing of wounds in 93% of this patient population within only one month after each patient’s initial intervention. The method outperformed the gold standard methods of ultrasound and ABI. These findings suggest that the DVOS may be able to assist physicians in improving patient outcomes and reducing long-term pain by determining wound outcome earlier than existing technology can.

The authors would like to thank the patients who volunteered for this study for their time and participation. This work was supported in part by the National Heart, Lung, and Blood Institute (Grant No. NHLBI-1R01-HL115336); Wallace H. Coulter Foundation; Society of Vascular Surgery; Columbia University Fu Foundation School of Engineering and Applied Science; and New York University Tandon School of Engineering.

"Maheshwari N, Marone A, Altoé M, Kim SHK, Bajakian DR, Hielscher AH. Postintervention monitoring of peripheral arterial disease wound healing using dynamic vascular optical spectroscopy. J Biomed Opt. 2022 Dec;27(12):125002. doi: 10.1117/1.JBO.27.12.125002. Epub 2022 Dec 24. PMID: 36582192; PMCID: PMC9789744."

NYU Tandon School of Engineering researchers developed a new artificial intelligence technique to change a person’s apparent age in images while maintaining their unique identifying features, a significant step forward from standard AI models that can make people look younger or older but fail to retain their individual biometric identifiers.

In a paper published in the proceedings of the IEEE International Joint Conference on Biometrics (IJCB), Sudipta Banerjee, the paper’s first author and a research assistant professor in the Computer Science and Engineering (CSE) Department, and colleagues trained a type of generative AI model – a latent diffusion model – to “know” how to perform identity-retaining age transformation.

To do this, Banerjee – working with CSE PhD candidate Govind Mittal and PhD graduate Ameya Joshi, under the guidance of Chinmay Hegde, CSE associate professor and Nasir Memon, CSE professor – overcame a typical challenge in this type of work, namely assembling a large set of training data consisting of images that show individual people over many years. 

Instead, the team trained the model with a small set of images of an individual, along with a separate set of images with captions indicating the age category of the person represented:  child, teenager, young adult, middle-aged, elderly, or old. This set included images of celebrities captured throughout their lives.

The model learned the biometric characteristics that identified individuals from the first set.  The age-captioned images taught the model the relationship between images and age. The trained model could then be used to simulate aging or de-aging by specifying a target age using a text prompt.

Researchers employed a method called "DreamBooth" for editing human face images by gradually modifying them using a combination of neural network components. The method involves adding and removing noise – random variations or disturbances –  to images while considering the underlying data distribution. 

The approach utilizes text prompts and class labels to guide the image generation process, focusing on maintaining identity-specific details and overall image quality. Various loss functions are employed to fine-tune the neural network model, and the method's effectiveness is demonstrated through experiments on generating human face images with age-related changes and contextual variations.

The researchers tested their method against other existing age-modification methods, by having 26 volunteers match the generated image with an actual image of that person, and with ArcFace, a facial recognition algorithm. They found their method outperformed other methods, with a decrease of up to 44% in the rate of incorrect rejections.

UPDATE March 4, 2024: The data set that Faghih’s lab collected for this research is now available to the global research community on the PhysioNet platform. This dataset is unique, offering real-world insights into how common pleasures affect our physiological responses and cognitive performance.

The potential of this dataset is vast. It opens new avenues for research into the influence of everyday experiences on cognitive performance, potentially leading to smarter work environments or personalized life-enhancing strategies. Imagine tailoring your work environment with specific sounds or scents to boost productivity and creativity. By analyzing this dataset, researchers can discover patterns and connections previously unseen. This could lead to breakthroughs in understanding how to harness everyday experiences to enhance cognitive abilities. Ultimately, this research could pave the way for innovative applications in workplace productivity enhancement and educational method improvement.

“This dataset is more than a collection of data points; it is a window into the intricate relationship between daily pleasures and our brain's performance,” says Fekri Azgomi, Faghih’s former PhD student who collected this data. “As our lab, the Computational Medicine Laboratory, shares this dataset with the world, we are excited about the endless possibilities it holds for advancing our understanding of the human mind and enhancing everyday life.”

Original story below.

Listening to music and drinking coffee are the sorts of everyday pleasures that can impact a person’s brain activity in ways that improve cognitive performance, including in tasks requiring concentration and memory.

That’s a finding of a new NYU Tandon School of Engineering study involving MINDWATCH, a groundbreaking brain-monitoring technology.

Developed over the past six years by NYU Tandon's Biomedical Engineering Associate Professor Rose Faghih, MINDWATCH is an algorithm that analyzes a person's brain activity from data collected via any wearable device that can monitor electrodermal activity (EDA). This activity reflects changes in electrical conductance triggered by emotional stress, linked to sweat responses.

In this recent MINDWATCH study, published in Nature Scientific Reports, subjects wearing  skin-monitoring wristbands and brain monitoring headbands completed cognitive tests while listening to music, drinking coffee and sniffing perfumes reflecting their individual preferences. They also completed those tests without any of those stimulants. 

The MINDWATCH algorithm revealed that music and coffee measurably altered subjects’ brain arousal, essentially putting them in a physiological “state of mind” that could modulate their performance in the working memory tasks they were performing. 

Specifically, MINDWATCH determined the stimulants triggered increased “beta band” brain wave activity, a state associated with peak cognitive performance. Perfume had a modest positive effect as well, suggesting the need for further study. 

“The pandemic has impacted the mental well-being of many people across the globe and now more than ever, there is a need to seamlessly monitor the negative impact of everyday stressors on one's cognitive function,” said Faghih. “Right now MINDWATCH is still under development, but our eventual goal is that it will contribute to technology that could allow any person to monitor his or her own brain cognitive arousal in real time, detecting moments of acute stress or cognitive disengagement, for example. At those times, MINDWATCH could ‘nudge’ a person towards simple and safe interventions — perhaps listening to music  — so they could get themselves into a brain state in which they feel better and perform job or school tasks more successfully.”

The specific cognitive test used in this study — a working memory task, called the n-back test — involves presenting a sequence of stimuli (in this case, images or sounds) one by one and asking the subject to indicate whether the current stimulus matches the one presented "n" items back in the sequence. This study employed a 1-back test — the participant responded "yes" when the current stimulus is the same as the one presented one item back — and a more challenging 3-back test, asking the same for three items back.

Researchers tested three types of music - energetic and relaxing music familiar to the subject, as well as novel AI-generated music that reflected the subject’s tastes.  Consistent with prior MINDWATCH research, familiar energetic music delivered bigger performance gains — as measured by reaction times and correct answers — than relaxing music. While AI-generated music produced the biggest gains among all three, further research is needed to confirm those results.

Drinking coffee led to notable but less-pronounced performance gains than music, and perfume had the most modest gains.

Performance gains under all stimulations tended to be higher on the 3-back tests, suggesting interventions may have the most profound effect when “cognitive load” is higher.

Ongoing experimentation by the MINDWATCH team will confirm the efficacy of the technology’s ability to monitor brain activity consistently, and the general success of various interventions in modulating that brain activity. Determining a category of generally successful interventions does not mean that any individual person will find it works for them.

The research was performed as a part of Faghih’s National Science Foundation CAREER award on the Multimodal Intelligent Noninvasive brain state Decoder for Wearable AdapTive Closed-loop arcHitectures (MINDWATCH) project.  The study's diverse dataset is available to researchers, allowing additional research on the use of the safe interventions in this study to modulate brain cognitive states.

Faghih served as the senior author for this paper. Its first author is Hamid Fekri Azgomi, who earned his Ph.D. under Faghih and is now a postdoctoral scholar of neurological surgery at the University of California San Francisco School of Medicine.

Fekri Azgomi, H., F. Branco, L.R., Amin, M.R. et al. Regulation of brain cognitive states through auditory, gustatory, and olfactory stimulation with wearable monitoring. Sci Rep 13, 12399 (2023). https://doi.org/10.1038/s41598-023-37829-z

A new trip-planning app has shown encouraging results in improving navigation inside subway stations, according to a study published in IEEE Journal of Translational Engineering in Health and Medicine, promising the possibility of easier commutes for people who are blind and low-vision. 

Designed by researchers at NYU Tandon School of Engineering and NYU Grossman School of Medicine, Commute Booster routes public-transportation users through the “middle mile” — the part of a journey inside subway stations or other similar transit hubs — in addition to the “first” and “last” miles that bring travelers to and from those hubs. 

“The ‘middle mile’ often involves negotiating a complex network of underground corridors, ticket booths and subway platforms. It can be treacherous for people who cannot rely on sight,” said John-Ross Rizzo, MD, who led the research team that includes advisors from New York City’s Metropolitan Transit Authority (MTA). Rizzo is an associate professor in NYU Tandon’s Biomedical Engineering department and is on the faculty of NYU Grossman. “Most GPS-enabled navigation apps address ‘first’ and ‘last’ miles only, so they fall short of meeting the needs of blind or low-vision commuters. Commute Booster is meant to fill that gap.”

Subway signs are typically graphical or text-based, creating challenges for the visually impaired to recognize from distances and reducing their ability to be autonomous in unfamiliar environments. 

Commute Booster automatically figures out what signs a traveler will encounter along the way to a specific subway platform. Then, it uses a smartphone’s camera to recognize and interpret signs posted inside transit hubs, ignoring irrelevant ones and prompting users to follow relevant ones only.

In the recent study, researchers tested Commute Booster’s interpretation of signage from three New York City subway stations — Jay Street-Metrotech, Dekalb Avenue and Canal Street — that a traveler would encounter on a specific journey. The app proved 97 percent accurate in identifying signs relevant to reach the intended destination.  

Testing inside those three subway stations also revealed that Commute Booster could “read” signs from distances and at angles that reflect expected physical positioning of travelers. 

The Commute Booster system relies on two technological components. The first, general transit feed specification (GTFS), is a standardized way for public transportation agencies to share their transit data with developers and third-party applications. The second, optical character recognition (OCR), is technology that can translate images of text into actual editable text.

The GTFS dataset contains descriptions for locations and pathways within each subway station. Commute Booster’s algorithm uses this information to generate a comprehensive list of wayfinding signage within subway stations that users would encounter during their intended journey. The OCR functionality reads all texts presented to users in their immediate surroundings. Commute Booster’s algorithm can identify relevant navigation signs and locate the position of signs in the immediate environments. By integrating these two components, Commute Booster provides real-time feedback to users regarding the presence or absence of relevant navigation signs within the field of view of their phone camera during their journey.

Researchers plan to conduct a human subject study of Commute Booster in the near future. The app could be available for public use in the near term.

Rizzo, who was named to MTA’s board in June 2023, has a long track record of research that applies engineering solutions to challenges faced by people with disabilities, particularly those with visual disability .

In addition to Rizzo, the team involved in the Commute Booster study are NYU Tandon PhD candidate Junchi Feng; Physician-Scientist at NYU Langone’s Rusk Rehabilitation Mahya Beheshti;  MTA Senior Innovation Strategist Mira Philipson; MTA Senior Accessibility Officer Yuvraj Ramsaywack; and NYU Tandon Institute Professor Maurizio Porfiri. 

This research was supported by the National Science Foundation, the National Eye Institute and Fogarty International Center, as well as by the U.S. Department of Defense.

Researchers at NYU Tandon School of Engineering’s C2SMART Center have developed an algorithm to plan the most efficient routes for modular vehicle (MV) fleets — specially-designed vehicles that attach and detach from one another as they move people around cities — removing a significant obstacle to making this type of transportation system a reality. 

In a paper published in Transportation Research Part C: Emerging Technologies, the researchers employ a mathematical model called MILP (Mixed Integer Linear Programming) to optimize the service time for the passengers and the travel cost for the vehicles in an MV system. The model factors in passenger pickups and deliveries, en-route transfers, and variable capacity of the MVs to identify the best routes and schedules for the attachments and separations of the vehicles. 

Conventional mass transit and demand-responsive transportation systems can face challenges accommodating fluctuations in traveler demand, leading to long travel times, energy inefficiencies, traffic congestion and financial waste.  

Low-capacity vehicles like vans may be slow and overcrowded in peak times. High-capacity vehicles like buses may be largely unoccupied when demand is low. On-demand services like paratransit often deliver only one passenger at a time, making them expensive to operate.

MVs offer a flexible and efficient alternative. The independent vehicles in MV fleets can connect while in motion, creating platoons that travel as one unit until the vehicles detach. According to research lead Joseph Chow,  Institute Associate Professor in the Department of Civil & Urban Engineering and the Deputy Director of C2SMART, MVs can move people faster, with less energy consumption and operational expenses than many conventional systems.

“MVs offer a promising alternative to move people more efficiently in certain situations,” said Chow, who collaborated on the research with NYU Tandon Ph.D. student Zhexi Fu. “Imagine, for instance, employees at the same company. The individual vehicles could pick up people who live within similar enclaves, and join together in a platoon to deliver the entire group to its workplace. MVs also have significant potential to improve on-demand transportation that delivers people door-to-door, including those that serve people with disabilities.”

Currently, no city has an MV system in use, although Next Transportation Systems is piloting a MV test in Dubai now. According to Chow, the inability to track and route MV fleets has been a significant roadblock to potential real-world adoption. To build its routing model, the C2SMART team used the Anaheim network, a traffic simulation of Anaheim, California.

The research on MV routing is the latest in a long series of studies Chow has conducted around urban mobility. Among his previous studies include examinations of Dial-a-Ride programs,  e-scooter usage, and urban bus networks. Chow’s new research also advances the mission of C2SMART, a U.S. Department of Transportation (US DOT) Tier 1 University Transportation Center (UTC) designated to address the US DOT priority area of Congestion Reduction. 

A new paper from researchers at NYU Tandon School of Engineering and the NYU Grossman School of Medicine explores how interdisciplinary programs connecting medical and engineering education may foster innovation and prepare students in both disciplines for more successful careers. 

The paper, published in the Technology and Innovation journal of the National Academy of Inventors, describes initiatives at NYU as a case study, along with similar programs at Johns Hopkins University, Stanford University, Harvard and Massachusetts Institute of Technology.

NYU Tandon and Grossman have partnered on educational programs for about a decade. But for most of this time, the skill-sharing only went in one direction, explained lead author John-Ross Rizzo, a rehabilitation medicine specialist and professor at both schools. “It dawned on us that we spend a ton of time bringing engineers to the medical school, but almost zero time trying to get our doctors immersed in the engineering world,” Rizzo said.

By bringing medical students to the engineering field, as NYU has done in recent years, educators can enable a shared understanding of engineering concepts that contributes to more effective problem-solving, Rizzo and his colleagues argue. Clinicians and engineers are more capable of collaboration if they speak each other’s languages; new innovations that result from these partnerships may be better set up for long-term success.

Rizzo compared this interdisciplinary learning to earning belts in martial arts. A medical student might not become a “black belt in computer science,” but might learn enough for a “yellow belt” — a lower level of understanding, but enough to enable collaboration with the true experts. “We’re creating a smarter generation of students,” Rizzo said.

One way NYU students may gain this expertise is through participation in the NYU HealthTech Transformer Challenge, which pairs engineers and clinicians to work on “healthcare’s most pressing problems.” Finalists from the program have won funding from NYU and other sources to pursue their ideas at new startups. Other challenges and grant-funded research projects at NYU Tandon and Grossman have allowed graduate students to receive co-advising from engineering and medical professors.

The researchers also discussed barriers to setting up these interdisciplinary programs. Early initiatives may require extensive effort, including dedicated advocacy to bring different school administrators on board. It may be especially tough to convince medical school leaders to devote student time to engineering work outside their typical course load. Part of the challenge is a lack of data: while NYU and similar programs have produced some clear success stories among individual students and startups, universities are not tracking their results in a comprehensive manner.

In the new paper, Rizzo and colleagues share lessons from NYU’s leadership in this interdisciplinary space and from programs at other institutions. The findings may provide inspiration for more universities to consider connecting medicine and engineering education. “I think this is a trend we’ll hear more about over the next decade,” Rizzo said.