Final Design
Digital Twin
Digital twins of NYC serve as a platform for studying driver-pedestrian interactions in a pseudo-naturalistic setting.
Flying through a digital twin of Flushing, Queens in Unity.
Aerial snapshot of digital twin of subsection of Flushing, Queens. A complete digital model which preserves the unique cultural character of this locale.
Digital twin of subsection neighboring Battery Park, Manhattan. A finalized replica of Lower Manhattan encompassing the iconic skyscrapers and urban landscapes that define the district.
First-person pedestrian exploration in VR digital twin. A sequence of four images showing a research participant navigating the VR digital twin of Flushing using an Oculus headset. Scan the QR code to experience the digital twin through the first-person view of a pedestrian.
Background
In the summer of 2023, I spent 10 weeks with the Future Autonomous Research (FAR) Lab at Cornell Tech where I developed digital twins of New York City to serve as a platform for studying driver-pedestrian interactions in a pseudo-naturalistic setting. This "high fidelity prototype" of NYC provides a sense of realism in which driver-pedestrian interactions can be studied in a safe and controlled virtual reality environment
Role
Human-Computer Interaction
Research Intern
Time
10 weeks, 2023
Team
Srijan Srivatsa (Intern)
Adam Yu Zheng Zhang (PhD Mentor)
Debargha (Dave) Dey (Postdoc Mentor)
Wendy Ju (Principal Investigator)
G O A L
​Develop digital twins of three areas in NYC which serve as a platform for studying driver-pedestrian interactions in a pseudo-naturalistic setting.
To ascertain the appropriate, culturally-mapped driving behavior for implementing AVs internationally, we need to subjectively and empirically understand the interaction patterns that unfold between road users in specific locales. New York City is our first location of digital twin development.
💡 Opportunity
-
We instinctively and trivially understand that traffic interactions differ across cultures, countries, or even cities within the same country.
-
In the future deployment of autonomous vehicles (AVs), it is critical to ensure that they can function appropriately as social agents in cross-cultural traffic scenarios, which points to the hypothesis that one universal driving behavior will not suffice for global deployment of AVs.
-
To understand road user interactions in different cultural traffic contexts, we observe two types of communication:
-
-
Explicit, e.g. flashing headlights, honking, verbal communication
-
Implicit, e.g. body language, movement patterns, vehicle kinematics
-
-
Process
Research
Gathering Naturalistic Video Data
We started off the project by scouting for potential intersections in various NYC neighborhoods. The architecture and geography of the city is incredibly rich and diverse, from the luxury boutiques and high-end shops of Madison Avenue to wide array of cultural experiences in Flushing Chinatown.
​
However, our team wanted to find roads/areas that parallel other places in the world. Specifically, we looked for bidirectional roads with one-lane each way that could act as cross-cultural benchmarks between countries.
​
For the first 2-3 weeks of the projects, my fellow intern Srijan and I scouted various locations in NYC that matched this description. We stood at these characteristic intersections with mounted GoPros, collecting video footage of pedestrians and vehicles.
Identification of Ideal NYC Subsections
After collecting and reviewing the GoPro video footage, we selected three urban areas across NYC to recreate for our digital twin.
Schematics highlighting selected districts for digital twin modeling. A map of NYC’s Queens (above) and Manhattan borough (right), highlighting the regions replicated in the digital twin models.
Base Model
Context Buildings and Street Systems
To kick off digital twinning, I imported the base model of street systems and context buildings from NYC Open Data Planning website into Rhino3D, a 3D modeling software. Adam helped me model the street and sidewalk geometry.
Simulating the base model of buildings in Flushing. Aerial view in Rhino3D displaying the base structure of buildings before texture application.
Simulating Battery Park’s street and sidewalk system. Aerial view in Rhino3D showcasing a digital representation of street + sidewalk infrastructure, with detailed road and sidewalk textures.
Details
3D Extrusions and Texture Mapping
After importing and cleaning the base models, I created physical details for a more realistic feel for the user when walking around the VR environment. I modeled physical extrusions on building externalities (see left figure) and applied textures from Google Maps street view onto bare buildings (see right figure).
Work-in-progress snapshot showcasing a partially constructed building with extrusions (left). Aerial view featuring textured surfaces on half the buildings, while others await textures, illustrating ongoing efforts in refining the virtual landscape (right).
Integration with VR
Adapt Digital Twin for Unity + VR Headsets
The final step was to import the .fbx digital twin models from Rhino3D into Unity, a game development platform, which in turn could be connected to Oculus Quest VR headsets. After importing the Rhino3D models into Unity, I populated Unity scene with street objects (e.g. lampposts, trees, cars, trash cans, etc.) for the most realistic feel. Finally, my team and I integrated the digital twin environment into the Oculus headset for an immersive pedestrian simulation experience.
Experimenting with a beta version of the digital twin.
First-person POV walking inside the digital twin.
Next Steps
(1) Integrating Digital Twin into Unity Environment
We will be incorporating the 3D models of the city into Unity so that they can be experienced live in VR. Objects (e.g. trash cans, light poles, trees, etc.) will be added to the scene to enhance the natural feel of a city landscape.
(2) Deploying Pedestrian and Driver Simulators
Both the driver and the pedestrian will navigate the digital twin in VR simultaneously.
(3) Running Participant Studies Globally
NYC participants will first interact within a virtual environment they are familiar with (our digital twin).
Eventually, this study will be replicated with similar intersections internationally with our partners in Israel and other universities who will produce their own local digital twins.
Participants will later interact in non-native digital twin environments (i.e. NYC participants experiencing the Israel environment and vice versa).
(4) Conclusion
The insights from this international study will elucidate cross-cultural similarities and differences in vehicle-pedestrian interaction, guiding the design of future AV behavior.
Reflection
All in all, this summer research project was not only my first introduction to 3D architectural modeling, but also a lesson in creating for the user. Attention-to-detail was of utmost importance. Tiny details like a disproportionate skew of brick wallpaper could disrupt the entire flow of the user's experience. I stayed mindful about creating as realistic of an experience for study participants walking around in the VR environment. I gained confidence in design intention and felt true project ownership.
​A major turning point for me was learning to take criticism. It's hard to let go of the temptation to defend yourself. A fellow intern taught me that, in light of time constraints, I would have to compromise some detail and accuracy of the facades. At first, I was a little irritated with these proposals, but quickly, I learned not to take things personally, to be so defensive; they're not attacking me, or attacking anyone, but rather, helping me improve my workflow and the project itself. It's easy to get defensive and take criticism to heart; learning to detach your emotions from your work is important to preserving your integrity.
​
Moreover, another pivotal lesson was realized in the power of sharing ideas and power stemming from people. From that one aforementioned conversation, new discussions started because I started asking for help. Another intern helped me with perspective alignment by finding a software. And another shared knowledge that we can script and code in Photoshop to automate the process of aligning perspectives. My principal investigator brought up the idea to ask for help in the Discord channel in order to recruit people to help with digital modeling. I feel so grateful for all the help and new ideas I was offered. People are willing to help and always willing to share their knowledge. Just ask.
I'll miss this view of Manhattan from the office!
Fellow interns and I on an ice cream and boba run (: