A groundbreaking digital preservation project has created the most comprehensive 3D map of Alcatraz Island ever produced, combining cutting-edge drone technology, artificial intelligence, and reality capture systems to document one of America’s most notorious historical sites. Pete Kelsey, founder of VCTO Labs, presented the remarkable achievements of this ambitious project at Intergeo 2025 in Frankfurt, Germany, where he detailed how his international team of over 20 volunteers completed what represents one of the largest-scale digital twin projects of an American landmark.
The project, completed in December 2023 in partnership with the National Park Service, represents a new frontier in cultural heritage preservation. During his presentation, Kelsey discussed the technical challenges and groundbreaking conclusions from what he described as one of the most comprehensive conservation projects ever conducted in the United States. Rising sea levels, seismic activity from the San Andreas Fault, and the impact of over one million annual visitors have created pressing concerns for the 22.5-acre island’s structural integrity.
Living and Working in America’s Most Famous Prison
Kelsey’s team spent three weeks living in the actual prison cells on Alcatraz, working around the clock to capture four terabytes of data using terrestrial laser scanning, drone-based LiDAR, multispectral scanning, photogrammetry, and robotic systems. The unusual living arrangements became part of the project’s unique character.
“I was in D-block, third tier, cell number 31. Home sweet home!” joked Kelsey during his presentation. Despite concerns about the island’s reported paranormal activity, Kelsey said his sleep was excellent, though he had to relocate once when a retired police officer on the team’s snoring became disruptive. His unnamed colleague’s temporary move to the mugshot room under the former hospital proved short-lived due to unexplained sounds from upstairs, including what sounded like a piano player and crowds of 50-100 people.
The harsh working conditions extended beyond ghostly encounters. The team had to bring all their own supplies, including sleeping bags and food, while sharing a single bathroom and brushing their teeth at drinking fountains. Working primarily during the early morning hours before tourists arrived, the team had only two to three-hour windows to capture data without disrupting the visitor experience.
Advanced Technology Arsenal
The project’s success depended on an unprecedented combination of reality capture technologies. With a background in construction and surveying, Kelsey emphasized the importance of establishing an advanced survey control network that achieved 6mm precision. This baseline made the otherwise nearly impossible task of combining and reconciling data from different sources manageable, creating a foundation from which the team could work.
“Starting with a survey control network helped a lot – there is no endgame without that,” Kelsey explained.
Drone Technology at Work
The project’s aerial data collection represented one of the most sophisticated multi-platform drone operations ever conducted at a National Park Service site, where flights are usually banned. The team deployed multiple specialized unmanned aerial vehicles, each selected for specific mission requirements across Alcatraz’s diverse terrain and structural challenges.
The Inspired Flight 1200i served as the primary heavy-lift platform, capable of carrying larger sensor payloads for extended flight times around the island’s perimeter and elevated structures. For precision mapping of the island’s complex topography, the team utilized the EagleNXT eBee fixed-wing drone, which provided efficient coverage of large areas while maintaining the accuracy required for survey-grade documentation.
Perhaps most critical to the project’s success was the deployment of the Elios 3 collision-resistant drone for confined interior spaces. This specialized platform allowed the team to safely navigate the narrow corridors, cells, and utility areas within the prison buildings where traditional drones would have been impossible to operate. The Elios 3’s protective cage and advanced collision-avoidance systems enabled detailed scanning of inaccessible areas that had never been comprehensively documented.
ESRI played a crucial role in the Alcatraz project, providing both photogrammetry expertise and advanced processing capabilities through their ArcGIS Reality suite of software products. The photogrammetry component represented one of three critical aerial data collection flights, alongside the LiDAR and multispectral scanning operations. ESRI’s contribution was essential for creating the photorealistic texture mapping that would later be integrated with the LiDAR point cloud data to produce the final digital twin.
Phoenix LiDAR Systems provided the RANGER-U120 high-precision, long-range LiDAR scanner, which became the cornerstone of the aerial mapping operations. This survey-grade system delivered the centimeter-level accuracy required for creating baseline documentation that could support decades of future monitoring. The RANGER-U120’s ability to penetrate vegetation and capture precise measurements of deteriorating concrete and steel structures proved essential for documenting the island’s current structural condition.
The aerial operations required careful coordination with Federal Aviation Administration regulations and National Park Service requirements, particularly given Alcatraz’s location in restricted San Francisco Bay airspace. Flight operations were further complicated by the island’s seabird nesting seasons, which limited access to certain areas during critical breeding periods. The team had to work within narrow time windows, typically during early morning hours before tourist operations began, maximizing their brief opportunities to collect comprehensive aerial data without disrupting visitor experiences.
For areas with environmental hazards containing lead and asbestos, (where the use of drones risked spreading toxic dust) the team deployed a Boston Dynamics robot dog nicknamed “Spot”, equipped with Emesent’s Hovermap LiDAR system. This robotic solution prevented human exposure to toxic materials while maintaining data collection capabilities. The Hovermap system could capture continuous scans at Spot’s maximum walking pace, making it significantly faster than conventional terrestrial laser scanning methods.
RIEGL USA contributed terrestrial laser scanning equipment, including the VZ-600i for interior scans and the VZ-2000i for long-range exterior scanning capable of reaching 2.5 kilometers distance, capturing portions of the San Francisco coastline.
Processing Power Challenges
One of the project’s most significant technical challenges came from Alcatraz’s limited internet connectivity, which ruled out cloud-based data processing. The solution came from AMD, which provided a prototype workstation equipped with a 96-core Threadripper PRO processor and half a terabyte of RAM.
“The office on Alcatraz was really cold, but that machine kept us warm,” Kelsey noted.
Epic Games Integration and Virtual Reality Applications
Epic Games played a significant role in creating the final digital twin experience using their RealityScan, Twinmotion, and Unreal Engine technologies. The RealityScan 2.0 software was used to generate a detailed digital replica from aerial LiDAR, photos, and Bentley’s Cesium tiles, while Unreal Engine provided the framework for creating immersive, interactive experiences.
The Epic Games integration enables multiple applications beyond simple documentation. In future, the model could support virtual tourism capabilities, allowing visitors to explore areas normally restricted to the public. “What about all the people who want to go behind the velvet ropes? With a pair of goggles, I can show that world to you,” Kelsey explained.
Historical Discoveries and Documentation
The comprehensive scanning revealed previously unknown historical features and documented famous elements of Alcatraz’s past. The team successfully mapped the complete escape route used in the infamous 1962 breakout by Frank Morris and brothers Clarence and John Anglin. Their scans captured the upstairs workshop where the escapees built their rafts and the routes they used to reach the water.
Multi-spectral scanning techniques revealed the remains of a helipad added during the 1960s, uncovering island history that even National Park Service staff had forgotten. The team also documented graffiti throughout the facility, including notes written by guards and images created by rioting inmates, though Kelsey noted wryly that some prisoner artwork was too explicit for public display.
Perhaps most significantly, the project revealed the original Civil War-era sallyport, or secured entrance, on the eastern side of the island – a feature that had not been visible for over a century. This discovery provided new insights into Alcatraz’s military history before its famous prison years.
Conservation and Climate Change Applications
The digital twin serves multiple conservation purposes, providing baseline documentation to monitor the effects of climate change, seismic activity, and visitor impact on deteriorating structures. The National Park Service can now compare future surveys against this comprehensive dataset to track changes over time.
“At the very least, we’ve created a baseline data set that future surveys can be based on. Whether it’s sea level rise or erosion, or San Francisco’s seismic activity, we’ll be able to track the changes and be a part of the future of the island,” Kelsey explained.
Future Revenue and Educational Applications of Digital Twins
Beyond conservation, the digital twin could create new revenue opportunities for the National Park Service through virtual tourism and educational applications. The technology enables immersive experiences for people unable to visit the island physically and provides educational institutions with unprecedented access to historical sites.
Kelsey envisions the project contributing to his “BHAG” (Big Hairy Audacious Goal): creating the world’s first virtual national park. The filming and virtual reality technology could become a significant revenue driver for the park service while expanding access to historical education.
“Reality capture is educating the client about the asset they now have, and how it could be used for anything, from maintenance to licensing,” Kelsey noted. The comprehensive documentation also supports maintenance planning by providing detailed structural information for preservation specialists.
Technical Lessons and Industry Impact
The Alcatraz project demonstrated several key principles for large-scale reality capture projects. Kelsey emphasized the importance of thorough planning, noting that nine months of preparation preceded the three-week data collection phase. Obtaining permits for drone operations in a national park and coordinating around seabird nesting seasons required extensive advance work.
The project’s success depended heavily on collaboration between multiple technology companies and volunteers who contributed equipment and expertise. Phoenix LiDAR Systems, Emesent, RIEGL USA, Boston Dynamics, AMD, Epic Games, and numerous other partners provided essential components of the comprehensive solution.
For drone professionals planning similar projects, Kelsey recommended starting with a precise survey control network and calling on experienced partners. “Having done large strategic projects like this before, I had a sense of how to do it. I relied on people a lot smarter than me – I called in every favor anyone ever owed me,” he explained.
Advancing Digital Preservation Standards
The Alcatraz digital twin project sets new standards for cultural heritage preservation, demonstrating how advanced geospatial technologies can protect America’s historical landmarks for future generations. As rising sea levels and extreme weather events increasingly threaten coastal historical sites, Kelsey’s work provides a replicable model for comprehensive digital documentation.
“We make the invisible visible. We make people see perspectives and points of view that have never been seen before,” Kelsey said, referring to reality capture professionals. The project’s combination of multiple technologies, robotic systems, and advanced processing capabilities creates a template for preserving other significant historical sites facing similar environmental and structural challenges.
The comprehensive dataset now serves as a permanent record of Alcatraz Island’s condition in 2023, ensuring that future researchers and conservationists will have precise documentation of this iconic American landmark regardless of how climate change and time may alter its physical structure.
More information on VCTO Labs and Pete’s other projects are available here.
Read More
Ian McNabb is a journalist focusing on drone technology and lifestyle content at Dronelife. He is based between Boston and NH and, when not writing, enjoys hiking and Boston area sports.
