Zone for AI to look for trespassing at railroad crossing

Research Spotlight: Exploring the Use of Artificial Intelligence to Improve Railroad Safety

Partnering with the Federal Railroad Administration, New Jersey Transit and New Jersey Department of Transportation (NJDOT), a research team at Rutgers University is using artificial intelligence (AI) techniques to analyze rail crossing safety issues. Utilizing closed-circuit television (CCTV) cameras installed at rail crossings, a team of Rutgers researchers, Asim Zaman, Xiang Liu, Zhipeng Zhang, and Jinxuan Xu, have developed and refined an AI-aided framework for detection of railroad trespassing events to identify the behavior of trespassers and capture video of infractions.  The system uses an object detection algorithm to efficiently observe and process video data into a single dataset.

Rail trespassing is a significant safety concern resulting in injuries and deaths throughout the country, with the number of such incidents increasing over the past decade. Following passage of the 2015 Fixing America’s Surface Transportation (FAST) Act that mandated the installation of cameras along passenger rail lines, transportation agencies have installed CCTV cameras at rail crossings across the country.  Historically, only through recorded injuries and fatalities were railroads and transportation agencies able to identify crossings with trespassing issues. This analysis did not integrate information on near misses or live conditions at the crossing. Cameras could record this data, but reviewing the video would be a laborious task that required a significant resource commitment and could lead to missed trespassing events due to observer fatigue.

Zaman, Liu, Zhang, and Xu saw this problem as an opportunity to put AI techniques to work and make effective use of the available video and automate the observational process in a more systematic way. After utilizing AI for basic video analysis in a prior study, the researchers theorized that they could train an AI and deep learning to analyze the videos from these crossings and identify all trespassing events.

Working with NJDOT and NJ TRANSIT, they gained access to video footage from a crossing in Ramsey, NJ.  Using a deep learning-based detection method named You Only Look Once or YOLO, their AI-framework detected trespassings, differentiated the types of violators, and generated clips to review. The tool identified a trespass only when the signal lights and crossing gates were active and tracked objects that changed from image to image in the defined space of the right-of-way. Figure 1 depicts the key steps in the process for application of AI in the analysis of live video stream or archived surveillance video.

Figure 1. General YOLO-Based Framework for Railroad Trespass Detection illustrates a step-by-step process involving AI algorithm configurations, YOLO-aided detection, and how trespassing detection incidents are saved and recorded to a database for more intensive analysis and characterization (e.g., trespasser type, day, time, weather, etc.)

The researchers applied AI review to 1,632 hours of video and 68 days of monitoring. They discovered 3,004 instances of trespassing, an average of 44 per day and nearly twice an hour. The researchers were able to demonstrate how the captured incidents could be used to formulate a demographic profile of trespassers (Figure 2) and better examine the environmental context leading to trespassing events to inform the selection and design of safety countermeasures (Figure 3).

Figure 2: Similar to patterns found in studies of rail trespassing fatalities, trespassing pedestrians were more likely to be male than female. Source: Zhang et al
Figure 3: Trespassing events were characterized by the gate angle and timing before/after a train pass to isolate context of risky behavior. Source: Zhang et. al

A significant innovation from this research has been the production of the video clip that shows when and how the trespass event occurred; the ability to visually review the precise moment reduces overall data storage and the time needed performing labor-intensive reviews. (Zhang, Zaman, Xu, & Liu, 2022)

With the efficient assembly and analysis of video big data through AI techniques, agencies have an opportunity, as never before, to observe the patterns of trespassing. Extending this AI research method to multiple locations holds promise for perfecting the efficiency and accuracy in application of AI techniques in various lighting, weather and other environmental conditions and, more generally, to building a deeper understanding of the environmental context contributing to trespassing behaviors.

In fact, the success of this AI-aided Railroad Trespassing Tool has led to new opportunities to demonstrate its use. The researchers have already expanded their research to more crossings in New Jersey and into North Carolina and Virginia. (Bruno, 2022) The Federal Railroad Administration has also awarded the research team a $582,859 Consolidated Rail Infrastructure and Safety Improvements Grant to support the technology’s deployment at five at-grade crossings in New Jersey, Connecticut, Massachusetts, and Louisiana. (U.S. DOT, Federal Railroad Administration, 2021) Rutgers University and Amtrak have provided a 42 percent match of the funding.

The program’s expansion in more places may lead to further improvements in the precision and quality of the AI detection data and methods.  The researchers speculate that this technology could integrate with Positive Train Control (PTC) systems and highway Intelligent Transportation Systems (ITS). (Zhang, Zaman, Xu, & Liu, 2022) This merging of technologies could revolutionize railroad safety. To read more about this study and methodology, see this April 2022 Accident Analysis & Prevention article.


Bruno, G. (2022, June 22). Rutgers Researchers Create Artificial Intelligence-Aided Railroad Trespassing Detection Tool. Retrieved from

NJDOT Technology Transfer. (2021, November 8). How Automated Video Analytics Can Make NJ’s Transportation Network Safer and More Efficient. Retrieved from

Tran, A. (n.d.). Artificial Intelligence-Aided Railroad Trespassing Data Analytics: Artificial Intelligence-Aided Railroad Trespassing Data Analytics:.

United States Department of Transportation: Federal Railroad Administration. (2021). Consolidated Rail Infrastructure and Safety Improvements (CRISI) Program: FY2021 Selections. Retrieved from

Zaman, A., Ren, B., & Liu, X. (2019). Artificial Intelligence-Aided Automated Detection of Railroad Trespassing. Journal of the Transportation Research Board, 25-37.

Zhang, Z., Zaman, A., Xu, J., & Liu, X. (2022). Artificial intelligence-aided railroad trespassing detection and data analytics: Methodology and a case study. Accident Analysis & Prevention.

Posted in Innovation Spotlight, News, Research Spotlight and tagged , , , , , , .