AutomatedBuildings.com Article - Coil “Crime Scene” Investigation

Imagine walking into your facility on a Monday morning to find water covering the floor. Water damage is everywhere – mechanical rooms, laboratories and office spaces. The laboratory experiments and the complex machinery used to run the experiments are damaged. The office computers, paperwork, chairs and carpets are wet. As an owner, operator, user or facility manager this is your nightmare scenario. The costs and the liabilities from such an incident can be substantial. But who is to blame for such a crisis? Was anyone at fault or was it simply an unfortunate equipment failure? Your first words would likely be “Get me the Facilities Manager!”

Now imagine that all of the facilities management functions are actually performed by a contractor. Does your interpretation of ‘liability’ change?

Unfortunately, such an event actually happened. Luckily for the building owner, Cimetrics was already providing Analytika, an ongoing commissioning, fault detection and diagnostics service, to the owner. Analytika enabled Cimetrics to investigate the crime scene, dubbed “The Coils Incident”, to determine precisely what happened and who should be held accountable. Utilizing the Analytika system, Cimetrics was able to conduct ‘Operational Forensics’ to determine the sequential events that led up to The Coils Incident and to determine the root cause of the failure. Ultimately, the outsourced contractor admitted negligence and was held responsible for damages of more than $60,000.

Operational forensics is the practice of using building automation system data and data analytics to determine the sequence of events leading up to an incident and to determine root cause. It is a systematic approach to putting together pieces of an operational puzzle using temperatures, pressures, equipment signals and alarms, to better understand what was done (or not done) to bring about unintended consequences.

The Technical Details

The facility, located on the east coast of the US, is a mixed-use space, which is comprised of offices and laboratories. The office air handling units (AHUs) are mixed air units with modulating outside air dampers that are capable of economizer operation when outdoor air conditions are favorable. The laboratory AHUs are 100% outside air (or ‘single pass’) units. All of the AHUs and major equipment are controlled by a Building Automation System (BAS) and the facility has at least one engineer on duty 24 hours a day seven days a week.

Two of the four laboratory AHUs’ coils ruptured due to freezing of the chilled water flowing through them. Since these AHUs are large capacity (greater than 35,000 CFM supply air and 500 GPM chilled water coils), and also 100% outside air units, they are highly susceptible to freezing during extreme cold outdoor air conditions, if proper precautions are not taken. When the air passing over or around chilled water (CHW) coils drops below freezing (32°F) and there is little or no water flowing through the coils, the water freezes and expands which causes the copper coils to burst. If the control valves that modulate the CHW flow are located on the exit side of the coils then chilled water will flow out of the burst coils once the ambient air temperature around the coils rises above freezing. This scenario occurred in this facility; chilled water flooded the mechanical room and leaked through the floor to the occupied spaces.

Cimetrics collects and stores data 24/7/365 for all HVAC equipment at the site. Hence, we were able to use operational forensics after the fact to identify the root cause of the damaged coils. Not only was the building owner able to hold the outsourced facility management contractor accountable, but they were able to put measures in place to ensure that similar disruptions are not repeated.

A detailed operational forensic analysis performed by Cimetrics determined the chain of events that led up to the failure. The analysis revealed that the outsourced facility management engineering staff could have easily prevented the coils from freezing, and that the engineering staff ignored several warning signs.

The BAS workstation has low temperature or ‘freeze stat’ alarms programmed into the sequences of operation. Cimetrics discovered that the engineering staff ignored the freeze stat alarms and took no action to investigate the cause of the alarms. It is important to note that Cimetrics had warned the client of the upcoming cold spell and that some of the AHU coils were susceptible to freezing if the alarms were ignored. These warnings were also disregarded by the facility operating contractor.

Cimetrics’ operational forensics also determined that the AHU freeze protection control logic was inadequate. Typically, for 100% outdoor air AHUs with chilled water cooling coils, the BAS control logic will command the AHU supply fan to shut down, the outdoor air dampers to shut and the preheat and CHW valves to be signaled full open when the air temperature entering the CHW coils approaches freezing (32°F). In addition to this software programming, hardware override switches automatically shut down the AHU when the inlet air temperature to the CHW coils approaches freezing.

Because the laboratory experiments are conducted around the clock, the laboratory AHUs are in operation 24 hours a day seven days a week. The figure above shows a 24-hour period with outside air temperature (blue line) on the left axis and the number of alarms (bars) for each of the (4) laboratory AHUs on the right axis. The outside air temperature was below freezing (32°F) for the majority of the day and dropped rather rapidly during the evening hours. As shown in the figure, three of the four AHUs went into alarm starting around 6:30 PM and continued in alarm mode through midnight. The fact that the units stayed in alarm mode for extended periods of time shows that these alarms were not addressed. Cimetrics confirmed this when the client visited the Engineering Office after The Coils Incident and noticed a note on the wall saying “Do not ignore alarms and do not operate the AHUs in operator mode”. As it turns out, the site personnel had a history of ignoring BAS alarms and running equipment in override.

The operational forensics revealed that neither the preheat valves or the CHW valves were signaled full open when the preheat temperature dropped to critically low levels after the unit shut down. The following figure shows that the preheat discharge air temperature dropped below freezing (32°F) for extended periods of time.

The figure above shows data for AHU-2 which was one of the units whose chilled water coils were frozen. The AHU preheat discharge air temperature low limit setpoint is 41°F (red). During times when the outside air temperature falls below 41°F, the preheat valve should modulate open to maintain the preheat air temperature to at least the low-limit setpoint. The analysis of AHU-2 data revealed that the preheat temperature dropped below the 41°F low-limit at around 7:30PM and the unit shut down on freeze alarm at around 7:00PM. The preheat valve signal remained at 25% OPEN even as the preheat temperature dropped below 41°F.

During this entire time period the CHW valve was signaled full CLOSED (or 0%). Opening the chilled water valve fully during extremely low preheat air temperature conditions (when the unit goes into freeze ALARM) may have prevented the chilled water coils from freezing. The pressure in the chilled water loop would have forced enough chilled water through the coils to delay, if not prevent, the water in the CHW coils from freezing.

Following the incident Cimetrics worked with the building engineers to help prevent the situation from happening again. In particular, several policy measures and BAS programming changes were enacted including:

While ongoing commissioning has significant energy and cost savings benefits, this episode shows that the benefits do not stop at cost savings. Unfortunately, sooner or later, a scenario like this is likely to happen at your facility. When it happens how prepared will you be to determine the root cause? Will you be prepared to address it quickly or, better yet, prevent it from happening all together? Rest assured, monitoring based ongoing commissioning can provide the operational forensics and preventative capability you need, and it is always there when you need it.

A Happy Ending

A serious system failure can be a catalyst for improved operations. Following the incident described in this article, the building owner, the facility management contractor, and the ongoing commissioning services provider (Cimetrics) established a more effective working relationship. This led to improved operational procedures and prioritization of maintenance activities, resulting in a lower risk of serious system failures and additional energy savings.

Managing the Outsourced Facility Management

The ability to perform forensic investigations as described above is especially important in buildings with outsourced facility operations. In fact, the outsourcing of facility and real estate management is increasingly common across some industries, as shown in the following figure.

About the Authors

Jim Lee – CEO, Founder – Cimetrics Inc.
Mr. Lee is the founder of Cimetrics and has acted as its CEO since its formation. Mr. Lee has been a leader in the embedded control networking and building automation community for over 20 years. As founder and former President of the BACnet Manufacturers Association (now BACnet International), the leading open systems networking consortium in the building automation industry, Mr. Lee’s aggressive promotion of the BACnet open protocol standard has helped make Cimetrics a high profile player in the arena. Mr. Lee has an earned B.A. in Physics from Cornell University.

Dean Taylor – Manager of Analytics – Cimetrics Inc.
Mr. Taylor is a Certified Energy Manager (CEM) with more than 15 years of experience in the field of data analytics including fault detection and diagnostics (FDD), energy reporting, modeling, commissioning, developing energy projects and performance contracts. He holds a bachelor’s degree in Mechanical Engineering from the University of South Florida and can be reached in the Boston office at (617) 350-7550 ext 240 or at dtaylor@cimetrics.com.

Cimetrics is a world leader and innovator in the field of physical world analytics. Cimetrics operates two business units – BACnet products www.cimetrics.com and Analytika www.analytika.com. Analytika for Buildings (AFB) provides energy savings, sustainability and tenant comfort to energy, facility and sustainability managers in the Pharmaceuticals & Life Sciences, Higher Education, Healthcare and Government market sectors. Analytika for Process (AFP) serves the needs of manufacturing and quality leaders and delivers improved process reliability, quality and throughput, while reducing the risk of catastrophic failures and regulatory compliance issues. Cimetrics’ BACnet products business provides hardware, embedded software and consulting services to Building Automation System OEM manufacturers and their channel partners. Cimetrics commands 60% market share of the global BACnet communications software market.

Events	Want Ads
Our Sponsors	Resources