Alan Ross
I am Alan Ross, and this is our Power Panel Discussion about monitoring, diagnostics, maintenance and reliability. Our guests today are Brad Bowness from Systems With Intelligence, Emilio Morales from Qualitrol, Bob Rasor from SD Myers, and Mark Tostrud from Dynamic Ratings.
Emilio we will start with you. What do you see in general for monitoring and diagnostics (M&D)?
Emilio Morales:
Right now, monitoring and diagnostics is evolving rapidly, not only for transformers and not only for the electric industry, but also for all industry assets. Basically, it is driven by advances in new sensor technology, data analytics, and the use of AI. We are at a point now where real-time data collection is not the challenge anymore. The challenge is how we process, interpret, and act based on that data. That is what defines the current state of monitoring and diagnostics. Another thing that is evolving are security challenges. In short, monitoring and diagnostics is shifting from being reactive to proactive, from isolated to integrated, and from raw data to intelligent decision making.
Nowadays, it is challenging for companies to implement a comprehensive monitoring system due to long lead times for transformers. The process—from writing specifications and conducting the bid process to working with OEMs—may now take five to six years, all while keeping the existing fleet operational.
Alan
Excellent, thank you, Emilio. Brad, your thoughts about what Emilio just shared?
Brad Bowness
Emilio covered a lot of the technical aspects of sensors and the fact that several utilities are moving forward with programs like this. But I think at a macro level, utilities are under a lot of pressure, pressure to harness their existing asset base to its fullest. Regulators and investors and customers from a rate base and the impact on rates perspective are really looking for utilities to get the most out of their existing assets. Putting monitors in place is allowing utilities to make better decisions, have better insight on their assets, and stretch the life of the existing asset base.
Utilities also have a lot of pressure from a maintenance cost perspective, looking at ways to automate and digitize legacy maintenance processes, taking advantage of the lower cost of sensors now, and then bringing that together into a centralized data and analytics group.
I think the number of data scientists, data M&D centers, and condition-based maintenance groups are growing within utilities.
Utilities are heavily investing in the data and analytics space, and with the input of the field sensors, they are looking to make more informed decisions. I think the M&D space is just going to continue to progress and take off here over the next number of years.
Bob Rasor
We are heavy in the industrial space, so we have some limitation on the asset’s value as to how much can be invested into the transformer and as Emilio said, with the longer lead times we see the value of continual data to bring around actions that will save a transformer from unplanned outages. We collaborate more with customers now to help interpret the data to make the right decisions.
Mark Tostrud
Utilities and industrials have a lot of challenges. They are facing an aging workforce with the loss of expertise, of how they used to do business, and as those people retire, the newer people are not exposed to the same historic information which creates a challenge. They are turning to monitoring as a result. Also, the aging infrastructure creates additional challenges as well. They typically have less O&M dollars to work with, but at the same time a growing asset base that they need to monitor and maintain. It is becoming more challenging.
One of the things that Emilio pointed out was that we see a real merging of the data from multiple sensors, multiple technologies, all coming together and making more and better use of that. One of the drivers I see is that we are not building a utility infrastructure for you and me. We are building this for our kids and our grandkids, and we must do things differently to do that. Bringing that data together and putting it in a place so it is easily accessible, we can see it and react to. It is key here. Right now, monitoring and diagnostics is evolving rapidly, not only for transformers and not only for the electric industry, but also for all industry assets. Basically, it is driven by advances in new sensor technology, data analytics, and the use of AI. Emilio Morales Alan
Excellent. Because I have transformer oil in my blood, as you all do as well, let’s focus on transformer monitoring specifically; Mark, where are we now and where do you think we are going?
Mark
When people talk about monitoring, there has been monitoring in many different formats over the years. In a sense, a typical gage that gives you an alarm output when a transformer is overheating is monitoring in some respects. Those types of devices, the old gages and things like that, they worked well. They got us where we are today, but they did not really have much predictive capability.
What we are seeing today is a movement more towards predictive monitoring: How can I get advanced warning of what is happening to the assets so I can make better decisions? When you think about asset management, while it has been around for years, oftentimes they really did not have the data to manage the assets properly. You might hear, “Okay, I get DGA data once a year on a transformer and I am supposed to decide how healthy my fleet is based on that? What I really need to know is can I overload it?” That could be very challenging.
People are really starting to make use of that data, but it is more than just the monitor. It is bringing data back, putting it on a dashboard, making that data accessible for everybody in the company to see, so they can leverage it. There is a process that we go through while doing that. The first step is getting that monitor in place, but then we must get the data out of the monitor and make sure we have data validation, making sure that it is accurate.
We load the data into a dashboard type format so that it is accessible to others, deciding what the user experience needs to be and then deciding how the layout is going to be because obviously we need to do some screens that control the view. Next, we must leverage the data, whether it is out of Pi or the data historian that we are working with. As it continues to evolve, it takes more of a vision. It is not just about the monitor. It takes a vision as to where you are going and what you want it to look like in 5 or 10 years.
Alan
Mark, that is excellent. It takes a vision. Bob, your thoughts?
Bob
One of the words I heard Mark used that always rings the bell is the word predictive. Tied to predictive must be multiple data points. Getting one sample a year, as you mentioned, can be very misleading. Even around moisture in a transformer, depending on when the sample is drawn, the load and so forth makes a difference. We like to use trending as a very powerful tool, and one of the best ways to trend is to have more data. That leads us to knowledge about the data, which is great. Tied to that knowledge is the wisdom as to where it all leads us. I just think the trending part of what we’ve experienced can lead to the best decisions to keep transformers reliable and power systems reliable.
Brad
I agree, and what I would add over and above the trending aspect is you must get to some critical mass. Utilities, especially, are very good at piloting things. You can always get a bit of money out of an R&D group or an innovation group to move forward with trying something new within a utility. That is my experience.
But to get into the programmatic view, you must have some wins. I think that is what we are starting to see, a number of real-world examples coming from utilities where we are seeing those wins or those saves, where they have deployed to a critical mass of some monitor, and that is leading to some “aha” moments where people are saying, “If I deploy this across the fleet, I can now bring this data into a centralized repository, start to look for insights, start to correlate data with my load and my performance and my customer criticality data. Now I am starting to make a truly informed decision and have a true health index around the power equipment.” I agree with Mark that you must have that vision, you have to have that corporate sponsorship.
You really must go into this thinking, ‘We are trying to build the monitoring and grid and capabilities for the next 25 to 50 years.” To do that, we have to be programmatic about it so we cannot just be treating these as R&D projects or innovation projects. They really must have that vision and sponsorship to get to that critical volume of data to generate the insights.
Emilio
I completely agree with everyone. Trending, like Bob said, then in addition to trending, we must make the correlation between parameters that can tell you what type of failure mode you are dealing with. But another thing is that companies who develop this type of monitoring system, have some issues regarding controlling access to the data. The end users do not like to share the data they have so right now, monitoring companies must rely on a theory about how the component fails; the failure mode, the symptoms and parameters, how they train, how they correlate the data. It would be nice if we could have more data to confirm that across all types of equipment but for the most part, we do not see data sharing very much. Putting monitors in place is allowing utilities to make better decisions, have better insight on their assets, and stretch the life of the existing asset base. Brad Bowness Capital money is hard to come by so spares are very important. You must make sure you are ready when the unthinkable might happen. It might just not be an aging transformer. It could be a catastrophic event.
Bob Rasor Alan
Mark, I think the idea that we look at this differently, as a vision is important for change. Bob share your experience with getting “buy-in” from the corporate entity to agree with what Mark said, that it will require a vision for change.
Bob
That is a great question, Alan. A couple of things come to mind. Budgets are a big driver in decisions, so when the critical moment hits, the money will show up. We want to avoid the conditions where the money is huge because we are responding to a failure as opposed to preventing one, and so we must make decisions along the way.
For instance, let us look at spare transformers. When we are looking at trending and we see an older transformer that is starting to have an issue, as Emilio said, you must correlate different readings. You might look at gas levels or leaks because it is a 30-year-old transformer, and the gaskets are failing. Some data, chemical-wise, might tell you that the transformer is on its way out. You must make sure that you have a plan in place for when it does go out, so you are not scrambling, because the scrambling places to go for a replacement can be problematic. Sometimes you will get, I hate the word lucky, but you could find one and you would be great shape to get it replaced quickly. Other times, you are on a waiting list for six months, so spares are very important to strategically select and make sure it is ready when you need it.
Is it not just the KVA, the voltage, the impedance, it is also the fit. If you have a spare that can go in a couple of different places, you want to analyze all your system, your fleet, and make sure you have the right spares and the right plan in place for when things do happen. You should have the spare ready to go which leads to being resilient. You get right back into shape.
We find that monitoring-wise, while we have talked a lot about instruments, the monitors themselves, getting back to the basics of just simple visual inspections, oil testing, and even infrared scanning is a great place to start. These are called out recently in a guide format in NFPA 70B. We are all familiar with 70E, which is the safety aspects of transformers and substations in the electrical world. NFPA 70B talks about guidelines for testing, annual testing for transformers with liquids, and they go into electrical testing every three to five years. You do not want to spend too much money on something that is running fine, but you also have some of these regulations now to say, “If you had an event and you weren’t following some clear guidelines, that is where you might have gone wrong”.
Capital money is hard to come by so spares are very important. You must make sure you are ready when the unthinkable might happen. It might just not be an aging transformer. It could be a catastrophic event. Fires, windstorms and floods means we must have a plan in place where wisdom comes in. That is where we see a lot of this M&D effort going.
Brad
In the context of the approval process with getting moving on these programs, it does take a long time to get into the capital planning process or even into the maintenance release of funds process. With long regulatory cycles, long business planning cycles, the challenge you have is that if you do not get a head start, it is very difficult to catch up.
Some of these sensors that need to be installed require outages. The work needs to be planned with existing work that might be occurring at a substation. The whole M&D space and implementation of monitors and sensors should be done on a programmatic basis. You should be rolling through your fleet month over month, quarter over quarter, and starting to gather this data so that you can start to get those insights in those centralized systems. If you do not do it, and suddenly you have a major event, then somebody is going to say, “Why haven’t we been doing this? There have been so many saves and success stories in other utilities, why didn’t we do this?” And then it is a hurry up and get it done event which, as Bob said, is the costliest way to do it. Also, you have the constraints of resourcing other projects, outage constraints and the like. It is very, very difficult to catch up. Think about doing this on a thorough programmatic basis based on some other utility’s successes and not on your own internal successes. There are enough success stories out there to support making those great decisions so that you do not end up with those catastrophic failures that you will end up having to explain to a regulator, or internal board, or customers in the future.
Alan
Mark, you brought vision in, and now we have added predictive correlation of data, and shared successes. Add to what Emilio, Bob and Brad have said.
Mark
I think Brad really hit it on the head. You need to address your new assets, which is the easier part because I can write a monitoring spec for my new assets. But the challenge is if the lead time on that asset is going to be three years, you need to be predictive enough as to where you think you are going, because what you are monitoring today may not exactly match with what you want to monitor three years from now. You need to write it openended enough so that as the technology evolves, you will still get that into the spec without a change order, which can be a little challenging. And as Brad said, you really need to address the existing fleet, too. That is certainly challenging to get those dollars to be able to do that.
Additionally, as Bob said, now, there are pressures with NFPA and NERC, where they are mandating a certain level of maintenance. Monitoring is a way to offset some of that, but it is very difficult to justify a monitoring program based on maintenance dollars alone, because there are just not enough savings there. You need to start looking at avoided cost of preventing failures, or the environmental damage that would occur. On critical assets, what is it going to cost your customers if you had an outage on that piece of equipment? Once you start including some of these more hidden costs, then it becomes much easier to cost-justify your maintenance dollars going to M&D. We are not building a utility infrastructure for you and me. We are building this for our kids and our grandkids, and we must do things differently to do that. Mark Tostrud Emilio
Whenever I visit customers in a seminar, people will ask, “Okay, so what should we do? What transformer do we have to monitor or what do we monitor?” Often, I tell them, look at the different variables that you have and try to address those first. The other thing is, justification-wise in a very practical sense, resort to the guide for the application of the diagnostic monitoring equipment, IEEE C 57, 143, which is currently under revision. It has a chapter or clause that talks about justification, so here is a guide, that gives very practical examples you can use. As everyone has said, one thing would be to prepare those units to receive monitoring equipment in the future. It is very simple and not a high cost. It could be a very standard way of doing things, to be ready for those transformers when they come out of the shop.
Alan
Brad, make a case for the value proposition for a monitoring and diagnostics program.
Brad
I think the first thing from my perspective is people who are working within the utility, running the day to day, are super busy. They are dealing with outages, projects, customer issues, regulatory issues, and everything else that is coming at them. It is hard to find time in the day to think about things from a value proposition perspective. I do think being able to pull somebody out of the day to day and giving them that business champion role, and having them have some time to think is important to be able to come up with that internal value proposition that is going to land well with the decision-making process.
Once that person is identified, I think the starting point should be the base case, really challenging yourself internally to say, “What is our base case? How much do we spend on maintenance today? How much do we spend on preventative maintenance, corrective maintenance? What is the value of extending assets by two, three years of their 50-year life? What is the cost of having that major catastrophic failure?”
Once you understand your base case, you can ask, “What can I do differently?” You can implement monitoring, diagnostics tools and capabilities, processes, systems, and analytics. These are the things you can impact on that base case. And now you have your value proposition. Now you can craft the story, that value proposition which is based on things internal as well as success stories in the marketplace.
Once you do an ROI and find out what you can influence, what is available in the marketplace you will see that the cost of many solutions has come way down. That creates a positive business case you can bring forward and take up the line to have funds allocated to a program. I think really looking at that and going through that process is extremely important to being able to move forward with type of initiative.
Emilio
Something else to add is that there is, in the industry, a lack of engineers. That is a big deal. On top of that, the retirement of subject matter experts you ask, “Where has that knowledge gone?” That is another value of monitoring our systems, where you can integrate that experience into the system, so that when you have to identify a problem, the problem is identified as a failure mode and then come up with actions about what needs to be done.
Another value is there is a study, not specifically on transformers, basically on GIS, gas insulated switchgear, where a lot of failures took place because of maintenance that was not needed. Going from time-based to condition-based or reliability-based you can avoid those possible mistakes that can cause a failure.
Bob
I think Brad touched on it, about having a person or a resource available who can sort through the fleet. We sort transformer fleets, asking “Is it critical to the mission of the whole operation or is it specific to one part of the plant? Is it a lower risk transformer if there is a problem?”
They all have a different story, and the story has to be integrated with how much money you can spend on maintenance, how much you will save if there is a problem. You can evaluate so that you could put the highest order M&D program on a highly critical transformer but on a smaller noncritical one it would be of no use and not even practical.
Emilio touched on the issue that the people who make the decisions have become fewer and fewer as they retire or move on to different things. The knowledge must be presented in a way that the new user can understand it and get the value propositions that Brad mentioned, in front of them so it helps them make wise decisions.
Mark
I have been a big fan of trying to make it a program and getting the data out of the field and making it more visible from day one, sharing your personal experience. If you can get that data and make it easily accessible for when you have a problem, you are in a better position to make those decisions and truly manage the asset properly. The other thing is, as you make that data available, there are going to be people who have that skill set to analyze it.
We talk about retiring SMEs which makes us strapped. As an example, when I was at the utility, we had initially one guy who analyzed DGAs. In the end, it turned out that a lot of us had that capability to analyze the DGA, but we needed access to the data. If we can make the data available and give them the tools, the Duval triangles, pentagons, gas ratios and things like that, make the diagnostics available to them, others are going to navigate.
Now you can start getting that down to the electrician level or maintenance level. Other people are going to be able to do that and help you with that. That is really some of the value add that you can start spreading the responsibility for reviewing the data across multiple departments and multiple individuals, which helps you really grow as an organization.
Alan
Thank you, gentlemen. This has been excellent. This article was originally published in the March issue of the Transformers Technology magazine, which you can access here.
To download the PDF version of this article, click here.