How Can Operators Make Better Integrity Decisions for Underground Gas Storage Wells?
Based on the technical paper “SCC Susceptibility Modeling to Demonstrate New Guidelines for Machine Learning in Pipeline Integrity.”
Machine learning can help pipeline operators narrow the areas that warrant the greatest attention for stress corrosion cracking (SCC) inspection—but only when the model is developed, validated and interpreted using a process suited to pipeline integrity decisions.
C-FER acted as the primary contractor to execute a project addressing both sides of that challenge. The project established a step-by-step, best-practice process for applying machine learning in pipeline integrity management, then demonstrated that process by developing models to evaluate SCC susceptibility. The demonstration used anonymized pipeline attributes and inspection records covering approximately 30,000 kilometres of transmission pipelines from six operators in Canada and the United States, aligned with publicly available soil data.
The XGBoost models improved performance over a simple industry-standard method, narrowing down the focus to a smaller portion of pipelines most likely to be susceptible to SCC.
The result is not a tool that identifies the exact joint where a crack will be found. It is a decision-support approach that can help operators prioritize crack-detection in-line inspection (ILI), SCC direct assessment, and risk modelling—while recognizing that inspection, field verification and engineering judgment remain essential.
SCC Occurrence Is a Susceptibility Problem, Not a Point Prediction
SCC is an environmentally influenced cracking mechanism that can occur on buried pipelines through the interaction of material, stress, coating condition and surrounding environment. Predicting where it will occur is difficult because SCC indications are rare relative to the total number of pipe joints in a system, and many unaffected joints may share nearly identical attributes with joints where SCC is found.
Operators commonly have information such as pipe age, coating type, material strength, operating stress and inspection history over long sections of pipeline.
What isn’t typically available is the variable that distinguishes one individual joint containing SCC from an adjacent joint without it.
As a result, susceptibility is a more practical modelling objective than exact prediction of crack locations. A susceptibility model can rank joints or segments according to their relative likelihood of containing SCC and identify areas that may be lower priority. It cannot credibly tell an operator to excavate a specific joint with the expectation that a crack will definitely be present.
This distinction is central to using a machine learning approach responsibly in this context. The value lies in making inspection programs more targeted—not replacing the inspection activities that confirm the pipeline’s actual condition.
Starting with the Decision, Not the Algorithm
Machine learning projects sometimes begin with an algorithm and a large dataset, but an ill-defined goal. The guidance developed through this work takes the opposite approach: begin by defining the operational problem, the decision the model must support, and the level of performance required for the model to be useful.
The process includes five main stages:
- Defining the problem and establishing model requirements
- Collecting and processing data
- Engineering and training the model
- Evaluating performance and determining whether the model succeeds
- Deploying, monitoring and improving the model
This structure helps operators examine whether machine learning is appropriate before investing in model development. It also provides a consistent basis for documenting how data were selected, how performance was measured, what limitations apply and how the model should be maintained.
For safety-critical applications, model complexity must also earn its place. A more advanced model should demonstrate meaningful improvement over a simpler, established method. If it cannot, the simpler approach may remain easier to explain, maintain and defend.
Avoiding False Confidence in Model Performance
Pipeline data create a validation challenge that can easily be overlooked.
Adjacent pipe joints are connected and often share the same diameter, wall thickness, material, coating, age and operating environment. If individual joints are randomly divided between training and test datasets, one joint may be used to train the model while a nearly identical neighbouring joint is used to test it.
The model may then appear to perform well because information about the test data is already represented in the training data. This is a form of “information leakage” caused by spatial autocorrelation. It can produce an overly optimistic estimate of how the model will perform on a genuinely independent pipeline segment.
The project addressed this problem by splitting data using larger regional, diameter and age groupings rather than randomly assigning individual joints. An exploratory comparison showed that random joint sampling could lead to overly optimistic performance results, likely due to information leakage and overfitting. The grouped result provided a more credible indication of generalization performance.
When evaluating machine learning applications, a high performance score is meaningful only if the validation data are sufficiently independent from the data used to develop the model.
Making Machine Learning Explainable
For operators, regulators and technical reviewers, performance alone is not enough. They also need to understand why the model produces its predictions and whether those predictions are consistent with established engineering knowledge.
The project used SHapley Additive exPlanations, or SHAP values, to show how individual inputs influenced model outputs. This allowed the development team to examine whether influential variables represented plausible relationships with SCC susceptibility or whether the model might be responding to noise or spurious correlations.
Interpretability supports quality control and gives decision makers a clearer technical basis for determining whether a model is suitable for deployment. If a model’s most influential relationships conflict with basic physical understanding, that may indicate overfitting, data-quality problems or a limitation in the model’s applicability.
Why This Matters to Operators
Used within its intended scope, SCC susceptibility modelling can support:
- Prioritization of crack-detection ILI programs
- Selection of SCC direct-assessment segments and excavation locations
- Relative ranking of pipeline segments
- Risk-model inputs for pipelines without crack ILI data
- Evaluation of piggable and non-piggable pipelines
- Documentation of the technical basis for integrity decisions
- More consistent review of machine learning models and practices
The practical benefit is not certainty. It is better differentiation. By identifying where SCC is relatively more or less likely, operators can focus inspection and engineering attention where it is expected to provide the greatest integrity value.
Building a More Consistent Machine Learning Practice
As operators collect and standardize more inspection, condition and operational data, opportunities to apply machine learning across pipeline integrity management will continue to grow. Potential applications extend beyond SCC susceptibility to inspection-data analysis, monitoring, threat evaluation, damage prevention and risk modelling.
The next step is broader application of a consistent process: clearly defined problems, appropriate data, independent validation, interpretable results and ongoing monitoring after deployment. Continued collaboration among operators, integrity specialists, data scientists, industry organizations and regulators can also support a shared vocabulary and a stronger community of practice.
For technical decision makers, the central message is straightforward: machine learning can add value to pipeline integrity management, but the algorithm is only one part of the solution. Confidence depends on how the problem is framed, how the data are handled, how the model is tested, and whether its outputs can be understood and incorporated into established integrity decisions.
How C-FER Helps Address the Challenge
C-FER brings pipeline integrity management, statistical analysis and risk-modelling experience to machine learning applications. That combination helps connect data-science methods with the physical mechanisms, inspection practices and operational decisions the models are intended to support.
For operating companies, C-FER can help:
- Define a machine learning problem around a specific integrity decision
- Assess whether available data are appropriate and sufficiently representative
- Clean, align and combine pipeline, inspection and environmental datasets
- Develop interpretable models using inputs available to operators
- Design validation approaches that account for spatial or operational relationships
- Compare advanced models against simpler engineering baselines
- Translate model outputs into inspection, direct-assessment and risk priorities
- Document assumptions, performance boundaries and deployment requirements
This approach treats machine learning as part of an integrity-management process rather than as a stand-alone prediction tool.
Contributor
Lowell McAllister, BSc, PEng
Senior Project Engineer
Lowell McAllister is a Senior Project Engineer in C-FER’s Integrity Management & Structures department.
His expertise includes quantitative risk assessment, data analytics, and structural modeling for pipeline integrity applications, including for emerging fuel applications (such as hydrogen and carbon dioxide pipelines).