Compare commits

..

No commits in common. "v10" and "main" have entirely different histories.
v10 ... main

5 changed files with 1 additions and 3811 deletions

3561
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,12 +0,0 @@
[package]
edition = "2024"
name = "web"
version = "0.1.0"
[dependencies]
fathom-function = { git = "ssh://git@github.com/fathom-io/pipeline-calculations.git", branch = "main" }
pipeline-application = { git = "ssh://git@github.com/fathom-io/pipeline-calculations.git", branch = "main" }
serde = { version = "1.0.219", features = ["derive"] }
tokio = { version = "1.43.0", features = ["macros", "rt-multi-thread"] }
uom = { version = "0.36" }
uuid = { version = "1" }

View File

@ -1,90 +1,2 @@
# ILI comparison
# a8d4322007b74927896300f2ccb738d1
The ILI Comparison module is a valuable tool designed to compare anomalies found in two
separate ILIs of the same pipeline. This module operates on an intelligent method that
incorporates user-defined criteria to ensure precision in the comparison process. The workflow
of this module begins with a girth-weld alignment step, which is crucial for identifying and
aligning the correct girth welds from the two in-line inspection reports. In this case, the
girth-weld alignment is applied to ILI reports conducted on 2011-06-01 and 2015-12-21, ensuring
accurate alignment between the inspections. Both ILIs were performed by Rosen using the MFL
(Magnetic Flux Leakage) technology.
Subsequently, the module matches metal loss corrosion anomalies based on the user's specified
criteria. This matching process evaluates the evolution of anomalies over a 5.5-year interval,
the time gap between the two ILIs. The output includes detailed reports that categorize the
results into matched features, matched anomalies, unmatched features, and unmatched anomalies.
In its final output, the module generates a comprehensive KPI Dashboard that showcases the
percentage of matched anomalies, providing users with an overview of the comparison results.
Furthermore, six charts visually illustrate the growth of matched anomalies over the 5.5-year
time interval between the two inspections. These visualizations provide valuable insights into
how anomalies have evolved and help users track changes in pipeline integrity over time.
In addition, the validated results from this module are used in the Local Growth Rate Module
for calculating the Final ACR and conducting future integrity assessments. The ILI Comparison
module, through the integration of historical and recent inspection data, offers an essential
framework for monitoring pipeline conditions and supporting maintenance decisions.
## Input
### Arguments
- `org_id`: as string which should be a valid `uuid` for the organization
- `project_id`: the id of the data project where the pipeline data is found
- `pipeline_id`: a `array` of string values representing a valid `uuid` for a pipeline
- `older_ili_id`: a `array` of string values representing a valid `uuid` for an ili report sequence
- `newer_ili_id`: a `array` of string values representing a valid `uuid` for an ili report sequence
- `weld_location_threshold`: a `float` value
- `feature_location_threshold`: a `float` value
- `upstream_girth_threshold`: a `float` value
- `orientation_threshold`: a `float` value
- `minimum_depth_growth_threshold`: a `float` value
- `minimum_length_growth_threshold`: a `float` value
- `surface_location_criteria`: an array of `string` each value should be one of
- `matching`
- `any`
- `target_minimum_match_rate`: a `float` value
## Creating the function on the platform
To create this function on the platform using the `cli` set up the port forwarding as shown in README.
Then run the following command to create the function.
```bash
cargo run functions create \
-f functions/ili_comparison \
-d "Compare two ILI reports" \
-o org_id=string \
-o project_id=string \
-o pipeline_ids=array \
-o matched_ids=array \
-o unmatched_ids=array \
-o summary_ids=array \
-i org_id=string \
-i project_id=string \
-i pipeline_id=array \
-i older_ili_id=array \
-i newer_ili_id=array \
-i weld_location_threshold=float \
-i upstream_girth_threshold=float \
-i feature_location_threshold=float \
-i orientation_threshold=float \
-i minimum_depth_growth_threshold=float \
-i minimum_length_growth_threshold=float \
-i surface_location_criteria=string \
-i target_minimum_match_rate=float
```
## Testing the function locally
You can run and test the function locally by running
```bash
cargo run
```
Then you can check it work with `curl` as follows
```bash
curl localhost:8080 -d $(jq '. | tojson' functions/ili_comparison/example_input.json)
```

View File

@ -1,21 +0,0 @@
{
"feature_location_threshold": "5",
"minimum_depth_growth_threshold": "0",
"minimum_length_growth_threshold": "-2",
"newer_ili_id": [
"d0dd7c6b-6c54-4149-b46c-5e5b033fe6dd"
],
"older_ili_id": [
"514167e0-f197-4edb-a382-f8db9b048613"
],
"org_id": "2cbfe270-d195-48ad-aed1-24145924635c",
"orientation_threshold": "15",
"pipeline_id": [
"01966d47-1d4c-7751-a1f1-0617caa3a00d"
],
"project_id": "680b61b0aedd6f9e639d8699",
"surface_location_criteria": "matching",
"target_minimum_match_rate": "25",
"upstream_girth_threshold": "0.05",
"weld_location_threshold": "9"
}

View File

@ -1,128 +0,0 @@
use fathom_function::tracing;
use pipeline_application::{
application::{
Application, IliComparisonOutput, MatchingCriteria, SurfaceLocationCriteria as SLC,
},
serialization::{
serialize_meter, serialize_millimeter, serialize_orientation_min, serialize_percent,
},
};
use uom::si::f64::{Angle, Length, Ratio};
use uuid::Uuid;
#[fathom_function::function]
async fn ili_comparison(input: Input) -> Result<Output, String> {
let mut output = Output::new(input.org_id, &input.project_id);
let app = Application::new_from_compile_env(input.org_id, &input.project_id).unwrap();
for ((previous_id, newer_id), pipeline_id) in input
.older_ili_id
.into_iter()
.zip(input.newer_ili_id)
.zip(input.pipeline_id)
{
let result = app
.ili_comparison(pipeline_id, previous_id, newer_id, &input.criteria)
.await
.map_err(|err| {
tracing::error!(
%pipeline_id, %newer_id, %previous_id, ?err,
"Error running comparison algorithm"
);
format!("{err:?}")
})?;
output.push_result(pipeline_id, result);
}
Ok(output)
}
#[derive(Debug, serde::Serialize, Default)]
struct Output {
org_id: Uuid,
project_id: String,
pipeline_ids: Vec<Uuid>,
/// TODO: remove when proper mapping can happen between function nodes
#[serde(rename = "ili_comparison_id")]
matched_ids: Vec<Uuid>,
unmatched_ids: Vec<Uuid>,
summary_ids: Vec<Uuid>,
}
impl Output {
fn new(org_id: Uuid, project_id: impl ToString) -> Self {
Self {
org_id,
project_id: project_id.to_string(),
..Default::default()
}
}
fn push_result(&mut self, pipeline_id: Uuid, ili_comparison_result: IliComparisonOutput) {
self.pipeline_ids.push(pipeline_id);
self.matched_ids.push(ili_comparison_result.matched_id);
self.unmatched_ids.push(ili_comparison_result.unmatched_id);
self.summary_ids.push(ili_comparison_result.summary_id);
}
}
#[derive(Debug, serde::Deserialize)]
struct Input {
org_id: Uuid,
project_id: String,
pipeline_id: Vec<Uuid>,
older_ili_id: Vec<Uuid>,
newer_ili_id: Vec<Uuid>,
#[serde(flatten)]
criteria: Criteria,
}
#[derive(Debug, serde::Deserialize)]
struct Criteria {
#[serde(with = "serialize_meter")]
weld_location_threshold: Length,
#[serde(with = "serialize_meter")]
feature_location_threshold: Length,
#[serde(with = "serialize_meter")]
upstream_girth_threshold: Length,
#[serde(with = "serialize_orientation_min")]
orientation_threshold: Angle,
#[serde(with = "serialize_percent")]
minimum_depth_growth_threshold: Ratio,
#[serde(with = "serialize_millimeter")]
minimum_length_growth_threshold: Length,
surface_location_criteria: SurfaceLocationCriteria,
#[serde(with = "serialize_percent")]
target_minimum_match_rate: Ratio,
}
#[derive(Debug, Clone, Copy, serde::Deserialize)]
#[serde(rename_all = "snake_case")]
enum SurfaceLocationCriteria {
Matching,
Any,
}
impl From<SurfaceLocationCriteria> for SLC {
fn from(value: SurfaceLocationCriteria) -> Self {
match value {
SurfaceLocationCriteria::Matching => Self::Matching,
SurfaceLocationCriteria::Any => Self::Any,
}
}
}
impl From<&Criteria> for MatchingCriteria {
fn from(value: &Criteria) -> Self {
MatchingCriteria::default()
.with_weld_location_threshold(value.weld_location_threshold)
.with_feature_location_threshold(value.feature_location_threshold)
.with_upstream_girth_threshold(value.upstream_girth_threshold)
.with_orientation_threshold(value.orientation_threshold)
.with_minimum_depth_growth_threshold(value.minimum_depth_growth_threshold)
.with_minimum_length_growth_threshold(value.minimum_length_growth_threshold)
.with_surface_location_criteria(value.surface_location_criteria)
.with_target_minimum_match_rate(value.target_minimum_match_rate)
}
}