Instantiate ML safety requirements argument pattern

Generate ML data

Develop ML safety requirements

Verify ML model

Validate ML safety requirements

Validate ML data

Define the safety assurance scope for the ML component

Instantiate ML data argument pattern

Integrate ML model

Instantiate ML deployment argument pattern

Create ML model

Instantiate ML safety assurance scoping argument

 Define data requirements

Test ML model

Instantiate ML verification argument pattern

Test the integration

Instantiate ML learning argument pattern

ML deployment argument pattern

System safety requirements

Data requirements justification report

Internal test results

Data generation log

Data requirements

ML verification argument pattern

Erroneous behaviour log

System description

Description of the system environment

Model development log

Model learning argument pattern

Operational scenarios

Verification log

ML safety requirements argument pattern

ML data argument pattern

ML component description

ML safety assurance scoping argument pattern

ML verification evidence

Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies.

Autonomous vehicle safety: An interdisciplinary challenge.

Challenges of safety assurance for industry 4.0

The myth of generalisability in clinical research and machine learning in healthcare.

High‐performance medicine: the convergence of human and artificial intelligence.

Concrete problems in AI safety.

The 'K' in K‐fold Cross Validation.

Guidelines for development of civil aircraft and systems.

Quantifying assurance in learning‐enabled systems.

Assuring the machine learning lifecycle: Desiderata, methods, and challenges.

Body of Knowledge for Assurance of Robotic and Autonomous Systems (RAS)

Body of Knowledge for Assurance of Robotic and Autonomous Systems (RAS) ‐ Guidance on model learning

Body of Knowledge for Assurance of Robotic and Autonomous Systems (RAS) ‐ Guidance on the sufficiency of training data

Body of Knowledge for Assurance of Robotic and Autonomous Systems (RAS) ‐ Guidance on the use of Machine Learning

Body of Knowledge for Assurance of Robotic and Autonomous Systems (RAS) ‐ Guidance on verification of learnt models

Steering System Verification Using Hardware‐in‐the‐Loop.

Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective.

How to develop machine learning models for healthcare.

Efficient strategies for leave‐one‐out cross validation for genomic best linear unbiased prediction.

Deep Learning with Python.

Clinically applicable deep learning for diagnosis and referral in retinal disease.

The risk of disabling, surgery and reoperation in crohn’s disease–a decision tree‐based approach to prognosis.

Towards a rigorous science of interpretable machine learning.

Keras Developers Guides, January 2021.

An introduction to ROC analysis.

Assuring the safety of machine learning for pedestrian detection at crossings.

Hands‐on machine learning with Scikit‐Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems.

DeepSafe: A data‐driven approach for checking adversarial robustness in neural networks.

Assurance Case Working Group. Goal Structuring Notation Community Standard. Technical Report SCSC‐141BA v2.0

A generic goal‐based certification argument for the justification of formal analysis.

Artificial intelligence in healthcare: accountability and safety.

A new approach to creating clear safety arguments.

Strategies for handling missing clinical data for automated surgical site infection detection from the electronic health record.

Safety verification of deep neural networks.

Road vehicles safety of the intended functionality.

Systems and software engineering ‐ requirements engineering.

Challenges of safety assurance for industry 4.0.

Reluplex: An efficient smt solver for verifying deep neural networks.

The marabou framework for verification and analysis of deep neural networks.

Safety case construction and reuse using patterns.

Keras Developers Guides, December 2020.

Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy.

Standard for evaluation of autonomous products.

Natural Language Processing in Action Video Edition.

Deep Reinforcement Learning Hands‐On: Apply modern RL methods, with deep Q‐ networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more. 

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the consort‐ai extension.

Machine learning techniques to discover genes with potential prognosis role in alzheimer’s disease using different biological sources.

International evaluation of an AI system for breast cancer screening.

Towards verifying robustness of neural networks against semantic perturbations.

Safety assurance objectives for autonomous systems. Technical Report SCSC‐153A 2.0 ed.

Safety management requirements for defence systems part 1: Requirements.

DeepXplore: Automated whitebox testing of deep learning systems.

Putting image manipulations in context: robustness testing for safe perception.

Assurance argument patterns and processes for machine learning in safety‐related systems.

Early stopping‐but when? In Neural Networks: Tricks of the trade, page 55–69. 

Toward requirements specification for machine‐learned components.

Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior.

Software considerations in airborne systems and equipment certification. Standard RTCA DO‐178C:2012.

Verification and validation of simulation models.

Human factors of using artificial intelligence in healthcare: Challenges that stretch across industries.

Reinforcement learning: An introduction.

A critique on the public discussion on skewness and bias in healthcare AI.

TensorFlow Convolutional Neural Network (CNN) Tutorial, December 2020.

DeepTest: Automated testing of deep‐neural‐network‐driven autonomous cars.

An intuitive model independent and accurate certainty measure for machine learning.

Using AI to predict retinal disease progression.

Machine learning with applications in breast cancer diagnosis and prognosis.

DeepRoad: Gan‐based metamorphic autonomous driving system testing.

Towards simulating foggy and hazy images and evaluating their authenticity.

Automotive

Aviation

Healthcare

Manufacturing

Agriculture

Home

Safe autonomous systems

Safe machine learning

Future guidance

About us

Training

Contact

Accessibility

Terms and conditions

Privacy policy and cookies

Acknowledgements

AAIP 2022. All rights reserved.

aaip-logo-white.svg

lrf-logo.svg

universityofyorklogowhite.png

The AMLAS Tool has been developed to help you to work through the AMLAS process and create a safety case for your machine learnt (ML) component.
Use the tool to:
<ul>
<li>systematically guide you through the AMLAS activities enabling you to add artifacts as they are created</li>
<li>automatically create a safety case for the ML component as you work through AMLAS</li>
<li>keep track of your progress and see how much you have left to do</li>
<li>share the status of your ML safety case with other people</li>
</ul>
Please note that Microsoft Visio is required to use the AMLAS Tool.
You can download a user guide for instructions on how to install and run the AMLAS Tool. A video tutorial is also available.
<ul>
<li><a href="https://res.cloudinary.com/dcqnz7rcj/image/upload/v1651741660/AMLAS_Tool_User_Guide_V1_May_2022_4f8df08740.pdf">Download AMLAS Tool User Guide</a></li>
<li><a href="https://res.cloudinary.com/dcqnz7rcj/raw/upload/v1651741686/AMLAS_Tool_V1_f10f5a6514.zip">Download AMLAS Tool Zip file for Visio</a></li>
</ul>

How to use the AMLAS tool tutorial

AAIP AMLAS Tool user guidance video May 2022.mp4

The AMLAS Tool

Introduction

This activity requires as input the system safety requirements ([[A]](/artefacts/?slug=system-safety-requirements)), descriptions of the system and the operating environment ([[B]](/artefacts/?slug=description-of-system-environment), [[C]](/artefacts/?slug=system-description)), and a description of the ML component that is being considered ([[D]](/artefacts/?slug=ml-component-description)). These inputs shall be used to determine the safety requirements that are allocated to the ML component.

The safety requirements allocated to the ML component shall be defined to control the risk of the identified contributions of the ML component to system hazards. This shall take account of the defined system architecture and the operating environment. At this stage the requirement is independent of any ML technology or metric but instead reflects the need for the component to perform safely with the system regardless of the technology later deployed. The safety requirements allocated to the ML component generated from this activity shall be explicitly documented ([[E]](/activites/?slug=define-the-safety-assurance-scope-for-the-ml-component)).

Consider an autonomous driving application in which a subsystem may be required to identify pedestrians at a crossing. A component within the perception pipeline may have a requirement of the form “When Ego is 50 metres from the crossing, the object detection component shall identify pedestrians that are on or close to the crossing in their correct position.”

Pedestrian identification

The allocation of safety requirements must consider architectural features such as redundancy when allocating the safety requirements to the ML component. Where redundancy is provided by other, non‐machine‐learnt components, this may reduce the assurance burden on the ML component that should be reflected in the allocated safety requirements.

Consider architectural features

The contribution of the human as part of the broader system should be considered. A human may provide, for example, oversight or fallback in the case of failure of the ML component. These human contributions, and any associated human factors issues e.g. automation bias ([59]), should be reflected when allocating safety requirements to the ML component.

Consider human contribution

This activity requires as input the ML safety assurance scoping argument pattern ([F]), as well as the artefacts from Activity 2 ([A], [B], [C], [D] and [E]). The activity uses these artefacts to create an instantiated ML assurance scoping argument ([G]) which documents the ML safety assurance scoping argument for the ML component and provides references to the contextual artefacts.

ML safety assurance scoping

This activity requires as input the ML safety requirements argument pattern ([I]), as well as the artefacts from the previous activities of this stage ([E], [H]and [J]). The activity uses the activities and outputs from the previous stages to create an instantiated ML requirements argument ([K]).

This activity requires as input the system safety requirements allocated to the ML component ([E]).

ML safety requirements shall be defined to control the risk of the identified contributions of the ML component to system hazards, taking into account the defined system architecture and operating environment. This requires translating complex real-world concepts and cognitive decisions into a format and a level of detail that is amenable to ML implementation and verification [55].

In a system used for cancer screening based on X‐ray images, a subset of the ML safety requirements will likely focus on defining an acceptable risk level by specifying true positive and false positive rates for diagnoses within the given environmental context [46]. This will take account of the performance achievable by human experts performing the same task, and the level of checking and oversight that is in place.

Defining acceptable risk level

From the safety requirement allocated to the ML component in Example 1 the concept of identifying a pedestrian at the system level must be translated into something meaningful for the ML model. Using knowledge derived from the system architecture, the ML safety requirement becomes “all bounding boxes produced shall be no more than 10% larger in any dimension than the minimum sized box capable of including the entirety of the pedestrian” [22].

Translating safety requirements

To some extent, the process of developing the machine learning safety requirements is similar to those of complex embedded software systems (e.g. in avionics systems or infusion pumps). However, due to the increasing transfer of complex perception and decision functions from an accountable human agent to the machine learning component, the difference between the implicit intentions on the component outputs and the explicit requirements that are used to develop, validate and verify the component is significant. This “semantic gap” exists in an open context for which a credible, let alone complete, set of concrete safety requirements is very hard to formalise [12].

Semantic gap

In machine learning, requirements are often seen as implicitly encoded in the data. As such, under‐specificity in requirements definition is an appealing feature. However, for rare events, such as safety incidents, this under‐specificity poses a significant assurance challenge. The developers are still expected to assure the ability of the system to control the risk of these rare events based on concrete safety requirements against which the machine learning component is developed and tested.

Under‐specificity

While there are likely to be a large range of requirements for the ML component (e.g. security, interpretability etc.) the ML safety requirements should be limited to those requirements that impact the operational safety of the system.

Other types of requirements such as security or usability should be defined as ML safety requirements only if the behaviours or constraints captured by these requirements influence the safety criticality of the ML output.

‘Soft constraints’ such as interpretability may be crucial to the acceptance of an ML component especially where the system is part of a socio‐technical solution. All such constraints defined as ML safety requirements must be clearly linked to safety outcomes.

Safety criticality

The ML safety requirements shall always include requirements for performance and robustness of the ML model. The requirements shall specifically relate to the ML outputs that the system safety assessment has identified as safety‐related (i.e. not just generic performance measures).

In this document, ML performance considers quantitative performance metrics (e.g. classification accuracy and mean squared error), whereas ML robustness considers the model’s ability to perform well when the inputs encountered are different but similar to those present in the training data, covering both environmental uncertainty (e.g. flooded roads) and system‐level variability (e.g. sensor failure [5]).

Performance and robustness

The performance of a model can only be assessed with respect to measurable features of the ML model. A model does not generally allow for us to measure risk or safety directly. Hence safety measures must be translated to relevant ML performance and robustness measures such as true positive count against a test set or point robustness to perturbations. Indeed not all misclassifications have the same impact on safety (e.g. misclassifying a speed sign of 40 mph as 30 mph is less impactful than misclassifying the same sign as 70 mph).

Measurable features

There is rarely a single performance measurement that can be considered in isolation for an ML component. For example, for a classifier component, one may have to define a trade‐off between false positives and false negatives. Over-reliance on a single measure is likely to lead to systems that meet acceptance criteria but exhibit unintended behaviour [1]. As such, the ML performance safety requirements should focus on reduction/elimination of sources of harm while recognising the need to maintain acceptable overall performance (without which the system, though safe, will not be fit for purpose). Performance requirements may also be driven by constraints on computational power (e.g. the number of objects that can be tracked). This is covered in more detail in Stage 6 (ML deployment).

Overall performance

One useful approach to defining robustness requirements is to consider the dimensions of variation which exist in the input space. These may include, for example:

- variation within the domain (e.g. differences between patients of different ethnicity);
- variation due to external factors (e.g. differences due to limitations of sensing technologies or effects of environmental phenomenon);
- variation based on a knowledge of the technologies used and their inherent failure modes.

Dimensions of variation

The ML safety requirement presented in [Example 1](/amlas/activities/define-the-safety-assurance-scope-for-the-ml-component#ex61acf1d200aeea0016b77ed0) may now be refined into performance and
robustness requirements [22]. Example performance requirements may include:

- The ML component shall determine the position of the specified feature in each input frame within 5 pixels of actual position. 
- The ML component shall identify the presence of any person present in the defined area with an accuracy of at least 0.93

Example robustness requirements may include:

- The ML component shall perform as required in the defined range of lighting conditions experienced during operation of the system.
- The ML component shall identify a person irrespective of their pose with respect to the camera.

Safety assessment shall not be limited to system‐level activities. It is not a mere top‐down process. Safety assessment shall be carried out in a continuous and iterative manner. A detailed safety analysis of the outputs of the ML model shall be performed. This may identify new failure modes. The results of this analysis shall be fed back to the system‐level safety assessment process for further examination, such as reassessing the risk rating for a hazard.

The activity of developing the ML safety requirements will likely identify implicit assumptions about the system or operating environment. Assumptions that are made shall be made explicit either as part of the description of the system environment or through defining additional safety requirements. Some domains refer to these as derived safety requirements.

Derived safety requirements could relate to the assumed reliability and availability of sensor outputs or the specified thresholds of tolerable risk. In the case of the latter, current societal norms might accept the delegation of the interpretation of these often qualitative criteria to an accountable human (e.g. a quantified driver or a clinical professional). Given the transfer of complex cognitive functions from human users to machine learning components, the process of developing concrete ML safety requirements will likely demand the interpretation of these thresholds at the design stage, typically through iterative interactions between domain experts and ML developers [27].

Tolerable risk

The activity of developing the ML safety requirements may also identify emergent behaviour (potential behaviour of the ML component that could not be identified at the system level). Where the emergent behaviour may contribute to a hazard, safety requirements shall be derived to ensure the emergent behaviour does not arise.

The ML safety requirements resulting from this activity shall be documented ([H])

This activity requires as input the ML safety requirements ([H]) defined in Activity 3. The validity of these ML safety requirements shall be assured with respect to the intent of the allocated system safety requirements in the defined system and environmental context.

The most commonly used approaches for validating ML safety requirements are:
- reviews: domain experts review all documentation to ensure that the specified ML safety requirements for the component will deliver the intended safe system operation.
- simulation: in which a system that obeys the specified ML safety requirements for the component is constructed and the outcomes observed in the simulator for a set of operational scenarios.

The results of the validation activities shall be documented ([J]).

ML safety requirements assurance

Data shall be generated that meets the ML data requirements established in Activity 4. This shall include three separate datasets:

- development data [N]
- internal test data [O]
- verification data [P]

We use the term development data to include training and validation data as it is normally referred to in the ML literature. Development data is used to create a model that is then tested by the development team using the internal test data. Once a model is deemed fit for release by the development team only then is it exposed to the verification data. 

The first two of the above datasets are for use in the development process (Stage 3) whilst verification data is used in model verification (Stage 4).

The generation of ML data will typically consider three sub‐process:

- collection
- preprocessing
- augmentation

*Data collection* shall be undertaken to obtain data from sources that are available to the data collection team which sufficiently addresses the ML data requirements. This may involve reusing existing data sets where they are deemed appropriate for the context, or the collection of data from primary sources.

It may be necessary to collect data from systems that are close to, but not identical to, the envisioned system. Such compromises and restrictions should be stated explicitly in the data generation log with a justification of why the data collected is still valid.

Justifying collected data

A vehicle gathering video data is an experimental variant of the proposed vehicle where variation in vehicle dynamics are assumed to have no impact on the video data with respect to prediction of distance to leading vehicles.

Vehicle gathering video data

Where existing data sources are used, a rationale should be provided in the data generation log as to how these data sources may be transferred to the current domain and any assumptions concerning relevance should be stated explicitly.

Rationale for existing data sources

Where it is impossible to gather real world samples it is common to use simulators. These may be software or hardware in the loop. Where data is collected for such simulators the configuration data should be recorded to allow for repeatable data collection and to support systematic verification and validation of the simulator within the operational context. Such simulators might need to be subjected to a separate assurance or approval process, such as discussed in [58] or similar to tool qualification in the aerospace guidance DO178C [57].

Using simulators

*Data preprocessing* may be undertaken to transform the collected data samples into data that can be consumed by the learning process. This may involve the addition of labels, normalisation of data, the removal of noise or the management of missing features.

Preprocessing of data is common and is not necessarily used to compensate for failures in the data collection process. Indeed the normalisation of data is often used to improve the performance of trained models. Consider for example pixel values in an image in the range [0,255] scaled to be floating point values in the range [0,1].

Preprocessing of data

Handling missing data is particularly important when using clinical health data with missing data rates reported from 30% to 80% [29]. The strategies used to tackle missing data should be stated explicitly in the data generation log with a clear rationale as to why the approach used is commensurate with the system under consideration. When an ML component is used to predict cardiovascular disease many records are found to be missing blood pressure measurements. A preprocessing rationale may be: “Analysis of the data has shown a correlation between the recorded values for patient’s blood pressure and age. Linear regression on the training set is therefore used to impute blood pressure where it is not recorded”.

Handling missing data

A common preprocessing activity is the addition of labels to data. This is particularly important in supervised learning where the labels provide a baseline, or ground truth, against which learnt models can be assessed. Whilst labelling may be trivial in some contexts this may not always be the case. For example, labelling may require a consensus of opinion for use in medical prognosis. In such cases a process to ensure consistent labelling should be developed, documented and enacted.

Labelling data

A set of images from retinal scans may be examined by clinical professionals to provide labels which reflect the appropriate diagnosis. A more advanced labelling process may be necessary if the system is required to identify regions of the image which are to be referred to experts. In this case labelling requires a region of the image to be specified as a closed boundary around those pixels in the image which relate to the region of concern.

Retinal scans

*Data augmentation* shall be undertaken to allow for the addition of data where it is infeasible to gather sufficient samples from the real world. This may occur when the real-world system does not yet exist or where collecting such data would be too dangerous or prohibitively expensive. In such cases, the data sets shall be augmented with data that is either derived from existing samples or collected from systems that act as a proxy for the real world.

The field of computer vision utilises sophisticated models of environmental conditions exist [69] and, by collecting an image of an object under one controlled lighting condition, it is possible to augment the data set with examples of that one object under many simulated lighting conditions. The data generation log should document this augmentation process and provide a justification that the simulated conditions are a sufficient reflection of reality.

Augmenting the dataset

An ML classifier for cancer accepts chest X‐rays of patients. The ML safety requirement states that the classifier should be robust to rotations in the image up to 15 degrees. Each sample in the collected data set may be rotated in 1 degree increments and labelled with the original images labels, appropriately translated.

ML classifier for cancer

*Verification data* is gathered with the aim of testing the models to breaking point. This requires a different mindset for the team engaged with collecting data for verification who are focused not on creating a model but finding realistic ways in which the model may fail when used in an operational system. Furthermore the nature of ML is that any single sample may be encoded into the training set and a specific model found which is able to avoid the failure associated with the sample. This does not mean that the resultant model is robust to a more general class of failure to which the sample belongs. It is imperative therefore that the information concerning verification data is hidden from the developers to ensure the models generated are robust to the whole class of failures and not just specific examples present in the verification data.

The dimensions of variation are not independent and as such combinations of difficult situations are less likely to be included in a data set, used for development, which aims to represent normal operating behaviour. For example, a vehicle is unlikely to be using high beam in foggy conditions on a rainy day where ice is present on the road and a vehicle is approaching on the incorrect side of the carriageway. As such this case, although within the operating domain of the vehicle, is unlikely to be found in the development set. A good verification data set should particularly focus on challenging conditions that are within the operating domain and therefore such a case may be present in the verification data set.

Combinations of variation

The ML data validation activity shall check that the three generated data sets are sufficient to meet the ML data requirements. The results of the data validation activity shall be explicitly documented ([S]). Data validation shall consider the relevance, completeness, and balance of the data sets.

Discrepancies identified between the data generated and the ML data requirement shall be justified. These justifications shall be captured as part of the data validation results ([S]).

Both financial and practical concerns can lead to data sets that are not ideal and, in such cases, a clear rationale shall be provided. For example, a young child crossing in front of a fast-moving car may be a safety concern but gathering data for such events is not practicable.

Financial and practical concerns

The JAAD open dataset [56] is used as development data for an ML component used to detect pedestrians. The cost of gathering and processing data for road crossings is expensive and substantial effort has been undertaken to generate the JAAD dataset. The labelling of pedestrians and range of poses observed is extensive and is clearly relevant for a perception pipeline concerned with the identification of pedestrians. The range of crossings types observed is limited however and a justification may be required as to why this is relevant for the intended deployment.

JAAD open dataset

Validation of *data relevance* shall consider the gap between the samples obtained and the real-world environment in which the system is to be deployed. Validation shall consider each of the sub‐activities undertaken in data generation and provide a clear rationale for their use.

Any simulation used for data augmentation is necessarily a simplification of the real world with assumptions underpinning the models used. Validating relevance, therefore, requires the gaps between simulation and modelling to be identified and a demonstration that these gaps are not material in the construction of a safe system.