FAT and SAT tests on industrial devices

Customers, manufacturers, integrators and maintenance personnel have focused their efforts on ensuring the reliability of the systems in the production process and on the automation and control systems that composes it. Commitments made by the manufacturer to support its products for a certain time, scheduled maintenance or contractual agreements of the systems’ operational requirements, in each specific case between the customer and the manufacturers or integrators, are several examples of what has been done to ensure this reliability.

Among the latter are the FAT tests (Factory Acceptance Test) and SAT (Site Acceptance Test), which are focussed on verifying the proper operation of the systems programmed and developed by the manufacturer and their fulfilment of the specifications agreed in the contract.

What are the FAT and SAT acceptance tests?

FAT and SAT tests are an important part of the execution of an industrial project and its installation. During this test period, a verification is made that the system works according to the specifications set out by the customer and to the product-specific protocols in the final installation.

These tests are run using a predefined list of procedures to guide operators on what data to use, the steps to follow and the expected results from each. The results obtained will have to be compared with the expected data.

A test will have been successfully completed when the results obtained fulfill the requirements for each or when the number of tests not fulfilled does not exceed the limit set. It must also be taken into account that, once the tests are successfully completed, the tested system may be accepted or rejected according to the conditions agreed between the customer and the manufacturer.

FAT (Factory Acceptance Test)

The main aim of the FAT test is to inspect the system according to the specifications indicated by the customer immediately after its conception.

The measures adopted in the FAT test include:

• Monitoring the proper functioning of the system.
• Verifying the quality in the system manufacturing process by inspecting the software, assembly and programming, system configuration, etc.
• Studying the compliance of the equipment and its performance according to the specifications indicated by the customer upon its purchase.
• Analysing the technical documentation, in order to check the consistency of this with the characteristics of the system and its installed components.

In short, the aim of the FAT is to get approval in order to deliver the system to the customer and install it at its location.

Normally, these tests are conducted at the manufacturer’s premises with the customer or somebody appointed by him, so that it is possible to give them first-hand knowledge about the functionality of the system.

It is important to keep in mind that there are several levels within the FAT tests. They can be conducted at a very basic level, such as configuring the main parts of the system with temporary wiring, making sure that everything works as it should, or more fully, where the manufacturer builds a complete system in order to test it for real. In the latter case, the system is commonly disassembled, moved to the customer’s location and reassembled once the tests are finished.

- Figure 1: Requirements and results of a FAT test. Source: TÜV Rheinland -

SAT (Site Acceptance Test)

Once all system components have been factory tested and approved by the customer, they will be installed at the customer’s location. The SAT tests are carried out once such installation has finished and after the initial configuration. The purpose of these tests is to check that the system in question is perfectly integrated into the site layout with respect to its final position, verifying that it meets all the operating requirements. The SAT tests may include some tests already done during the FAT phase, but under more real conditions.

During this test, a start-up process is also conducted in which all the physical inputs and outputs of each PLC or control device are commonly checked.

Moreover, in SCADA projects, the SAT tests usually consist of a period of continuous operation lasting one or two weeks that must end without major errors. If an error occurs during this period, the parties involved must meet to discuss the changes needed.

Passing these tests entails the final acceptance of the system and a process guarantee test.

The checks that are usually performed during SAT tests include:

• Performing a visual check of the final system and its components.
• Verifying the functionality of the utilities and the configuration made.
• Checking the internal pressure of the devices, as well as the ventilation.
• Monitoring the safety devices.
• Training the operators.
• Checking the mechanical and software system.

- Figure 2: Requirements and results of SAT tests -

What about cybersecurity in the FAT and SAT tests?

The current acceptance tests have been very effective for many years in guaranteeing the proper functioning of ICS devices. However, the convergence with the IT world and the cybersecurity incidents that it involved have meant that they are no longer enough to guarantee the continuity of the devices assessed.

FAT and SAT rarely include cybersecurity assessment tests on ICS devices, which is a significant omission, since in order to keep cybersecurity at an acceptable level, it must be tested throughout the life cycle of devices and systems. Therefore, it is necessary that such tests include specific cybersecurity assessments.

There are already some trends that argue for the creation of a new set of practices called CAT (Cybersecurity Acceptance Testing). These tests would be carried out after FAT and SAT and they would include technical audits verifying that the device complies with good practice requirements and guidelines and regulations, such as IEC 62243 or NERC CIP, among others. The elements to take into account in these tests include:

• Installation of the latest patches and versions of firmware/software.
• Correct configuration and bastioning of the devices.
• Recommendation of security devices, such as firewalls, to include in the system.
• Conduct penetration tests to identify existing vulnerabilities. These types of tests must not be conducted when the system is in production.

Conclusion

As we have seen in this article, acceptance tests are very important to ensure that the systems purchased fulfill all the requirements set out in the contract and, therefore, that they work as intended. However, with increasing attacks on ICS systems, it is vital to include new evidence related to the cybersecurity of devices and systems. It is the task of all companies concerned in ICS, contractors and manufacturers to start to develop and include aspects related to it in current tests.