How Can a Functional Description in a Manual Be Converted into a Reproducible Test Procedure?
Article 8 of a 12-Part Series on UR E27 Strategies for Marine Equipment Suppliers
※ The product examples, Test Case IDs, setting values, and file names described in this article are illustrative examples created to explain how a reproducible UR E27 Test Procedure can be structured.
The previous article examined how a statement that a function has been implemented in a product can be converted into verifiable Evidence by linking the applicable requirement, product design, Configuration, and test results. Among these elements, the Test Procedure is one of the areas that generates the most rework in actual projects.
Equipment manufacturers may assume that cybersecurity tests can be prepared without significant difficulty by referring to existing FAT procedures or user manuals. A manual, however, explains how to use a product, while an FAT procedure is primarily intended to confirm that the product's operational functions perform normally. UR E27-related cybersecurity tests may need to verify not only normal operation but also access denial, failure handling, potential bypasses, Event logging, and recovery behavior.
How Can a Functional Description in a Manual Be Converted into a Reproducible Test Procedure?
For example, assume that the manual includes the following statement:
"Users can log in to the system by entering an ID and Password."
Testing this statement should not end with confirming that a valid user can log in successfully. It may also be necessary to verify whether accessible functions are differentiated by user role, what restrictions are applied after repeated incorrect Password entries, whether changes in account status are recorded in the Log, and which privileges are required to recover a locked account.
Preparing a Test Procedure is therefore not a matter of converting sentences from a manual into a sequence of test steps. It requires translating requirements into verification objectives and then defining those objectives as repeatable actions and observable results.
This article examines how to define Test Case preconditions, test steps, expected results, acceptance criteria, failure conditions, restoration procedures, and objective test records based on the Product Baseline and the Requirement–Design–Evidence Matrix.
※ In this article, a Test Procedure means a procedure that defines the test target, test environment, preconditions, test steps, expected results, acceptance criteria, and restoration method. A Test Record means a record of the actual actions performed, results obtained, Screenshots, Logs, and determinations made in accordance with that procedure. The specific test scope and classification society witnessing requirements should be confirmed against the applicable classification society's Rules and Guidance and the project-specific approval conditions.
1The Purpose of a Test Procedure Is to Enable Another Tester to Confirm the Same Result
A good Test Procedure should allow a person who did not develop the product to perform the same actions under defined conditions and confirm the same results. A procedure that can be understood only by its author or that depends heavily on the tester's experience lacks reproducibility.
The following statements indicate the general direction of a test but are not sufficient for actual execution.
- Verify the user authentication function.
- Test whether unnecessary Ports are blocked.
- Confirm that Backup and Restore are performed correctly.
- Confirm that Remote Access is securely controlled.
- Confirm that security Events are recorded in the Log.
For example, the statement "test whether unnecessary Ports are blocked" does not identify which Interface on which equipment should be tested, which Ports should be permitted or blocked, which Tool should be used, or whether a lack of response can be interpreted as evidence that the Port has been blocked.
A reproducible procedure should define at least the following information.
- Test equipment and Network Interface
- Source equipment and IP address used to perform the test
- Ports and Protocols to be checked
- Tool and Version used for the test
- Firewall and Service status before testing
- Communications that should be permitted and those that should be blocked
- Acceptance criteria for responses, Timeouts, Resets, and other outcomes
- Logs and Screenshots used to demonstrate the test result
- Settings that must be restored after the test
The quality of a Test Procedure should be judged not by the amount of detail it contains, but by whether another tester can repeat the same test without having to make additional assumptions.
2A Requirement Should First Be Converted into a Verification Objective Rather Than Tested Directly
UR E27 requirements define the capabilities or controls that a product should provide, but a requirement statement does not automatically constitute a single Test Case. One requirement may contain several controls. Some may be confirmed through document review or analysis, while others may require product testing or vessel integration testing.
The test target should therefore be developed through the following sequence in the Requirement–Design–Evidence Matrix.
For example, requirements related to accounts and privileges may be divided as follows.
| Control Implemented in the Product | Verification Objective | Appropriate Verification Method |
|---|---|---|
| Creation of individual user accounts | Confirm that users are individually identifiable and do not share the same common account | Functional test and account-list review |
| Separation of role-based privileges | Confirm that an Operator cannot use administrator-only functions | Positive and Negative Tests |
| Modification of Default accounts | Confirm that the initial authentication information used at shipment does not remain in the delivery Configuration | Configuration review and access test |
| Restriction after repeated login failures | Confirm that further access is restricted after the configured number of authentication failures | Failure-condition test |
| Recording of account changes | Confirm that account creation, deletion, and privilege changes are recorded in the Security Log | Functional test and Log review |
Defining the verification objective first reduces the risk of treating an entire requirement as one vague test.
3Each Test Case Should Include at Least the Following Information
The Test Case format may vary by manufacturer and project. To ensure reproducibility and traceability, however, it should include at least the following fields.
| Field | Information to Be Recorded |
|---|---|
| Test Case ID | Unique identifier that can be linked to other documents |
| Related Requirement | Applicable Rule or Requirement ID |
| Verification Objective | Control to be confirmed through the test |
| Test Target | Product name, equipment ID, and Product Baseline |
| Preconditions | Account, Network, setting, and operating status before testing |
| Test Tools | Equipment, Software, and Tool Version used |
| Test Data | Accounts, files, setting values, and input information |
| Test Steps | Specific actions to be performed by the tester in sequence |
| Expected Results | Product response that should be observed at each step |
| Acceptance Criteria | Objective conditions for determining Pass or Fail |
| Result Records | Screenshots, Logs, Configuration Exports, and measured values |
| Restoration Procedure | Method for returning the product to its normal state after testing |
| Related Evidence | Identification of the Test Record and supporting attachments |
Not every Test Case requires a lengthy explanation. However, the conditions and criteria required to evaluate the result should not be omitted.
4Preconditions Should Include Enough Information to Reproduce the State of the Test Target
Even when the same test procedure is used, the result may differ if the pre-test Configuration is different. Preconditions should therefore record the states that may affect the test rather than merely state that "the system shall be operating normally."
- Product Baseline ID
- Hardware, Software, and Firmware Versions
- Applied Patches and Builds
- Network connection status and IP addresses
- Enabled Ports, Protocols, and Services
- Firewall Rules
- User accounts and roles
- Password Policy and login-failure restriction values
- Time-synchronization status
- Log storage location and remaining capacity
- Backup files and test data
- Activation status of Remote Access and optional functions
For example, if a repeated-login-failure test is performed without first confirming that the account lockout threshold is set to five attempts, blocking the sixth login attempt does not necessarily demonstrate that the designed control operated correctly.
Preconditions are not only a preparation checklist for starting the test. They are also Evidence of the Configuration in which the test result was generated.
5Test Steps Should Separate the Action Performed from the Result to Be Confirmed
A common problem in Test Procedures is combining the action and the result in a single sentence.
For example, the following statement is difficult to use as a specific test step:
"Confirm that a general user cannot access the administrator menu."
It is better to separate the action and the result into individual steps, as follows.
- 1Log in using a test account assigned the Operator role.
- 2Select the user-management menu.
- 3Confirm that the system restricts the display or execution of the menu.
- 4Attempt to access the same function through a direct URL or shortcut.
- 5Confirm the access-denial message and the corresponding Security Log.
- 6Log in using an Administrator account and confirm that the same menu can be accessed normally.
This procedure verifies both the Operator's denied access and the Administrator's authorized access. If only access denial is tested, a function that is disabled for all users could be incorrectly interpreted as a correctly implemented privilege control.
6Expected Results and Acceptance Criteria Should Be Defined Separately
An expected result describes how the product should respond at each test step. An acceptance criterion defines the condition used to determine whether that result constitutes a Pass or Fail.
If these two elements are not distinguished, the test may be left with a subjective determination such as "operates normally."
| Category | Insufficient Expression | Specific Expression |
|---|---|---|
| Login-failure restriction | Access is blocked after repeated failures | Further login attempts for the account are restricted for 10 minutes after five consecutive failures |
| Remote Access | Connection is permitted only when approved | No Session is created without a registered approval number, and the connection attempt is recorded in the Security Log |
| Log generation | The relevant Event is recorded | The designated Log records the user ID, Event type, result, Source equipment, and Timestamp |
| Backup | The Backup is created successfully | A Backup file containing the designated configuration files and Application data is created, and its Hash is recorded |
| Restore | The previous state is restored | The previous accounts, settings, and communication status are restored, and the main operational-function tests are passed |
| Software Update | Only approved Software is installed | Installation of a Package that fails integrity verification is rejected, and the failure Event is recorded |
Acceptance criteria should, where possible, be expressed in terms of observable states, setting values, times, attempt counts, or recorded information.
7Cybersecurity Testing Should Verify Failure, Bypass, and Recovery Behavior as Well as Normal Operation
A conventional functional test focuses on whether the intended function is performed under normal inputs and conditions. Cybersecurity testing should also determine whether the control remains effective under unexpected conditions.
| Test Direction | What the Test Confirms | Account-Management Example |
|---|---|---|
| Positive Test | Does the function operate normally for an authorized user under permitted conditions? | An Administrator creates a user account |
| Negative Test | Is an unauthorized user or input rejected? | An Operator's attempt to create an account is denied |
| Bypass Test | Can the control be bypassed through another path or method? | Access to an administrator function is attempted through a direct URL or Engineering Tool |
| Failure Test | Does the product maintain a secure state during a communication failure, storage-capacity shortage, or authentication failure? | Unauthorized access is not permitted when the central authentication Server connection is lost |
| Recovery Test | Can the product be restored safely to its normal state after a failure? | An authorized administrator unlocks a locked account and confirms the associated records |
It is not necessary to apply all five test directions mechanically to every security function. The necessary directions should be selected based on the product architecture and associated risks. However, if the test scope covers only normal usage scenarios, it may not be possible to determine how the control behaves during an actual incident or failure.
8Account and Privilege Tests Should Verify Privilege Boundaries Rather Than Merely Successful Login
Account-management testing often ends after confirming that a valid account can log in successfully and an incorrect Password is rejected. The more important objective, however, is to verify that permitted and restricted functions are correctly differentiated by user role.
For example, if the product has Operator and Administrator roles, the following items may be tested together.
- Screens and commands permitted for each role
- Privileges to change setting values
- User-account management privileges
- Software Update privileges
- Privileges to delete Logs or change Log settings
- Backup and Restore privileges
- Engineering Tool access privileges
- Recording of attempts to exceed assigned privileges
In actual testing, the privileges defined in the User and Privilege Matrix should be linked to the relevant Test Cases. Functions permitted to an Operator should execute normally, while Administrator-only functions should be restricted through menus, direct paths, and external Tools.
This approach verifies the privilege Matrix through actual product behavior rather than leaving it as a role table that exists only in the documentation.
9Remote Access Testing Should Verify the States Before, During, and After the Connection
If a Remote Access test is limited to confirming that a connection can be established, it cannot verify the approval, recording, and termination controls. The full Remote Access flow should therefore be tested in three stages.
Before Connection
- Is Remote Access disabled by default?
- Is it impossible to create a Session without prior approval?
- Are the authorized user, equipment, and access path defined?
- Are the permitted connection period and target systems restricted?
During Connection
- Are user authentication and privilege restrictions applied?
- Are the connection start time, user, and Source recorded?
- Is access to systems outside the approved target prevented?
- Can the Session be forcibly terminated in an emergency?
After Connection
- Is the connection end time recorded?
- Does the Remote Service return to the disabled state?
- Are no temporary accounts or active Sessions left behind?
- Can the changes performed and files transferred during the Session be traced?
For example, even if an authorized Service Engineer's remote connection appears to terminate normally, the access control cannot be considered fully closed if a Background Service continues to run and an external connection Port remains open. The Service, Process, Port, and Log states should therefore be verified together with the Session status displayed on the screen.
10Backup and Restore Should Be Defined as Separate Test Cases
The creation of a Backup file does not demonstrate that recovery is possible. Backup and Restore have different objectives and failure conditions and should therefore be defined as separate Test Cases.
A Backup test may verify the following.
- Data and settings included in the Backup scope
- Privileges required to perform the Backup
- Storage location and access controls
- Filename, creation time, and Version information
- Integrity-verification method
- Indications and Logs generated when a Backup fails
- Retention period and management of previous files
A Restore test should modify the actual product state, use the Backup to return it to its previous state, and then verify the product functions.
For example, the following procedure may be used.
- 1Record the accounts, Alarm setting values, and communication settings before testing.
- 2Create a Backup of the current Configuration and record its Hash.
- 3Add a test account and change the Alarm and communication settings.
- 4Confirm that the changes have been applied to the product.
- 5Perform a Restore using the generated Backup.
- 6Confirm that the test account has been removed and the previous settings have been restored.
- 7Confirm that communication with the upper-level system and the main operational functions operate normally.
- 8Confirm that the Restore operation and result are recorded in the Log.
The test should not end when a message indicates that the Restore has been completed. The restored Configuration and the actual operational functions should also be verified.
11Security Log Testing Should Verify the Reliability of the Recorded Content and Time as Well as Event Generation
A statement that the product generates Logs does not establish whether security Events can be investigated or traced. Each Test Case should perform a specific action and then confirm that the corresponding Log is generated.
- Successful and failed logins
- Account creation, deletion, and privilege changes
- Administrator-setting changes
- Software Updates
- Backup and Restore operations
- Remote Access start and end
- Attempts to change or delete Log settings
- Security Service start and stop
At a minimum, the Log should contain the Event type, user or Process, occurrence time, Source, target, result, and necessary details.
Time synchronization should also be verified. For example, if an Event is recorded at 10:03 on the Operator Station, 10:07 on the PLC Gateway, and 10:10 on the Remote Access Gateway, it becomes difficult to reconstruct the sequence of a connection and subsequent configuration change.
Time-synchronization testing should therefore verify more than successful NTP connectivity.
- Difference between the current time on each device and the reference time
- Product behavior when the NTP connection is lost
- Event generated when the time is changed
- Privileges required to change the time
- Time-correction method after reconnection
- Effect on the Timestamps of previous Logs
12Software Update Testing Should Verify the Handling of Integrity Failures as Well as Successful Installation
A Software Update test should not be limited to confirming that an approved Package can be installed successfully. It should also confirm that a modified or incorrect Package is rejected and that the product maintains a secure state after an installation failure.
The following Test Cases may be considered.
- Installation of an approved Update Package
- Attempted installation of a Package with an incorrect Checksum or electronic signature
- Attempted installation of a Package intended for another Model or Version
- Update attempt by an unauthorized user
- Power or communication interruption during the Update
- Rollback to the previous Version after an Update failure
- Verification of the Version and Configuration before and after the Update
- Logging of the Update Event and the person who performed it
If a product is updated through USB media, restrictions on unauthorized storage devices, media-entry inspection, Malware scanning, and usage records should also be verified as separate controls.
13Screenshots and Logs Should Be Retained as Result Records Linked to the Relevant Test Case
Screenshots and Logs are objective Records supporting the test result. If the files are merely attached without context, however, it may be difficult to determine what result they demonstrate.
Each result record should, where practicable, be linked to the following information.
- Evidence ID
- Test Case ID and test step
- Test equipment and Version
- Date and time recorded
- Action performed
- Result being verified
- Location on the screen or in the Log that should be reviewed
- Location where the original file is retained
- Whether sensitive information has been masked
For example, submitting only a screen showing an account-lockout message does not establish how many failed attempts triggered the restriction. The Test Record should link the time of each login attempt, the number of failures, the time the lockout began, the account status, the generated Log, and the account-unlock result.
Result files should also be named so that the relevant Test Case and step can be identified. A name such as TC-AC-03_STEP05_Account-Locked_20261018 is more useful than Screenshot1 or Log-Final.
14Failures and Deviations Identified During Testing Should Not Be Deleted
Pre-testing or FAT may reveal behavior that differs from the expected result. If the failure record is deleted and only the successful result after correction is retained, it becomes difficult to determine how the product was corrected and verified.
When a test failure occurs, at least the following information should be recorded.
- Related Test Case and test step
- Expected and actual results
- Product Baseline and Configuration at the time of testing
- Cause of the failure
- Area in which the problem occurred, such as product functionality, settings, or the Test Procedure
- Corrective action
- Other requirements and Test Cases affected
- Scope of required regression testing
- Retest result
- Final determination and approver
For example, if a Software modification is made to stop a Service that remained active after a Remote Access Session ended, it may not be sufficient to repeat only the connection-termination test. It should also be confirmed that the modification has not adversely affected normal remote-support reconnection, Log generation, or emergency-termination functions.
A Deviation should also be recorded when the test is performed differently from the approved procedure. If a change in the Test Tool Version, test sequence, or Network affects the result, the validity of the result should be evaluated separately.
15The Scope of Product Testing, FAT, and Vessel Integration Testing Should Be Distinguished
Not every security function can be verified during a single FAT. Some controls can be tested at the product level, while others can be confirmed only after the product is connected to the shipyard-provided Network and other vessel systems.
| Test Category | Primary Purpose | Test Examples |
|---|---|---|
| Product-Level Test | Verify the product's own security functions and settings | Role-based privileges, login-failure restrictions, Local Logs, and Update integrity |
| FAT | Verify the actual project-specific supply Configuration and its major Interfaces | Project account settings, Backup and Restore, Gateway communication, and Remote Access functions |
| Vessel Integration Test | Verify connections to the vessel-wide security architecture and the associated responsibility boundaries | Central Log Server, NTP, vessel Firewall, and common Remote Access Gateway |
| Post-Delivery Verification | Confirm the As-Built Configuration after installation and commissioning | Final IP addresses, accounts, enabled Services, Firewall Rules, and Remote Access status |
For example, the product's ability to forward Syslog messages can be verified at the product level. Whether the central Log Server on a specific vessel actually receives and retains those Events, however, should be confirmed through an integration test after the vessel Network has been configured.
When allocating the test scope, it is not sufficient to state only that an item is "the shipyard's responsibility." The function to be provided by the product, the conditions to be prepared by the shipyard, the party responsible for performing the test, and the final Evidence should be allocated specifically.
16Internal Pre-Testing Should Be Completed Before a Classification Society or Shipyard Witness Test
A witness test should not be the first occasion on which the product functions are checked. The Test Procedure, Test Configuration, and expected results should be verified internally before the classification society or shipyard witnesses the results.
At a minimum, the following items should be confirmed before the witness test.
- Is an approved or reviewed Test Procedure being used?
- Does the Product Baseline of the test target match that of the delivery product?
- Have the test accounts and privileges been prepared?
- Are the necessary Test Tools and Licenses available?
- Have the Network, Log Server, and NTP connections been prepared?
- Have the test data and Backup files been prepared?
- Have the expected results of each Test Case been confirmed internally?
- Has the post-test restoration procedure been verified?
- Have the storage locations for Screenshots and Logs been defined?
- Have the suspension criteria and response procedure for a test failure been defined?
- Have the Test Cases requiring witnessing been distinguished from items that may be verified through document review?
If issues discovered during internal pre-testing are not corrected before the witness test, the resulting burden may extend beyond the test failure itself to schedule rearrangement, repeated classification society attendance, and FAT retesting.
17Seven Common Mistakes When Preparing a Test Procedure
- 1 Using a Sentence from the Manual Directly as a Test StepA statement such as "an account can be created" does not identify who may create it, under what conditions it may be created, or which records should remain afterward.
- 2 Recording the Test Result Only as "Normal" or "Pass"Without the actual result and the supporting Screenshot, Log, or Configuration Record, the basis for the determination cannot be verified.
- 3 Testing Only the Normal Method of UseThe test confirms only normal access by an authorized account and does not examine unauthorized users, incorrect inputs, repeated failures, or bypass paths.
- 4 Failing to Record the Version of the Test TargetIf the tested Software differs from the delivered Software, the applicability of the existing test result must be reassessed.
- 5 Including Too Many Controls in a Single Test CaseIf login, privileges, Logs, Remote Access, and Backup are combined in one Test Case, it becomes difficult to determine where the failure occurred and which requirements are affected.
- 6 Failing to Prepare a Restoration ProcedureThe product may not return to its normal operating state after account lockout, Firewall modification, Service termination, or Restore testing.
- 7 Excluding Failed Test Records from the Final ResultInitial failures, corrective actions, and retest results are important Evidence of the product-change and verification process.
18In Practice, a Test Procedure Can Be Prepared in the Following Sequence
- 1Establish the Product Baseline and project-specific supply Configuration.
- 2Identify the controls to be verified through testing in the Requirement–Design–Evidence Matrix.
- 3Define the verification objective for each control.
- 4Distinguish the scope of product-level testing, FAT, and vessel integration testing.
- 5Select the necessary test directions from Positive, Negative, Bypass, Failure, and Recovery Tests.
- 6Assign a Test Case ID and link it to the related requirements and Design Evidence.
- 7Define the test target, preconditions, Test Tools, and test data.
- 8Separate the actions and expected results for each test step.
- 9Define objective acceptance criteria for determining Pass or Fail.
- 10Define how Screenshots, Logs, and Configuration Records will be obtained.
- 11Prepare the post-test restoration procedure and the method for confirming restoration.
- 12Perform internal pre-testing and confirm the reproducibility of the procedure.
- 13Record identified failures and Deviations and perform impact analysis and retesting.
- 14Reflect classification society or shipyard Comments and finalize the Test Procedure and result records.
- 15Confirm differences among the Test Configuration, delivery Configuration, and As-Built Configuration.
The central principle of this sequence is not to prepare the Test Procedure first and then fit the requirements into it. The verification objective should be derived from the applicable requirement and the control implemented in the product, and a Test Case should then be prepared to confirm that objective.
19Conclusion: A Good Test Procedure Makes the Product's Security Behavior Predictable
In UR E27-related cybersecurity testing, the number of Test Cases is not the most important consideration. It should be clear which requirement and control each test verifies, under which Configuration and conditions it was performed, and what Evidence supports the Pass or Fail determination.
A reproducible Test Procedure should maintain the following chain.
When this chain is maintained, the affected product functions and requirements can be identified even when a test fails. If the product changes, the manufacturer can review the changed functions and related Test Cases instead of repeating every test from the beginning.
A good Test Procedure should meet the following conditions.
- A tester who did not develop the product can perform the same procedure.
- The Product Baseline and Configuration of the test target are clear.
- Actions and expected results are separated at each test step.
- Acceptance criteria are defined objectively.
- Normal operation and, where necessary, failure, bypass, and recovery behavior are verified.
- Screenshots and Logs are linked to the relevant Test Case and test step.
- Test failures, corrections, and retest histories are retained.
- Consistency between the Test Configuration and the actual delivery Configuration can be confirmed.
- Responsibility boundaries between product testing and vessel integration testing are defined.
A statement in a manual that "the function is provided" is only the starting point of testing. It becomes verifiable Test Evidence only when the manufacturer can repeatedly confirm that the function operates as intended under defined conditions, is safely restricted under unauthorized conditions, and can be restored to its normal state after a failure.
The next article will examine how to address Gaps identified during testing or classification review.
Part 9 will address the question, "How can an unmet requirement be converted into a manageable Gap rather than concealed?" It will examine the distinction between an unmet requirement and an exemption, the separation of product improvements from external security measures, the design of Compensating Controls, residual risk and responsibility allocation, materials for consultation with the classification society and shipyard, and the preparation of Gap Closure Evidence.
How Can an Unmet Requirement Be Converted into a Manageable Gap Rather Than Concealed?
Article 9 will examine the distinction between an unmet requirement and an exemption, the separation of product improvements from external security measures, the design of Compensating Controls, residual risk and responsibility allocation, materials for consultation with the classification society and shipyard, and the preparation of Gap Closure Evidence.
Related Reading:
- Chapter 4: Understanding Why IACS Introduced E26 and E27 — The Purpose Behind the Requirements
- Is a SoC-Centric Approach Enough for Ship Cybersecurity? — Statement of Compliance vs. Type Approval
- Anatomy of a Ship ZCD — Understanding the Building Blocks of an IACS UR E26 Zone and Conduit Diagram
- Why Most Ship ZCDs Fail — Seven Common Mistakes in IACS UR E26/E27 Projects
IT/OT Integrated Cybersecurity specialist with expertise in IACS UR E26/E27 compliance, N2SF, Cybersecurity Strategy & Governance, BPR, and Digital Transformation. Former IT Consultant at KPMG (2019–2022). M.S. Information Systems, University of Maryland, Robert H. Smith School of Business.
⚓ Join the ShipPaulJobs Community
Join →
A crucial reminder that cybersecurity must go beyond formal approvals. Functional Descriptions should serve as practical engineering guides, helping shipowners and shipbuilders translate regulatory requirements into real cyber resilience.
ReplyDeleteCompliance is the starting point, not the finish line.