Why Did a Product Declared “Ready for Delivery” Come to a Halt Just Before FAT?
Article 2 of a 12-Part Series on UR E27 Strategies for Marine Equipment Suppliers
※ The company, product, schedule, and classification review details in this article are fictional and have been created to illustrate problems that may arise during UR E27 implementation.
A marine equipment manufacturer received a request for quotation from a shipyard. The proposed product was a network-based control system that monitored major onboard equipment and transmitted certain control commands. The manufacturer had supplied similar products to numerous vessels and already held an existing Type Approval.
The RFQ included the following statements:
- Applicable IACS UR E27 requirements shall be complied with.
- Cybersecurity-related documents shall be submitted in accordance with the shipyard and classification society requirements.
- Required certificates or equivalent evidence shall be provided.
After reviewing the RFQ, the sales representative responded as follows:
The response did not appear obviously incorrect. However, the following matters had not been verified internally at the time:
- Whether the existing Type Approval covered UR E27 conformity
- The exact hardware and software versions of the product being quoted
- The differences between the standard product and the project-specific supply configuration
- The actual implementation level of remote access, account management, logging, and backup functions
- The scope of available design documentation and test evidence
- Whether TA, SoC, SoF, or project-specific verification would be required
- Whether additional development and testing would be needed to satisfy the requirements
The quotation-stage response consisted of only two sentences. Yet those two unverified sentences later became a de facto commitment that affected the project's technical conditions, schedule, and cost.
1. Quotation Stage: “We Can Comply” Becomes a Contractual Expectation
The shipyard's procurement team compared the suppliers' prices, delivery schedules, technical suitability, and approval status. The manufacturer presented its previous delivery record and existing Type Approval and stated that it could comply with UR E27. The shipyard therefore understood that the required documentation could be submitted without extensive product modification.
The problem was that the two parties interpreted “we can comply” differently.
The equipment manufacturer understood it to mean:
The shipyard, however, understood it to mean:
This difference did not become apparent immediately after the contract was signed. Discussions initially focused on price, delivery, and the basic technical specification. Cybersecurity matters were included in the list of Vendor Documents to be submitted later.
The manufacturer was selected as the preferred supplier and subsequently awarded the contract. Delivery was scheduled for 11 months later, with FAT planned two months before delivery. The product design and major component orders had to be finalized much earlier.
Even at this stage, the company had no designated person responsible for overall UR E27 compliance. The sales team assumed that the development team would prepare the technical documentation. The development team assumed that the quality team would manage the classification submissions. The quality team planned to request information from each department once the shipyard finalized the document requirements.
2. Immediately After Contract Award: A Single Requirement Expands into Multiple Documents
After contract award, the shipyard issued its Vendor Document Requirement. In addition to the drawings and manuals normally submitted, it included the following cybersecurity-related documents:
- Product and system configuration description
- Hardware, operating system, firmware, and application software versions
- Network architecture diagram and external interface list
- List of Ports, Protocols, and Services
- User account and privilege management method
- Security configuration and Hardening guide
- Security Event and Log generation capabilities
- Remote access architecture and control method
- Backup and Restore procedures
- Malware protection method
- Vulnerability and security update management procedure
- Requirement-by-requirement cybersecurity conformity statement
- Relevant test procedures and test results
- Applicable certificates or equivalent verification evidence
The sales team forwarded the list to the development team and asked them to prepare the documents using existing materials. At first, the development team did not expect any significant difficulty. Network diagrams and user manuals already existed, the operating system supported user IDs and Passwords, and the product had functions for saving and restoring data.
Once document preparation began, however, a series of questions emerged:
- Should the maintenance laptop connection Port also be treated as an external interface?
- Should the management Port of the internal Ethernet Switch be shown?
- Was the supplier-installed remote support application a standard or optional function?
- Would the Windows administrator account be handed over to the shipyard or retained by the supplier?
- Could operational logs and security logs be distinguished?
- Did the Backup function save only configuration data, or could it also recover the operating system and application software?
- Was there a procedure for returning the product to a normal state if Restore failed?
- Was there test evidence showing that unnecessary Services had been disabled?
- Which department would assess a newly identified vulnerability and notify the customer?
The existing documents had been prepared to explain how to install and operate the product. They could demonstrate that the product performed its intended operational functions, but they did not systematically explain how cybersecurity requirements had been implemented and verified.
The manufacturer eventually requested an extension to the submission deadline. The shipyard accepted the request on the condition that certain documents be submitted first and the remainder supplemented later.
The project experienced its first schedule change.
3. Design Submission: Differences Between the Product and Its Documentation Begin to Surface
The manufacturer's initial network architecture diagram showed the main control panel, operator Station, internal Switch, and connection to the shipyard's upper-level network. However, when the shipyard reviewer compared the architecture diagram with the product manual, several inconsistencies became apparent.
First, the product manual described an external connection for remote support, but this path was not shown in the network diagram.
Second, a Service Port used to connect a maintenance laptop had been omitted from the drawing.
Third, USB storage devices could be used to update the application software, but neither the conditions for using removable media nor the relevant controls were described.
Fourth, the internal Switch had a management Web Interface and a default administrator account, but it was unclear whether the default credentials had been changed or unnecessary Services disabled.
Fifth, the project-specific supply configuration included a customized Gateway that was not part of the standard product, but the scope of the submitted Type Approval did not include this Gateway.
The shipyard requested the following additional information:
- A revised architecture diagram showing all physical and logical interfaces
- The connected system, communication direction, Protocol, and purpose of each interface
- The conditions under which remote access would be enabled and the applicable approval process
- Controls governing USB Port use
- Evidence that default accounts and unnecessary Services had been addressed
- Confirmation of whether the customized Gateway was covered by the existing approval
- The exact configuration and version of the product to be supplied
What had started as a simple drawing revision expanded into a product configuration verification exercise. The development team had to recheck the BOM for the actual Panel, the software deployment list, Switch settings, and Gateway version.
During this process, the company also discovered that the versions recorded in the sales proposal, development configuration list, procurement records, and installation manual were inconsistent.
The problem could no longer be resolved simply by correcting the documents. The company first had to determine which configuration would constitute the actual delivery Baseline.
4. Classification Comments: The Question Shifts from “Does the Function Exist?” to “How Can It Be Demonstrated?”
The shipyard submitted the revised documentation to the classification society. The society responded that merely stating that security functions existed was insufficient and requested more specific evidence.
For example, the manufacturer described account management as follows:
The classification comments raised questions such as:
- How are user roles differentiated?
- What is the difference between operator and administrator privileges?
- Are default accounts changed after installation?
- Who is responsible for creating, modifying, and deleting accounts?
- Is access restricted after repeated failed login attempts?
- Can the Password policy be configured or modified?
- Have all access paths subject to authentication been identified?
- Are test procedures and results available to demonstrate these functions?
A similar issue arose with logging. The manufacturer stated that the system retained Event Logs, but the actual logs were primarily intended to record equipment operating conditions and Alarms.
The following cybersecurity-related activities were either not recorded or could not be distinguished:
- Successful and failed user logins
- Changes to administrator settings
- Account creation and privilege changes
- Software updates
- Remote access start and end times
- Deletion of audit logs or changes to logging settings
Backup and Restore presented the same problem. The product included a data Export function, but this function merely copied part of the operational history to an external storage device. It could not reasonably be treated as a complete Backup and Restore process capable of recovering the system to a normal operating state after a system failure or malware infection.
From this point onward, responding to the comments was no longer a documentation exercise. It became an investigation into how the product actually operated.
5. A Belated Gap Analysis: Distinguishing Missing Documentation from Missing Product Functions
The manufacturer brought together external specialists and internal development and quality personnel to conduct a requirement-by-requirement Gap Analysis. The findings fell into three categories.
First, Functions Existed, but Documentation and Test Evidence Were Insufficient
- Role-based user privileges were implemented, but no privilege Matrix existed.
- Firewall settings had been applied, but the rationale for the permitted Rules had not been documented.
- Remote access records were retained, but neither the retention period nor the review responsibility had been defined.
- Certain security settings were applied, but there was no procedure for verifying them during installation and handover.
- A configuration Backup function existed, but no Restore test record was available.
These items could be addressed without developing entirely new functions, provided that the manufacturer prepared the design descriptions, configuration standards, and test procedures and then performed the necessary tests.
Second, Configuration Changes or Limited Software Modifications Were Required
- The same default administrator account and Password were used across multiple products.
- Unused Services remained enabled.
- USB Ports could be used without separate authorization.
- Failed login attempts were recorded, but there was no restriction after repeated failures.
- The remote support application was configured to run continuously.
- Administrator configuration changes were not adequately captured in the security logs.
These appeared to be simple configuration changes, but the installation procedure, service procedure, and user manual also had to be revised.
Third, Full Compliance Would Be Difficult to Achieve in the Short Term
- The existing application architecture made granular privilege separation difficult.
- Certain components could not transmit security Events to an external system.
- The company had no established policy for vulnerability notification or the duration of security update support.
- The product used an operating system approaching end of life, making long-term security patch availability uncertain.
- The customized Gateway was developed and verified outside the established product development process.
These issues required product modification, compensating controls, clearer allocation of responsibilities, or adjustment of the approval scope. The supplier could not independently declare them “compliant” without consulting the shipyard and classification society.
The Gap Analysis finally established the product's actual condition. However, it was conducted five months after contract award. The major design decisions had already been finalized, and Panel production had begun.
6. Product Modification: One Small Security Function Changes Multiple Deliverables
The manufacturer decided to address the items that could still be modified within the project schedule.
- Enforce a change of default credentials
- Disable unnecessary Services
- Disable remote access by default
- Require administrator approval and logging for remote access
- Add logs for failed login attempts and configuration changes
- Apply restrictions to USB use
- Develop a security configuration Checklist
- Improve the Backup and Restore procedures
At first glance, these appeared to be a handful of configuration changes and software enhancements. Their actual impact, however, was much broader than expected.
For example, disabling remote access by default and permitting it only through an approved process required corresponding changes to:
- The network architecture diagram
- Firewall Rules
- The remote support procedure
- User and administrator manuals
- Service engineer work instructions
- The account and privilege Matrix
- The Event and Log list
- Test procedures
- Customer handover documentation
Restricting USB use affected the existing maintenance process through which field engineers installed software and collected logs. Completely disabling USB access would make servicing difficult. Allowing limited use required a procedure for authorization, inspection, and usage logging.
Adding failed-login records also required the manufacturer to confirm log storage capacity and time synchronization. If the timestamps of different Stations were inconsistent, reconstructing the sequence of events during an incident would be difficult.
7. Approval Scope: The Existing Type Approval Could Not Provide the Complete Answer
The Type Approval presented at the beginning of the project covered basic environmental, functional, and safety-related tests for the standard product. However, it did not clearly establish:
- Whether UR E27 requirements were included within the approval scope
- Whether the relevant software version was covered by the certificate
- Whether the customized Gateway could be treated as part of the same approved product
- Whether the modified remote access and security logging functions belonged to the approved configuration
- Where the boundary lay between the individually approved product and the complete system supplied to the project
The manufacturer had initially expected the existing certificate to be sufficient. However, the fact that the same product name appeared on the certificate did not prove that the entire project-specific supply configuration conformed to UR E27.
The manufacturer therefore had to discuss the following alternatives with the classification society and shipyard:
- Adjust the supply configuration so that it remained within the existing approval scope
- Perform additional verification of the modified product
- Pursue a project-specific SoC or SoF
- Address certain requirements through the vessel's integrated design or shipyard-level controls
- Propose compensating controls for requirements that could not be satisfied immediately
This discussion was delayed not because the company could not decide which of TA, SoC, or SoF was superior. It was delayed because the exact product scope and modifications had not been established early enough.
8. Test Preparation: A Manual Description Is Not a Test Procedure
Two months before FAT, the shipyard asked the manufacturer to incorporate cybersecurity test items into the existing FAT program. The manufacturer began preparing test procedures based on descriptions in the user manual.
A valid test procedure, however, required far more detail than the manual provided:
- Hardware and software versions of the test target
- Network and account configuration before testing
- Equipment and Tools to be used
- Privileges assigned to the tester
- Step-by-step test instructions
- Expected results and acceptance criteria
- Screenshots, Logs, or Records demonstrating the results
- Method for restoring the system after testing
- Actions to be taken if the test failed
For example, a procedure stating only that the tester should “confirm that access is denied when an incorrect Password is entered” would not verify controls against repeated failed login attempts. The test would also need to confirm how many failed attempts triggered a restriction, how long the restriction lasted, whether the Event was recorded, and whether it was visible to an administrator.
A Restore test could not end with confirming that a Backup file had been created. The tester had to modify the actual configuration, restore the previous state using the Backup file, and verify that the system resumed normal operation.
Remote access testing likewise needed to verify more than whether a connection could be established:
- Can a connection be made without prior authorization?
- Can only designated accounts and access paths be used?
- Are the identity of the user and the connection start and end times recorded?
- Can remote access be terminated during an emergency?
- Are any residual accounts or Sessions left after disconnection?
Some of the manufacturer's functions worked under normal operating conditions but had never been tested under failure and recovery conditions.
During internal pre-testing, the Restore process failed to complete correctly, and one Service remained active after the remote access Session ended. Timestamps also differed by several minutes across devices.
Rework was required before FAT.
9. Just Before FAT: A Technical Problem Becomes a Schedule Problem
The revised software was completed three weeks before FAT. It could not, however, be immediately adopted as the delivery configuration.
At a minimum, the manufacturer had to:
- Perform internal functional testing of the revised software
- Conduct regression testing of existing control functions
- Test the cybersecurity functions
- Update the installation Image and deployment Package
- Record the version and Checksum
- Revise the user, installation, and service manuals
- Update the network diagram and interface list
- Revise the test procedures
- Analyze the impact of the changes and confirm the approval scope
- Decide whether the changes would also apply to other units of the same model already in production
The development team had initially considered only the cybersecurity enhancements, but the quality team determined that their impact on the control functions also had to be assessed. Restricting certain user privileges prevented some maintenance functions from operating. Strengthening the Firewall Rules also blocked part of the communication with the upper-level system.
The shipyard proposed maintaining the planned FAT date while recording the incomplete cybersecurity items on a Punch List. However, items requiring classification witnessing or product modification could not easily be treated as simple document corrections.
FAT was therefore divided into two stages.
The first FAT covered the major operational functions, while incomplete cybersecurity items remained subject to separate retesting. An additional test was later conducted on the modified product.
The manufacturer had to reassign developers and test personnel for the second test, while the schedules of the shipyard and classification society also had to be rearranged.
The quotation-stage statement that the company “could comply with the relevant requirements” had now become a tangible cost involving additional development, document revisions, and retesting immediately before FAT.
10. The Actual Losses Identified After Project Completion
The product was eventually delivered after additional corrective actions and testing. It would therefore be inaccurate to describe this case simply as a certification failure. The product was delivered, and the shipbuilding project continued.
Nevertheless, the manufacturer incurred the following losses.
Direct Costs
- External consulting and classification response costs
- Additional software development costs
- Cybersecurity and regression testing costs
- FAT retesting and travel expenses
- Document revision and translation costs
- Costs associated with replacing certain components and the operating system
Schedule Costs
- Design approval delays
- Split FAT and retesting
- Delayed deployment of developers to other projects
- Repeated responses to shipyard inquiries
- Changes to delivery schedules and production plans
Sales and Confidence Costs
- Reduced confidence in the supplier's quotation-stage statements
- Classification as a supplier requiring additional shipyard oversight
- More stringent verification during technical evaluations for subsequent projects
- Reassessment of whether the same product could be used in other projects
Internal Operational Costs
- Disputes over responsibilities among sales, development, and quality teams
- Reorganization of product-specific version and configuration information
- Review of previously delivered products for the same vulnerabilities
- Changes to the service organization's remote support practices
- Establishment of vulnerability and security update management procedures
The largest loss was not the cost of any single additional test. The greater burden came from the urgent involvement of multiple departments at an unplanned stage, disrupting the company's overall priorities.
11. Where Did the Project Go Wrong?
On the surface, the project suffered from numerous classification comments, incomplete security functions, and delayed test preparation. However, if the root cause is reduced to “insufficient product security functions,” the same problem may recur.
12. What Would Have Been Different If a Prepared Manufacturer Had Managed the Same Project?
A prepared manufacturer does not necessarily satisfy every requirement from the outset or hold TA for every product. The difference is that it can distinguish verified facts from matters requiring further confirmation and communicate both clearly.
Had a prepared manufacturer received the same RFQ, it could have taken the following actions.
During the Quotation Stage
- Confirm the applicable vessel and classification society
- Confirm the product's onboard function and integration scope
- Define the hardware and software Baseline to be supplied
- Verify the scope of the existing certificate
- Identify differences between the standard product and the project-specific configuration
- Classify each requirement as fulfilled, partially fulfilled, or requiring further confirmation
- Reflect the necessary additional development, verification period, and cost in the quotation conditions
In that case, the response would not simply have stated, “We can comply.”
For example, the manufacturer could have described the scope and conditions as follows:
Immediately After Contract Award
- Finalize the submission document list and assign responsible personnel
- Develop a Requirement–Design–Test Evidence Matrix
- Prepare matters requiring early clarification with the shipyard and classification society
- Distinguish product Gaps from ship-level integration responsibilities
- Agree on the certification or verification route at an early stage
- Incorporate required functional changes before the product design is frozen
Before FAT
- Confirm that the delivery and test configurations are identical
- Complete pre-testing of the security functions
- Test Backup and Restore and remote access under failure conditions
- Secure objective Records, including configuration values, Screenshots, and Logs
- Confirm version consistency across manuals, drawings, and test procedures
- Obtain prior acceptance of any unmet requirements and compensating controls
13. Conclusion: Small Uncertainties Become More Expensive as the Project Progresses
In this scenario, the problem began with a single statement during the quotation stage:
At the time, the uncertainty appeared minor. As the project progressed, however, it expanded as follows:
In a UR E27 project, costs are not driven only by missing functions. The greater problem is discovering those deficiencies too late.
During the quotation stage, conditions can still be discussed and reflected in the proposal. During early design, the product functions and configuration can still be modified. Before approval documents are submitted, inconsistencies between the product and the documentation can still be corrected. Immediately before FAT, however, the same issue can lead to product rework, retesting, and delivery delays.
The first capability an equipment manufacturer needs is therefore not a list of certificates. It is an internal assessment process that enables the company to distinguish the following when the customer asks its first question:
- Items already satisfied by the current product
- Items requiring additional documentation and test evidence
- Items requiring product modification
- Items requiring agreement on the allocation of responsibilities with the shipyard or classification society
- Items that can be addressed within the existing approval scope
- Items requiring separate certification or project-specific verification
Only when these distinctions can be made does a quotation-stage response become a technically valid commitment.
The next article will address the question that forms the starting point of this assessment: Is Our Equipment Subject to UR E27?
Article 3 will examine how to determine whether a product is actually subject to UR E27 by reviewing its functions, network connectivity, relationship with the vessel's functions, and system boundaries in a structured sequence.
Related Reading:
- Chapter 4: Understanding Why IACS Introduced E26 and E27 — The Purpose Behind the Requirements
- Is a SoC-Centric Approach Enough for Ship Cybersecurity? — Statement of Compliance vs. Type Approval
- Anatomy of a Ship ZCD — Understanding the Building Blocks of an IACS UR E26 Zone and Conduit Diagram
- Why Most Ship ZCDs Fail — Seven Common Mistakes in IACS UR E26/E27 Projects
IT/OT Integrated Cybersecurity specialist with expertise in IACS UR E26/E27 compliance, N2SF, Cybersecurity Strategy & Governance, BPR, and Digital Transformation. Former IT Consultant at KPMG (2019–2022). M.S. Information Systems, University of Maryland, Robert H. Smith School of Business.
⚓ Join the ShipPaulJobs Community
Join →
Comments
Post a Comment