After this lesson you can explain how a Convolutional Neural Network processes spatial data using filters and pooling; describe how Recurrent Neural Networks handle sequences and why they suffer from the vanishing gradient problem; explain how LSTM and GRU address long-range dependencies; and articulate why these architectures were insufficient for language modelling at scale, motivating the Transformer.
From 2012 to 2017, two neural network architectures dominated applied AI: the Convolutional Neural Network (CNN) for spatial data (images, video, spectrograms) and the Recurrent Neural Network (RNN) — particularly its LSTM variant — for sequential data (time series, audio, natural language). Every major AI product of that era — image recognition systems, speech-to-text, machine translation — was built on one of these two architectures or a combination of them.
Understanding these architectures is not historical trivia. CNNs remain the workhorse of computer vision systems deployed in shipyards and ports today (Lesson 4). RNNs still appear in legacy time-series systems for engine monitoring. And the specific limitations of RNNs — their inability to efficiently capture long-range dependencies — is the precise problem the Transformer was designed to solve, which is why understanding RNNs makes the Transformer intuitive rather than mysterious.
1. From Perceptrons to Deep Networks — A Brief Arc
The simplest neural network is a single-layer perceptron: an input layer of n features connected to an output neuron by weighted connections, producing a weighted sum that is passed through an activation function (sigmoid, tanh, or ReLU). A perceptron can only learn linearly separable patterns — a severe limitation discovered by Minsky and Papert in 1969 (Perceptrons, MIT Press).
Adding hidden layers — neurons between input and output that are not directly connected to either — creates a Multi-Layer Perceptron (MLP). With enough hidden neurons and non-linear activation functions, MLPs can theoretically approximate any function (Universal Approximation Theorem, Cybenko 1989). In practice, training deep MLPs was difficult until the 2010s: without GPU acceleration and large datasets, the backpropagation signal vanished through many layers (the vanishing gradient problem), and deep networks failed to converge.
The CNN and RNN were architectural innovations that allowed deeper networks to train reliably on specific data types by introducing structural priors — assumptions about the nature of the data built into the architecture itself.
2. Convolutional Neural Networks — Seeing Like a Machine
A CNN is designed for data with spatial structure — data where neighbouring elements are meaningfully related. In an image, the pixel at position (100, 100) is likely correlated with the pixel at (101, 100). An MLP treats all pixels as independent inputs, ignoring this spatial structure. A CNN exploits it through three key ideas:
Multiple convolutional layers are stacked: early layers detect low-level features (edges, blobs), mid layers detect textures and object parts, later layers detect high-level semantic features (faces, vessel shapes, containers). This hierarchical feature learning is what makes CNNs so effective for vision tasks.
The AlexNet breakthrough (Krizhevsky, Sutskever, and Hinton, 2012) used 5 convolutional layers + 3 fully-connected layers trained on 2 GPUs for 5 days, winning ImageNet with 15.3% top-5 error vs. 26.2% for the runner-up. By 2015, ResNet-152 (He et al., CVPR 2016) introduced residual connections — skip connections that allow gradients to flow directly across many layers — and achieved 3.57% error, below human performance of ~5% on the same benchmark.
3. Recurrent Neural Networks — Processing Sequences
A Recurrent Neural Network (RNN) is designed for sequential data: time series, sentences, audio. Unlike a CNN or MLP which processes a fixed-size input in a single pass, an RNN processes one element of the sequence at a time, maintaining a hidden state — a vector that summarises what the network has seen so far.
At each time step t, the RNN takes the current input x_t and the previous hidden state h_{t-1}, and computes a new hidden state h_t and an optional output y_t. The same parameters are shared across all time steps — another form of parameter sharing analogous to the CNN's filter sharing. The hidden state acts as the network's "memory" of the sequence so far.
RNNs are appropriate for: engine sensor time series (predict the next state of engine RPM, temperature, and vibration given the last N measurements); AIS vessel trajectory prediction (predict next position given history of positions); anomaly detection in sequential operational logs (detect abnormal sequences of events in ECDIS or PMS logs).
Legacy maritime condition monitoring systems deployed in the 2015–2018 period often used LSTM-based anomaly detection. Understanding RNNs is necessary to audit and assess these systems under IACS UR E26 Section 4.3 requirements.
4. The Vanishing Gradient Problem — RNN's Core Limitation
Vanilla RNNs suffer from a critical failure: the vanishing gradient problem (Hochreiter, 1991; Bengio et al., 1994). During backpropagation through time (BPTT), gradients must be multiplied through the weight matrix at each time step. If those weights are less than 1, the gradients shrink exponentially as they propagate back through long sequences. After 20–30 steps, the gradient signal is effectively zero — the network cannot learn from events that occurred early in a long sequence.
For language modelling, this meant that vanilla RNNs could only effectively learn from the last ~5-10 words of context. For sentence-level translation, document summarisation, or long-term time-series forecasting, this was a severe limitation.
The Gated Recurrent Unit (GRU) (Cho et al., 2014, arXiv:1406.1078) is a simplified variant of LSTM with two gates instead of three, slightly fewer parameters, and comparable performance on most tasks. GRUs are faster to train and are preferred in resource-constrained settings.
By 2016, LSTM-based sequence-to-sequence models (Sutskever, Vinyals, and Le, 2014) achieved state-of-the-art results in machine translation and speech recognition. Google replaced its statistical machine translation system with an LSTM-based neural system in November 2016 — translating 100 billion words per day.
5. The Remaining Limits of RNNs — Why the Transformer Was Necessary
LSTMs solved the vanishing gradient problem but retained two structural limitations that became critical at the scale of modern language models:
⚠️ Sequential Computation
An RNN processes one token at a time, in order. To process position 500 of a sequence, positions 1–499 must be computed first. This means RNN training is inherently sequential and cannot be parallelised across the sequence length — making it slow on GPUs, which are designed for parallel computation. Training on billion-token corpora took impractically long.
⚠️ Fixed-Size Bottleneck
In sequence-to-sequence models (e.g., translation: English sentence → French sentence), the entire source sentence was compressed into a single fixed-size hidden state vector, which the decoder then expanded. For long sentences, this bottleneck lost information. The attention mechanism (Bahdanau et al., 2014) was introduced to allow the decoder to look directly at all encoder hidden states — a partial fix that became the seed of the Transformer.
RNN에서 Transformer로의 전환이 LLM 아키텍처에 어떤 구조적 변화를 가져왔는지, 그리고 현재 LLM에서 나타나는 AI 윤리 문제를 심층적으로 분석합니다.
Transformer 기반 LLM이 단순한 언어 모델을 넘어 추론(Reasoning)과 행동(Acting)을 결합하는 ReAct 패러다임으로 어떻게 발전했는지 논문을 통해 확인하세요.
The Transformer (Vaswani et al., 2017) replaced sequential recurrence with parallel self-attention across the entire sequence in a single computation. By attending to all positions simultaneously, it eliminated both the sequential bottleneck and the fixed-size compression problem. With this, LLMs became feasible at scales that RNNs could not approach.
6. The Deep Learning Era in Numbers — A Timeline
7. Maritime Connection — CNNs and RNNs in Vessel Systems Today
Both architectures remain active in maritime AI deployments. Understanding their capabilities and limitations is necessary to evaluate vendor claims and assess cyber risk for AI-dependent vessel systems.
During an E26 CRSI assessment, I encounter LSTM-based engine monitoring systems where the vendor documentation describes the model's training data but provides no adversarial robustness evaluation. This is a gap that the NIST AI RMF MEASURE function is designed to close — but maritime operators are often unaware that such evaluation is expected.
The practical question for an IACS UR E26 compliance assessment: "Has your AI-based monitoring system been tested against manipulated sensor inputs that simulate the attack patterns documented in the MITRE ATLAS framework?" In most cases the answer is no.
For CNN-based vision systems: has the model been evaluated for sensitivity to physical adversarial patches — a class of attack where a sticker on a container can cause misclassification? These are demonstrated in research with physical-world applications.
- CNNs exploit spatial structure through local filters, parameter sharing, and pooling. Dominant architecture for image analysis in maritime AI.
- Residual connections (ResNet, 2015) allow 100+ layer training, dramatically improving vision task performance.
- RNNs process sequences via hidden state. Used in maritime time-series forecasting and log anomaly detection.
- The vanishing gradient problem limits vanilla RNNs. LSTM and GRU solve this through gated cell state mechanisms.
- RNNs are inherently sequential — cannot be parallelised on GPUs. Primary motivation for replacing them with the Transformer.
- CNN and LSTM systems in maritime OT are susceptible to adversarial input manipulation — addressed by IACS UR E26 cyber resilience requirements.
Lesson 4 — Computer Vision and Object Detection applies CNN architecture to the full vision pipeline: ImageNet, YOLO, SAM, CLIP, and maritime AI vision applications in ports and shipyards.
- Hochreiter & Schmidhuber (1997). Long Short-Term Memory. Neural Computation, 9(8).
- Cho et al. (2014). GRU paper. arXiv:1406.1078.
- Krizhevsky, Sutskever & Hinton (2012). AlexNet. NeurIPS 2012.
- He et al. (2016). ResNet. CVPR 2016. arXiv:1512.03385.
- MITRE ATLAS — atlas.mitre.org
- NIST AI RMF 1.0 — airc.nist.gov
⚓ Join the ShipPaulJobs Community
Join →
Comments
Post a Comment