From Mathematical Foundations to AI Chatbot Development for Maritime Document Automatio

NLP Deep Dive: Transformer, BERT & GPT — From Mathematical Foundations to AI Chatbot Development for Maritime Document Automation

Attention Is All You Need · BERT / GPT / ELMo Comparison · SQuAD / KorQuAD · Self-Attention · Maritime AI Applications

Captain Paul

Maritime 4.0 · AI, Data & Cyber Security
LinkedIn: linkedin.com/in/shipjobs

 2020.09~ · R&D Series

Research Context

The 2017 paper "Attention Is All You Need" and the subsequent rise of the Transformer architecture triggered an explosion in AI chatbots and dialogue systems, with BERT and GPT-2 cementing pre-trained language models as the new standard across industries.

Through hands-on AI chatbot development and a structured NLP course, this R&D note explores what Natural Language Processing is, why it has grown so rapidly, and how it applies to the maritime industry — from document automation to crew communication systems.

 Research Objectives

① Understand core NLP concepts and the drivers behind its rapid growth
② Analyze the key mechanisms of Transformer, BERT, and GPT
③ Build a BERT-based AI chatbot through hands-on fine-tuning
④ Explore maritime applications: document QA, compliance support, incident analysis

---

I. AI vs Natural Language Processing — What Changed?

Natural Language Processing (NLP) is the branch of AI that enables computers to understand, interpret, and generate human language. While classical AI excelled at structured data, NLP targets unstructured text — contracts, reports, conversations, emails, and regulatory documents.

易 Traditional AI

Rule-based processing
Structured data only
Explicit programming required
Narrow domain coverage

 NLP (Transformer Era)

Context-aware understanding
Unstructured text processing
Large-scale pre-training
Cross-domain generalization

SQuAD vs KorQuAD — Reading Comprehension Benchmarks

SQuAD (Stanford)

English machine reading comprehension benchmark with 100k+ question-answer pairs from Wikipedia. BERT exceeded human-level F1 score (93%).

KorQuAD (Korean)

Korean MRC dataset developed by LG CNS. The de facto benchmark for Korean NLP research; KorBERT-based models achieve top performance.

 3 Drivers Behind NLP's Explosive Growth

2017

Attention Is All You Need — Transformer overcomes sequential bottleneck of RNNs; full parallelism unlocked

2018

BERT / GPT / ELMo — Pre-train once, fine-tune anywhere; generalist language models for all downstream tasks

★

Compute + Big Data — Falling GPU/TPU costs combined with massive internet text corpora made large-scale training practical

---

II. Mathematical Background & NLP Pipeline

NLP Processing Steps

1

Morphological Analysis (Tokenization)

Split text into meaningful minimum units (morphemes/tokens). Korean requires dedicated tools (KoNLPy, MeCab) due to its agglutinative grammar; English uses subword tokenization (BPE, WordPiece).

2

Word Embedding

Map words into dense high-dimensional vectors. Evolved from static embeddings (Word2Vec, GloVe) to BERT's dynamic contextual embeddings that change meaning based on surrounding words.

3

Language Modeling

Model the probability distribution over word sequences. The trajectory: Markov Model → RNN → Attention → Transformer — each step capturing longer-range dependencies.

4

Fine-Tuning & Task Application

Adapt a pre-trained model (BERT) to a specific task (QA, classification, NER) with minimal labeled data. The "pre-train once, fine-tune anywhere" paradigm slashes development cost.

Language Model Evolution — Markov → RNN → Transformer

Markov Model

Predicts next word from N prior words. Cannot capture long-range dependencies.

RNN / LSTM

Sequential processing with hidden state. Partially solves vanishing gradient. No parallelism.

Transformer ★

Self-Attention attends all positions simultaneously. Full parallelism. Foundation of BERT & GPT.

---

III. Key Research: BERT vs GPT vs ELMo

Three landmark models published in 2018–2019 established distinct pre-training strategies that dramatically advanced NLP performance across all benchmarks.

PAPER-01

BERT — Google AI (2018)

"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

Training Strategy

Bidirectional — learns left and right context simultaneously via Masked Language Model (MLM) + Next Sentence Prediction (NSP)

Strength

Best-in-class for comprehension tasks: QA, NER, sentiment analysis. Surpassed human-level on SQuAD.

python — BERT Fine-Tuning for QA

from transformers import BertTokenizer, BertForQuestionAnswering
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
model     = BertForQuestionAnswering.from_pretrained('bert-base-multilingual-cased')

# Encode question + context document
inputs  = tokenizer(question, context, return_tensors="pt")
outputs = model(**inputs)

start = torch.argmax(outputs.start_logits)
end   = torch.argmax(outputs.end_logits) + 1
answer = tokenizer.decode(inputs["input_ids"][0][start:end])

PAPER-02

OpenAI GPT — OpenAI (2018)

"Improving Language Understanding by Generative Pre-Training"

Training Strategy

Autoregressive (Unidirectional) — predict the next token from all prior tokens. Optimized for text generation.

Strength

Fluent text generation — email drafting, report writing, dialogue response generation. Foundation for ChatGPT.

PAPER-03

ELMo — AllenAI (2018)

"Deep Contextualized Word Representations"

Key Contribution

Bidirectional LSTM generates dynamic word embeddings — the same word receives different vectors depending on context. Solved polysemy for the first time at scale, paving the way for BERT.

---

IV. Transformer Architecture — Core Components

Self-Attn

Self-Attention

Computes pairwise relationships between all words in a sentence simultaneously. The model learns which words to "attend to" when encoding each token — implemented via Query, Key, and Value matrix operations.

Multi-Head

Multi-Head Self-Attention

Runs multiple attention heads in parallel, each learning different relationship aspects (syntactic, semantic, co-reference). Results are concatenated, giving the model a richer, multi-perspective representation.

Pos.Enc

Positional Encoding

Since Transformers have no built-in order awareness, position information is injected using sin/cos functions added to token embeddings. This lets the model distinguish "ship arrives port" from "port arrives ship."

FFN+Softmax

Feed Forward + Softmax ★

Non-linear transformation (ReLU) applied after attention to increase expressiveness. The final Softmax layer converts logits into a probability distribution over the vocabulary for next-token prediction.

python — Scaled Dot-Product Attention

import torch
import torch.nn.functional as F

# Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) · V
def scaled_dot_product_attention(Q, K, V):
    d_k    = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / d_k**0.5
    attn   = F.softmax(scores, dim=-1)
    return torch.matmul(attn, V)   # weighted sum of Values

---

V. Hands-On Curriculum & Maritime Applications

 Study Curriculum (From September 2020)

① Theory Foundations

Math background · NLP basics · Morphological analysis · Word embedding · Language models

② Key Papers

Transformer · BERT · GPT-1 · ELMo — original paper deep-reads

③ Environment Setup

Korean morphological analyzer · Preprocessing · Sequence labeling · STT / TTS integration

④ AI Chatbot Build ★

BERT-based QA chatbot fine-tuning · KorQuAD dataset · Deployment pipeline

References: TensorFlow 2 & Machine Learning for NLP (book) · T Academy & Saltlux BERT Course · Machine Learning for Everyone — Prof. Sung Kim, HKUST



SMS Document QA — Safety Management System Automation

A BERT-based QA system can retrieve specific procedures and regulations from hundreds of SMS pages instantly. Dramatically reduces audit preparation time under ISM Code requirements.



Compliance Chatbot — IACS / IMO Regulation Q&A

A chatbot fine-tuned on IACS UR E26/E27 and IMO MSC-FAL.1/Circ.3 documents answers compliance team queries in real time, reducing reliance on external legal consultants.



Bridge Communication Automation — VHF / GMDSS Log Analysis

STT converts VHF radio traffic to text; NER then extracts vessel names, positions, and hazard flags automatically. Enables auto-generation of voyage logs and distress situation reports.



Incident Report Analysis — Causal Pattern Mining ★

NLP classifies and summarizes maritime accident reports (MAIB, ATSB, KMST) to surface recurring causal patterns automatically, strengthening proactive safety management and risk assessment.

 Field Note

NLP is no longer a lab technology. The document-intensive nature of maritime operations makes it one of the most suitable industries for NLP adoption.

Thousands of pages of SMS manuals, regulatory circulars, incident reports, and radio logs — if a BERT-based system processes this unstructured data in real time, a single surveyor's workload becomes manageable AI-assisted intelligence. For Korean maritime documents, fine-tuning KorBERT (ETRI) or KLUE-BERT is the practical starting point.

Conclusion & Next Steps

From Transformer fundamentals to BERT / GPT / ELMo comparison and hands-on chatbot fine-tuning, this R&D note maps the complete theoretical and practical path of modern NLP.

The R&D journey started in September 2020 goes beyond AI chatbot development — it points toward maritime document automation, real-time compliance support, and incident analysis as concrete industry targets.

Next step: Build a KorBERT-based Korean maritime regulation QA system — automating the search and interpretation of IACS UR E26/E27 clauses for compliance teams.

#NLP #Transformer #BERT #GPT #AttentionIsAllYouNeed #DeepLearning #AIChatbot #KorBERT #MaritimeAI #DocumentAutomation #ISMCode #Maritime40

References & Further Reading

1. Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
   https://arxiv.org/abs/1706.03762
2. Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.
   https://arxiv.org/abs/1810.04805
3. Radford, A. et al. (2018). Improving Language Understanding by Generative Pre-Training (GPT-1). OpenAI Technical Report.
   https://openai.com/index/language-unsupervised/
4. Peters, M. et al. (2018). Deep Contextualized Word Representations (ELMo). NAACL 2018.
   https://arxiv.org/abs/1802.05365
5. Rajpurkar, P. et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016.
   https://arxiv.org/abs/1606.05250
6. Lim, H. et al. (2019). KorQuAD 1.0: Korean QA Dataset for Machine Reading Comprehension. arXiv preprint.
   https://arxiv.org/abs/1909.07005

Captain Paul

Maritime 4.0 · AI, Data & Cyber Security
NLP · Edge AI · Smart Ship · IACS UR E26/E27 · IMO Compliance

More R&D Articles ↗

⚓ Join the ShipPaulJobs Community

Join →