Sharpening the Axe — Preparing the AI Modeling Work Environment
Sharpening the Axe — Preparing the AI Modeling Work Environment
Engineering Preparation · SFTP, SSH, GitHub & Collaboration Tools
I have resumed engineering and modeling tasks, which I had temporarily set aside to focus on project management and new business planning.
For a project involving optimizing (tuning) individual models for image and text processing, as well as integrating multiple models — a skill honed during my Ph.D. program — I now require a complex and well-structured work environment.
To ensure optimal efficiency under the given constraints, I plan to set up, refine, and document the work environment for future reference.
"If I had eight hours to chop down a tree, I would spend six hours sharpening the axe."
— Abraham Lincoln, 16th President of the United States
Review of Work Environment
1. Managing Work on an Unstable Server & Data Backup
- ✅Backup & Record Management: Using
SFTPto store and synchronize data in a structured manner. - ✅GitHub Usage: Limited to baseline model development and version control for key components.
-
✅Code & Data Management:
- Using SFTP Sync to manage code and data across the team until the environment stabilizes.
- Ensuring team members' PCs are synced for redundancy and preventing data loss.
2. Remote Access Setup
- ✅Primary:
ssh(Company Server, AWS) - ✅Secondary:
ngrok(For remote access to Google Colab)
🚧 Corporate Security Policy Issues
- Internal servers can connect to other internal servers, but external servers cannot access internal servers via SSH.
- This stems from security policies restricting inbound SSH connections.
- AWS is technically external, so SSH should work unless explicitly blocked for security, cost, or management reasons.
3. Personal Work Environment
- ✅Hardware: Apple Mac M1 Pro (2020)
-
✅Development Tools:
- VS Code Extensions: SFTP, SSH, Python, Jupyter Notebook, Beautify, JSON
- Jupyter Notebook: Main local development environment
- Google Colab: For cloud-based experiments
4. Storage & Data Accessibility
-
✅Company Internal Server
- Access to image and text datasets for analysis
- GPU-enabled servers available for model training
-
✅Apple Mac M1 Pro as an SFTP Server
- Acts as a sync server to store backups and mitigate GPU server instability
- Each unit member's PC archives and synchronizes important work
-
✅Google Drive (Temporary Storage)
- Large dataset & image storage
- Collaboration with external teams for data exchange
- Code storage & sharing, ensuring project continuity
5. Version Control & History Management
- ✅Git History installed on both the server and team members' PCs to maintain commit logs and traceability.
6. Collaboration & Communication
-
✅Messaging Platform: Slack
- Used for daily task tracking and logging issues within the unit.
- Chosen over internal company messengers to facilitate future external collaboration.
- Slack is integrated with GitHub, making it valuable for monitoring baseline models and code updates.
- Team members must use company emails for Slack accounts.
Summary & Next Steps
- 🔹Set up and validate SSH access to AWS (ensure company policy allows it).
- 🔹Refine SFTP sync strategy to improve data redundancy and workflow efficiency.
- 🔹Test Google Drive integration for seamless external collaboration.
- 🔹Monitor and adjust Slack usage as the team grows and external partners join.
This setup balances security, flexibility, and efficiency, ensuring smooth project execution in a corporate environment with strict security policies and infrastructure limitations.
Core Insights:
Maritime professional focused on AI-assisted vessel intelligence, classification society regulations, and OT/IT cybersecurity. Building tools that help the industry navigate Maritime 4.0.
🌐 More Articles ↗
Comments
Post a Comment