Global Workflow

Global-workflow is the end-to-end workflow designed to run global configurations of medium range weather forecasting for the UFS weather model. It supports both development and operational implementations. In its current format it supports the Global Forecast System (GFS), Global Ensemble Forecast System (GEFS), and Subseasonal Forecast System (SFS) configurations.

Status

• State of develop (HEAD) branch: GFSv17+ development

• State of operations (dev/gfs.v16 branch): GFS v16.3.24 tag: [gfs.v16.3.24]

Code managers

• David Huber - @DavidHuber-NOAA / david.huber@noaa.gov

• Rahul Mahajan - @aerorahul / rahul.mahajan@noaa.gov

Announcements

General updates: NOAA employees and affiliates can join the gfs-announce distribution list to get updates on the GFS and global-workflow. Contact a code manager to get added to the list or removed from it.

GitHub updates: Users should adjust their “Watch” settings for this repo so they receive notifications as they’d like to. Find the “Watch” or “Unwatch” button towards the top right of the authoritative global-workflow repository page and click it to adjust how you watch the repo.

Table of Contents

• 1. Contributing to the Global Workflow
  • 1.1. Code managers
  • 1.2. Where to do development?
  • 1.3. Protected branches
  • 1.4. How to get changes into develop (HEAD) branch
  • 1.5. Development Tools
    • 1.5.1. Continuous Integration (CI)
    • 1.5.2. Comparison Tools
  • 1.6. Code standards
    • 1.6.1. Using PyCodeStyle
  • 1.7. Pull request standards
  • 1.8. How to sync fork with the authoritative repository
• 2. Testing Global Workflow Jobs
  • 2.1. Overview
    • 2.1.1. Testing Philosophy
    • 2.1.2. Framework Components
  • 2.2. Framework Architecture
    • 2.2.1. Directory Structure
    • 2.2.2. Configuration Management
    • 2.2.3. CMake Integration
  • 2.3. Test Case Configuration
    • 2.3.1. Naming Conventions
    • 2.3.2. YAML File Structure
    • 2.3.3. Multi-Cycle Tests
  • 2.4. Common Test Patterns
    • 2.4.1. Atmosphere-Only Forecast
    • 2.4.2. Coupled Forecast
    • 2.4.3. Ensemble Member Test
    • 2.4.4. Product Generation Test
  • 2.5. Running Tests with CMake
    • 2.5.1. Environment Setup
    • 2.5.2. Configuration
    • 2.5.3. Running Tests
    • 2.5.4. Validation Modes
  • 2.6. Adding New Tests
    • 2.6.1. Step-by-Step Procedure
    • 2.6.2. Best Practices
  • 2.7. CI/CD Integration
    • 2.7.1. GitLab CI Pipeline
    • 2.7.2. Platform Support
  • 2.8. Troubleshooting
    • 2.8.1. Common Issues
    • 2.8.2. Debugging Strategies
  • 2.9. Additional Resources
    • 2.9.1. Related Documentation
    • 2.9.2. Directory Reference
    • 2.9.3. Development History
• 3. Global Workflow Components
  • 3.1. Component repositories
  • 3.2. External dependencies
    • 3.2.1. Libraries
    • 3.2.2. Observation data (OBSPROC/prep)
• 4. GFS Configuration
  • 4.1. Jobs in the GFS Configuration
• 5. HPC Settings and Help
  • 5.1. Minimum system software requirements
  • 5.2. Feature availability by HPC
  • 5.3. HPC helpdesks
  • 5.4. Restricted data access
  • 5.5. Stacksize on R&Ds (Hera, Orion, Hercules)
  • 5.6. Forecast hangs due to issue with ssh-keys
• 6. Plotting Output
  • 6.1. Analysis output
  • 6.2. Forecast output
• 7. Run Global Workflow
  • 7.1. Clone and build Global Workflow
    • 7.1.1. Quick Instructions
    • 7.1.2. Clone workflow and component repositories
    • 7.1.3. Build components
    • 7.1.4. Link components
  • 7.2. Initial Conditions
    • 7.2.1. Staged Initial Conditions
    • 7.2.2. Prepare Initial Conditions
  • 7.3. Experiment Setup
    • 7.3.1. Step 1: Set user settings
    • 7.3.2. Step 2: Run experiment generator script
    • 7.3.3. Step 3: Check user and experiment settings
    • 7.3.4. Step 4: Run workflow generator script
    • 7.3.5. Step 5: Confirm files from setup scripts
  • 7.4. Configure Run
  • 7.5. Start your run
    • 7.5.1. Start your run from within your EXPDIR
    • 7.5.2. Set up your experiment cron or scron
  • 7.6. Monitor ROCOTO Run
    • 7.6.1. Using command line
    • 7.6.2. Use ROCOTO viewer
  • 7.7. View Experiment output
    • 7.7.1. Forecast-only
    • 7.7.2. Cycled
• 8. Wave Component
  • 8.1. Adding a new wave Grid
    • 8.1.1. Naming the Wave Grid
    • 8.1.2. Adding Fix Files
    • 8.1.3. Updating Config and Script Files
    • 8.1.4. Updates for New Product Grids
    • 8.1.5. New Staged ICs
• 9. Configuring NOAA Cloud Service Providers
  • 9.1. Login to the NOAA CSP
  • 9.2. Configure the NOAA CSP Instance
  • 9.3. Add CSP Lustre Filesystem
  • 9.4. Attach CSP Lustre Filesystem
  • 9.5. Using the NOAA CSP Cluster
  • 9.6. Running the Global Workflow
• 10. Common Errors Known Issues
  • 10.1. Error: Reserved Variables Causing Workflow Issues
  • 10.2. Error: “ImportError” message when running setup script
  • 10.3. Error: curses default colors when running viewer
  • 10.4. Issue: Directory name change for EnKF folder in ROTDIR
• 11. Setup Globus Connections for HPSS
  • 11.1. Setting Up Globus
  • 11.2. Common Globus Issues
• 12. Global Chemistry and Aerosol Forecast
  • 12.1. Overview
  • 12.2. Key Features
  • 12.3. Running GCAFS
  • 12.4. GCAFS Workflow
  • 12.5. Emissions Preprocessing
  • 12.6. GOCART Configuration Files
    • 12.6.1. Core Configuration
    • 12.6.2. Aerosol Species Configuration
    • 12.6.3. Output and Diagnostics
    • 12.6.4. Emissions Configuration
    • 12.6.5. ExtData File Format Details
    • 12.6.6. AERO_HISTORY.rc File Details
  • 12.7. Output Products