This error occurs in CI/CD pipelines when Git attempts to update submodules in a shallow clone. Shallow clones (using --depth) lack the full commit history that submodules need to resolve their target commits, causing the update to fail.
When you perform a shallow clone with `git clone --depth 1`, Git only fetches a limited number of commits from the repository's history. This optimization significantly reduces clone time and disk usage, which is why many CI/CD systems use it by default. However, Git submodules reference specific commits in their parent repository. When Git tries to update a submodule, it needs to: 1. Find the commit SHA that the submodule should be checked out to 2. Locate that commit in the submodule's history 3. Check out the submodule at that specific commit With a shallow clone, the parent repository may not have enough history to determine which commit the submodule should point to. Additionally, fetching the submodule itself may fail if the referenced commit is not within the shallow depth. This issue is particularly common in CI environments like GitHub Actions, GitLab CI, Jenkins, and CircleCI, where shallow clones are the default behavior for performance reasons.
Update your GitHub Actions workflow to fetch the full repository history:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Fetch all history for all branches and tags
submodules: recursive # Checkout submodulesIf you only need submodules without full history, you can try a compromise:
- uses: actions/checkout@v4
with:
fetch-depth: 2 # Minimal history, may work for some submodule setups
submodules: recursiveFor the most reliable results, use fetch-depth: 0.
In GitLab CI, set the submodule strategy and disable shallow cloning:
variables:
GIT_SUBMODULE_STRATEGY: recursive
GIT_DEPTH: 0 # Disable shallow clone (0 = full clone)
# Or use GIT_SUBMODULE_DEPTH for submodule-specific depth
variables:
GIT_SUBMODULE_STRATEGY: recursive
GIT_SUBMODULE_DEPTH: 0If you're using GitLab Runner 13.0+, you may also need:
variables:
GIT_SUBMODULE_FORCE_HTTPS: "true" # If using SSH URLs in .gitmodulesIn Jenkins, modify your pipeline or job configuration:
Declarative Pipeline:
pipeline {
agent any
stages {
stage('Checkout') {
steps {
checkout([
$class: 'GitSCM',
branches: [[name: '*/main']],
extensions: [
[$class: 'SubmoduleOption',
disableSubmodules: false,
recursiveSubmodules: true,
trackingSubmodules: false],
[$class: 'CloneOption',
depth: 0, // Full clone
noTags: false,
shallow: false]
],
userRemoteConfigs: [[url: 'https://github.com/your/repo.git']]
])
}
}
}
}Job Configuration:
In the Git plugin settings, uncheck "Shallow clone" and enable "Recursively update submodules".
CircleCI uses shallow clones by default. Override this in your config:
version: 2.1
jobs:
build:
docker:
- image: cimg/base:stable
steps:
- checkout
- run:
name: Initialize submodules with full history
command: |
git fetch --unshallow || true
git submodule sync --recursive
git submodule update --init --recursiveThe git fetch --unshallow command converts a shallow clone into a full clone. The || true prevents failure if the clone is already complete.
If you can't modify the CI configuration, add a script step after checkout:
#!/bin/bash
set -e
# Check if we're in a shallow clone
if [ -f "$(git rev-parse --git-dir)/shallow" ]; then
echo "Unshallowing repository..."
git fetch --unshallow
fi
# Sync submodule URLs in case they changed
git submodule sync --recursive
# Initialize and update all submodules
git submodule update --init --recursive
echo "Submodules updated successfully"Save this as scripts/init-submodules.sh and call it in your CI pipeline.
If submodules are private repositories, ensure proper authentication:
GitHub Actions:
- uses: actions/checkout@v4
with:
fetch-depth: 0
submodules: recursive
ssh-key: ${{ secrets.SUBMODULE_DEPLOY_KEY }}GitLab CI:
before_script:
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan github.com >> ~/.ssh/known_hostsGenerate a deploy key with read access to all submodule repositories and add it to your CI secrets.
Sometimes CI systems can't authenticate SSH URLs. Convert to HTTPS in CI:
# In your CI before_script or setup step
git config --global url."https://github.com/".insteadOf "[email protected]:"
git config --global url."https://gitlab.com/".insteadOf "[email protected]:"
# Then update submodules
git submodule sync --recursive
git submodule update --init --recursiveFor GitLab CI, you can use the built-in variable:
variables:
GIT_SUBMODULE_FORCE_HTTPS: "true"Understanding the shallow clone limitation:
A shallow clone is like reading only the last page of a book. Git submodules, however, might reference commits from earlier in history. When Git tries to check out the submodule at a specific commit, that commit simply doesn't exist in the truncated history.
Performance considerations:
Fetching full history (fetch-depth: 0) increases CI time, especially for large repositories. Consider these optimizations:
# GitHub Actions - fetch only tags and the current branch
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true
submodules: recursive
# Or use a blobless clone (faster for repos with large files)
- run: |
git clone --filter=blob:none --recurse-submodules https://github.com/org/repo.gitSubmodule depth vs repository depth:
In GitLab CI, you can set different depths:
variables:
GIT_DEPTH: 1 # Shallow clone the main repo
GIT_SUBMODULE_DEPTH: 0 # Full clone for submodules only
GIT_SUBMODULE_STRATEGY: recursiveCaching considerations:
If your CI caches the Git directory between runs, stale shallow clone data can cause issues:
# Clear submodule cache before updating
git submodule deinit --all -f
git submodule update --init --recursiveAlternative: vendoring submodule content:
For very slow CI pipelines, consider vendoring (copying) submodule content directly into your repository:
# One-time: copy submodule content instead of using submodules
git submodule update --init
cp -r submodule-dir/ vendor/submodule-dir/
git submodule deinit submodule-dir
git rm submodule-dir
git add vendor/submodule-dir/
git commit -m "Vendor submodule content for CI performance"This eliminates submodule fetch issues but requires manual updates when the upstream changes.
kex_exchange_identification: Connection closed by remote host
Connection closed by remote host when connecting to Git server
fatal: unable to access: Proxy auto-configuration failed
How to fix 'Proxy auto-configuration failed' in Git
fatal: unable to access: Authentication failed (proxy requires basic auth)
How to fix 'Authentication failed (proxy requires basic auth)' in Git
fatal: unable to access: no_proxy configuration not working
How to fix 'no_proxy configuration not working' in Git
fatal: unable to read tree object in treeless clone
How to fix 'unable to read tree object in treeless clone' in Git