Migrating from GitHub to AWS Codecommit

At work, we recently made the move from GitHub to AWS Codecommit for our private repositories. There’s lots of tradeoffs moving to CodeCommit at this time, like the loss of a web UI, wikis, issue tracking, pull requests, etc. But we made the move as part of our company-wide cost saving measures, as we found CodeCommit to be quite a bit cheaper than GitHub (like free, since we fall into the free tier at this point).

So, how did we migrate all of our GitHub repositories at once? Well, we didn’t do it by hand, that’s for sure.

First, we needed to generate a list of all of our GitHub repositories. We used the GitHub API to get the list, which I pushed into a text file, repos.txt:

curl --silent -u $GITHUB_USER:$GITHUB_PASSWD https://api.github.com/orgs/$GITHUB_ORG/repos?per_page=100 -q | grep "\"name\"" | awk -F': "' '{print $2}' | sed -e 's/",//g'

Note: If you have two-factor authentication enabled in GitHub, you’ll need to get a personal API token to use in place of your password. You can generate this token here.

Next, we used a script to loop through all the repositories, clone them locally, create a repository in CodeCommit, and finally push to CodeCommit. You’ll need the aws cli tools installed locally. I used ~/_trash as my local working folder.

cd ~/_trash
while read r; do
  echo $r
  aws codecommit create-repository --repository-name $r --region us-east-1
  git clone --mirror git@github.com:$GITHUB_ORG/$r.git 
  cd $r.git
  git push ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/$r --all
  git push ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/$r --tags
  sleep 10
  aws codecommit update-default-branch --repository-name $r --default-branch-name master --region us-east-1
  cd ..
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

We ran through this a few times before pulling the trigger for real. To delete all the repositories in CodeCommit quickly during our testing, we used this script:

while read r; do
  echo $r
  aws codecommit delete-repository --repository-name $r --region us-east-1
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

Once we had all of the repositories moved, we just needed to repoint the origin for our local repositories to CodeCommit. After a few weeks of making sure we had everything, we backed up all of our GitHub repositories one last time using this script, and then we deleted all of our repositories in GitHub.

while read r; do
  echo $r
  curl -X DELETE -u $GITHUB_USER:$GITHUB_PASSWD https://api.github.com/repos/$GITHUB_ORG/$r
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

Note your GitHub account or generated token will need to have the delete_repo scope in order to delete a repository.

You can find all the scripts in this Gist.