Thursday, September 24, 2015

Migrating from GitHub to AWS Codecommit

At work, we recently made the move from GitHub to AWS Codecommit for our private repositories. There’s lots of tradeoffs moving to CodeCommit at this time, like the loss of a web UI, wikis, issue tracking, pull requests, etc. But we made the move as part of our company-wide cost saving measures, as we found CodeCommit to be quite a bit cheaper than GitHub (like free, since we fall into the free tier at this point).

So, how did we migrate all of our GitHub repositories at once? Well, we didn’t do it by hand, that’s for sure.

First, we needed to generate a list of all of our GitHub repositories. We used the GitHub API to get the list, which I pushed into a text file, repos.txt:

curl --silent -u $GITHUB_USER:$GITHUB_PASSWD https://api.github.com/orgs/$GITHUB_ORG/repos?per_page=100 -q | grep "\"name\"" | awk -F': "' '{print $2}' | sed -e 's/",//g'

Note: If you have two-factor authentication enabled in GitHub, you’ll need to get a personal API token to use in place of your password. You can generate this token here.

Next, we used a script to loop through all the repositories, clone them locally, create a repository in CodeCommit, and finally push to CodeCommit. You’ll need the aws cli tools installed locally. I used ~/_trash as my local working folder.

cd ~/_trash
while read r; do
  echo $r
  aws codecommit create-repository --repository-name $r --region us-east-1
  git clone --mirror git@github.com:$GITHUB_ORG/$r.git 
  cd $r.git
  git push ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/$r --all
  git push ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/$r --tags
  sleep 10
  aws codecommit update-default-branch --repository-name $r --default-branch-name master --region us-east-1
  cd ..
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

We ran through this a few times before pulling the trigger for real. To delete all the repositories in CodeCommit quickly during our testing, we used this script:

while read r; do
  echo $r
  aws codecommit delete-repository --repository-name $r --region us-east-1
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

Once we had all of the repositories moved, we just needed to repoint the origin for our local repositories to CodeCommit. After a few weeks of making sure we had everything, we backed up all of our GitHub repositories one last time using this script, and then we deleted all of our repositories in GitHub.

while read r; do
  echo $r
  curl -X DELETE -u $GITHUB_USER:$GITHUB_PASSWD https://api.github.com/repos/$GITHUB_ORG/$r
done < ~/dev/utility-scripts/aws/codecommit/repos.txt

Note your GitHub account or generated token will need to have the delete_repo scope in order to delete a repository.

You can find all the scripts in this Gist.

Tuesday, September 8, 2015

Google OnHub - initial impressions

My Google OnHub was delivered this past weekend. Here's my initial thoughts:

  • Packaging was very nicely done. Hardware these days comes in some really beautiful boxes that are well designed. I'd rate packaging 10/10.
  • There are no instructions in the box. None. It was pretty intuitive to get it out of the box and figure out what to do, but I did hear from a friend that he wasn't sure how to get the outer shell off. He didn't want to break it by pulling or twisting the wrong way.
  • Initial setup using an iPhone 6 with iOS 9 beta failed. After connecting to the setup Wifi network, the Google On app would never find the OnHub. I worked on this for about 45 minutes before giving up and trying the setup on an iOS 8 device. Setup worked perfectly.
  • Once I figured the whole iOS 9 thing out, setup was so simple. Great job on making it painless!
  • Wifi signal strength in our house has had a major upgrade. Corners of the house that were always very weak signal now have full strength connections.
  • Throughput for both wireless and wired clients test out with speeds matching our service from Comcast.
I also like that I do not have to be connected to the local network in order to manage the OnHub. So, when someone at home can't connect, I can open the OnHub app anywhere I am and manage the network (so long as my home Internet connection is up). That's pretty cool!