Good value

Codeship currently offers 100 builds a month for free on private repos that include BitBucket and Github - this is GOOD. It’s hard to find hosted CI systems that will work with non-Github repository hosting.

Good support

Codeship’s support team know their stuff.

I’ve had multiple discussions with Codeship’s support via email and Twitter and I’ve found that they get back to you with timely email replies and suggestions about why things are breaking. Great for a free service!

Good speed

Builds are QUICK! So quick that I thought they weren’t using a real DB and wrote a test to prove that they were listening to the settings I’d pushed specifically for Codeship.

False negatives

Some of my builds have received false negatives meaning they have failed when they shouldn’t have. This is annoying but manageable and can often be solved by re-running the build.

For example, I’ve had a couple of builds happen on an instance where the database wasn’t available when the tests started running, so everything went RED. Slack then went RED. Client team start worrying.

Codeship’s solution for this was “Add sleep 3 to make the tests wait 3 seconds before running”. This works but why isn’t Codeship’s instance build process checking this before starting the run?

False positives

Builds have received false positives. They have passed when they shouldn’t have - and this is far worse than a false negative.

This week a build on Codeship had git submodule init fail, but the build didn’t go red. The same condition happened last week and the build did fail. While I’m writing this, Codeship are looking into the problem.

Scarce documentation

It’s not very clear how to work with teams, amongst other topics. Team members can’t see how many remaining free builds are available each month so they can’t see if they’re using up the allowance of free builds quickly or not.

Artefacts need special attention

It’s hard to get build artefacts off Codeship’s build instances since the system burns the instance once the build is complete regardless of result. This means if you don’t push artefacts off the server yourself with a script, they’re gone.

Compare with CircleCI - they keep build artefacts in a special folder available for further processing, downloading, or access after the build is complete.

Test commands are in Codeship not repo

A minor gripe I have with the Codeship system is that the test setup and run commands are stored in the configuration for the project on their site. With Travis for example, the server runs the commands it finds in .travis.yml.

The benefit of putting the test commands in the repo is that they can be easily coordinated with git commits - if you want to change how something is run, you can do it all in the code, commit and push.

The Codeship way means copy-and-pasting run commands up onto their website and then pushing new code to be tested with those settings. It makes it hard to prove which branch was run with which settings, or if things were changed to make builds pass.