A blog by Zeshan Amjad, featuring thoughts on topics like career development, software engineering, design and finance.

How to speed up your CI by 35% and still be unhappy


Recently, I'd noticed our Lint CI job was getting painfully slow for our main backend monorepo.

😢 **The performance gap was staggering:**

- **Locally:** ~1.2 seconds
- **CI:** 6 minutes, with some runs taking up to 9 minutes

There was a clear culprit eating up most of this 6 minute run time:

```bash
run golangci-lint
[...]
Go packages loading (compiled_files|...) took 4m27.254679062s
[...]
```

The `Go packages loading...` step inside `golangci-lint` was taking ~4.5 minutes all by itself, almost all of it spent waiting for `packages.Load` to compile our repo's ~736 packages.

## 🥊 The first problem: untar clashes

I started investigating by looking at our CI logs and immediately spotted error messages during cache restoration for the `setup-go` Github action:

```bash
/usr/bin/tar: ../../../go/pkg/mod/golang.org/x/telemetry/config@v0.31.0/go.mod: Cannot open: File exists
/usr/bin/tar: Exiting with failure status due to previous errors
Warning: Failed to restore: "/usr/bin/tar" failed with error: The process '/usr/bin/tar' failed with exit code 2
Cache is not found
```

`Cache is not found` means we're not getting hits on some very important caches that `setup-go` manages for us (more on that later!).

### ✅ How did we fix the tar issue?

This was caused by a known bug in Go 1.23.0 which causes a `tar` error when restoring from cache as per this [Github issue](https://github.com/actions/setup-go/issues/506).

**Solution:** Upgrade to Go 1.23.1 or above!

### 🏃 Did that speed things up?

The tar error goes away 🎉 but **this doesn't seem to have any impact on runtimes** – and it seems like the `setup-go` cache **still isn't being hit**. We still saw "Cache is not found" messages and were stuck on the same achingly long 6 minute waits.

---

## 🔐 Why are we still having cache misses after fixing the tar bug?

### ❌ Could the cache key have changed?

`setup-go` has a cache key that's defined [here](https://github.com/actions/setup-go/blob/d35c59abb061a4a6fb18e82ac0862c26744d6ab5/src/cache-restore.ts#L35) as:

```json
 setup-go-${platform}-${arch}-${linuxVersion}go-${versionSpec}-${fileHash}

 where fileHash refers to the hash of the go.sum file
```

I ran a test with a new PR where I didn't change any Go files, and just added a word to the README. We'd expect this to generate the exact same hash key since no Go files changed. And that's exactly what happened! The cache keys for `go-setup` are the **same** across my older PRs and this new PR.

Even with identical cache keys between runs, we kept seeing cache misses, **so it can't be the keys**.

### ⬆️ Was the cache actually stored successfully?

So we know that the keys are the same, but we still fail to read the cache back. Maybe it wasn't stored successfully?

From looking at our Github Actions cache, we can see 3 cache results stored for that key, so a cache result definitely existed before we did the second run.

![/static/ci-unhappy/image.png](/static/ci-unhappy/image.png)

One thing of interest though is that the cache entries are for different branches and pull request numbers...

### 🔐 Why can't our PR branch see those stored caches?

Turns out, there are [some restrictions](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache) on when a branch can access another branch's cache:

> When a cache is created by a workflow run triggered on a pull request, the cache is created for the merge ref and can only be restored by re-runs of the pull request. **It cannot be restored by other pull requests targeting that base branch.**

This meant every new PR was isolated from existing caches. To confirm this theory, we pushed another commit to an _existing_ branch that had already created a cache.

🎉 Success! We now see a cache hit in `setup-go` for the exact same key that had just failed above.

```json
Cache hit for: setup-go-Linux-x64-ubuntu24-go-1.23.4-074a11b95fa8664b58aa16a5f0327e0aa92b4fe68afdae9be223b7b7eadd5f66
```

### 🤝 Is there any way for PRs to share caches?

A cache that only works for your individual pull request isn't _that_ useful. Your first CI run for a PR will always have a cache miss, and run super slowly.

Making this worse, we're big fans of stacking PRs using [Graphite](https://graphite.dev/). Each component of a stack is a separate PR that can't use any of the caches from the PRs below it.

One thing I did notice was that the `Go build cache` (separate to the `setup-go` cache) was successfully restored across multiple different PRs.

**❓ Why was one cache shared across branches?**

There was a `Go build cache` entry created with the same key against the `main` branch.

The [Github docs](<https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#:~:text=Workflow%20runs%20can%20restore%20caches%20created%20in%20either%20the%20current%20branch%20or%20the%20default%20branch%20(usually%20main).>) clarify why this matters:

> Workflow runs can restore caches created in either the current branch or the default branch – usually `main`

**🎉 So PR branches can share caches that are created against the `main` branch** – but why was it _only_ the `Go build cache` that had a cache hit on the main branch?

### 📦 Why weren't there caches on `main` for any of the other caches?

`Go build cache` only had a cache hit because there happened to be a `main` branch entry in the cache. But that raises the question, where are all the `main` branch entries for our other caches like `setup-go`?

![/static/ci-unhappy/image%201.png](/static/ci-unhappy/image%201.png)

They're being evicted! We have a 10GB non-changeable limit on the Github actions cache, when this is full, Github starts **deleting** cache entries.

### 🔍 Let's look into how our 10GB limit is being used

A deep dive into our GitHub Actions cache revealed some shocking waste. What's eating up all our space?

**81 total cache entries** consuming 30.55GB (well beyond the 10GB limit)

🔴 **69 entries (85%) were PR-specific** and couldn't be shared between branches

Breaking this down further, we have **two main caches** in our Lint workflow (I've omitted some smaller, irrelevant ones):

**`setup-go` cache**

- This stores `GOMODCACHE` (for our dependencies) and `GOCACHE` for our compiled Go code
- Using 13GB of storage

**`go build` cache**

- Also... stores `GOCACHE`...
- What's the difference between this and the `setup-go` cache? Good question! I'll get to that below.
- Using 17.55GB of storage

### 🤏 How can we use our 10GB cache more effectively?

✅ **Disable and delete all PR caches**

PR caches have an 86.1% cache miss rate, I calculated this using data from `gh cache list -L 100`

The cache miss rate is probably even higher than this, of the 10 cache hits – 9 were on my test PRs where I was _trying_ to hit the PR cache.

I disabled writing to the cache if we were on a PR branch for actions that had that as a config setting.

For actions that didn't support that, I added a `delete-pr-cache` workflow step to delete PR caches after every run.

❓**Figure out the difference between the `setup-go` cache, and the `go build cache`**

The `go build cache` was something we manually added to our Lint workflow, compared to the other caches which are managed by the underlying Github actions themselves.

Our `go build cache` explicitly saves the `GOCACHE` folder, with a cache key of:

`${{ runner.os }}-gobuild-${{ **hashFiles('**/*.go')** }}`

Notice something?

**`hashFiles('**/\*.go')`** means that **if we change a single Go file\*\* anywhere in the repo, the cache key will change and we won't get a cache hit.

That means this manual go cache is practically useless – we'd only ever be able to use it if literally no code in the repo had changed.

Also, `setup-go` already handles caching and restoring `GOCACHE` – so this was just duplicated storage.

✅ We removed our manually managed `go build cache` entirely.

### 🥳 What does our actions cache look like after these changes?

![🎉 Our Github actions cache now only stores **usable** entries from main, and we're down from 30GB to **under 1GB.](/static/ci-unhappy/image%202.png)

🎉 Our Github actions cache now only stores **usable** entries from main, and we're down from 23GB to **under 1GB.**

---

# 👎 Did resolving these issues speed up CI?

So, we now have a single `setup-go` cache with 🎯 **consistent cache hits** that handles storing GOMODCACHE and GOCACHE.

But our CI **still isn't noticeably faster…** 😭

We know that it's the `Go packages loading...` step inside `golangci-lint` that's the biggest bottleneck. Let's find out which cache matters most for that step.

## 🎯 Which cache actually affects package loading speed?

Inside the `Go packages loading...` step, it's the call to the `packages.Load` function from the Go standard library that takes the longest. I tested this by building a copy of `golangci-lint` locally and using a debugger to see which function call was slowest.

To find the real bottleneck, I ran experiments locally by selectively clearing different caches:

👎 **Testing GOMODCACHE (dependencies):**

- No significant impact on package loading speed

💘 **Testing GOCACHE (compiled artifacts):**

- Clearing GOCACHE: 1.2s → 49s package loading (40x slower!)
- Total runtime: 1.5s → 65s
- Cache rebuilds to 2.8GB after clearing

> ✨ It's the **GOCACHE** containing the compiled artifacts for our monorepo's source code that has the biggest impact on package loading time.

The `packages.Load` function clearly depends heavily on GOCACHE containing compiled artifacts for our source code.

## 😕 But `setup-go` is already getting cache hits for GOCACHE...

That's true! We have a fully populated and restored GOCACHE in CI now, but we're **still** having slow runs...

### 🍞 Could the `setup-go` cache be stale?

The `setup-go` cache key only changes when you update your go version or change `go.sum` – so it could be that the cache is out of date.

I checked a recent run where the `setup-go` cache was hit (so this should've populated GOCACHE and GOMODCACHE).

The cache file was created was created 7 hours before the run, so most files won't have changed. But the run still takes the same amount of time (~5 minutes).

> ⏳ We do need to think of a way to keep the `setup-go` cache more fresh – but even with a fresh cache, things are still just as slow.

### 🤷 Could it be encore wrapper generation?

We have 108 generated `encore.gen.go` files that get regenerated before we run `golangci-lint` in CI.

The `encore gen wrappers` step might invalidate cache entries, but 108 isn't that many, and they're pretty small files (a few KB) – so this doesn't seem likely.

### 🔴 Could the CI runner be resource constrained?

Maybe the runner is just I/O or memory constrained... the cache is ~14k items and 2.8GB uncompressed.

We can see resource usage coming out of `golangci-lint` on CI:

`Memory: 2565 samples, avg is 49.9MB, **max is 1071.3MB**`

So perhaps the runner is failing to load the full set of compiled packages into memory and is then paging to/from disk.

[Standard Github CI runners](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for--private-repositories) have 2 cores and only 7GB of RAM, so I can see that this might be overly restrictive.

It's non-trivial to get more detailed CPU/memory usage stats out of a Github runner, so let's just test this out experimentally.

### 💻 Let's try a more powerful machine

We bumped our Lint jobs to use a 4 core, 16GB RAM machine instead of the default.

![/static/ci-unhappy/image 3.png](/static/ci-unhappy/image%203.png)

💥 It works!

We've seen a **34% decrease** in run time from ~5.5 mins down to ~3.5 mins.

---

## 🍎 Why is it still not as fast as running locally?

You might notice that 3.5 mins is still a far cry from the speeds we achieve locally. I have a few theories on why this is:

- CI machines have **much slower disk I/O** performance so loading the 2.5GB GOCACHE into memory from disk still takes a while
- CI machines have much worse **single core performance** than our laptops: compiling, traversing, and linting code is _often_ bound by single core performance
- There may be some other issue with our GOCACHE (e.g. we have a path mismatch or the files aren't actually being read by the Go compiler), I think this is _unlikely_ but never say never

I called it a day at this point.

I'm still unhappy, but we can live with ~3 min runtimes for now.

I'm not sure if I can live with an unsolved mystery though! Maybe just one more Saturday debugging... 🏃‍♂️


Writing a CV They Can't Ignore


Securing a job this year is going to be a struggle for most people. Whether you're a new graduate trying to get your foot in the door of an industry, or you've been made redundant and are looking to get back on your feet. Whatever your circumstances, your chances of getting a great new job usually start in the same place as everyone else: your CV. Here's how to write a CV that recruiters will _want_ to read.

---

Before we get started, **who am I, and why should you listen to me?** I'm currently an Associate at a global asset management firm, and I review the CVs of vulnerable people for a British charity ([The Prince's Trust](https://www.princes-trust.org.uk/)). Recruitment, and resumes in particular, are something I see all the time, and I'd love to share a few tips to help you improve yours. Still interested? Let's get started!

# 1 page, max. No, really.

I completely see the appeal of including everything you've ever done of note in your CV. These achievements were likely hard-won, and you have a right to be proud of them, no matter how small they are. But a lot of the CVs I've read come out at over the three-page mark. The chances of all three pages being critically relevant to the role that you've applied for are near-zero.

Think about when you're trying to choose between a handful of different products to buy - are you looking to read three pages of prose about each one? Or would you prefer a **single, focused page**, with the most relevant features for you highlighted in bold?

> If your CV is the length of a short novel, you are likely not even making it to the evaluation stage.

Recruiters are inundated with CVs to read, and the best way to make sure yours isn't passed over immediately is making it as easy as possible to extract information from it. If your CV is the length of a short novel, you are likely not even making it to the evaluation stage. It's too difficult to pull out the relevant information to make a decision, from such a large sea of text. You're getting passed over. Brevity, and focus will give you a fighting chance of making it through. So stick to one page.

# Customise your CV for each type of role

One of the most common reasons I've seen for lengthy CVs is that the candidate is applying for a variety of jobs, often in entirely different industries, with vastly different skills required. Instead of trying to cram _all_ of your experience into a single one-size-fits-all CV, try to create separate CVs for each _type_ of role you're applying for.

![/static/cv/image_(1).png](</static/cv/image_(1).png>)

For example, when applying for a customer service role, you could use the version that emphasises your retail experience and social skills. If you're applying for a job as an engineer, you can use the CV that emphasises your degree and engineering projects, and so on. This focus will give you a much higher chance of success in each role.

All of this customisation sounds like a lot of work, but here's a really quick, simple way to get it done, without writing any new content:

1. Create a "parent" document, with all of the possible things you could include on your CV.
2. For each type of job (e.g. sales, recruitment, engineer), create a new blank document and copy over the sections, jobs, or achievements that are most relevant for that job type. Make sure you keep to a single page!

# Bullet points, not prose.

![/static/cv/image_(4).png](</static/cv/image_(4).png>)

When it comes to making it easy for employers to pick out information from your CV, bullet points have a massive advantage over prose. Here's a bullet pointed list to explain why:

- They naturally force you to divide your thoughts into easily-digestible, bite-sized chunks.
- They release you from having to make your sentences be entirely grammatically correct, freeing up valuable space on the page.
- Your ideas become more cohesive, as a single bullet point stands alone, and has to make sense without the reader having read everything around it.
- The reader can clearly see the start and end of each piece of information on the page, and their eyes can dart and dash around to whatever catches their attention.

All of this adds to the readability of your CV, and massively reduces the amount of time needed to figure out whether you're a suitable candidate.

# Use the AR format to talk about your achievements.

A common piece of advice for CVs is to use the **STAR** (Situation-Task-Activity-Result) format for each bullet point. I think that this is excessive and can lead a lot of people into writing long sections of prose, which is (as already mentioned) harder to read quickly. I recommend flipping this idea on its head, and including the minimum information needed to make the point. Try starting with just Activity-Result, or **AR**, for each bullet point, and add only as much context as you need for the sentence to make sense.

Here are some examples:

---

### Before

_In July 2019, I actively sought a month-long intensive Italian language course to bring my year abroad to a close. I spent my evenings and weekends developing my language skills through rigorous self-study, as well as orchestrating events and trips for myself and fellow students, including a wine-tasting evening at a local restaurant._

### After (using AR format)

- _Month-long **intensive Italian language course** completed during a year abroad._
- _Organised 4 cultural trips and events for fellow students, **communicating fluently** with local businesses to negotiate a **40% volume discount** for our group's excursions._

---

### Before

_Between March 2018 and September 2019, I was employed at Johnson's News Accounting Centre during university holidays. This was initially a temporary administrative role, but my efficient independent management of a bulk mail output led to my contract being extended and my position changed to Accounts Payable Clerk. I had no prior experience in accounting, but my quick progression demonstrates my ability to quickly adapt to new professional challenges. The role required me to contact suppliers by telephone, process and manage invoice payments, chase senior employees to commence the receipting process, use relevant software systems including SAP, and carry out various administrative tasks. I also voluntarily completed out-of-hours work at busy periods, such as year-end._

### After (using AR format)

- _Independently managed two **national-scale bulk mail outputs** (March 2018 & July 2018)._
- _Success in this administrative project led to my **contract being extended** and being **promoted** to Accounts Payable Clerk._

---

> Apply this idea across your entire CV, and your biggest accomplishments will be screaming out to the hiring manager

If I asked you to tell me the most impressive things mentioned in the _"Before"_ excerpts, it'd probably take you a while, and you'd hate every minute of it. But notice how much easier the _"After"_ versions are to scan, and to pick out key achievements from. Apply this idea across your entire CV, and your biggest accomplishments will be screaming out to the hiring manager - leaving you with a much better chance of making it past résumé screens.

# Results, outcomes, and concrete numbers.

> This is a crucial component of a good CV, and is what elevates your content from a boring description of your duties to a solid sales pitch.

If you're using the STAR format for bullet points, or my trimmed down AR format from above, they both suggest you end on the same point: results. Whatever task or activity you did, what was the outcome? What impact did you have? This is a crucial component of a good CV, and is what elevates your content from a boring description of your duties to a solid sales pitch.

If you "served customers efficiently", why not mention that you reduced the average customer's waiting time by 20%? If you "implemented a smarter checkout system", why not mention that it increased revenues by £100k after it was deployed?

![/static/cv/image_(2).png](</static/cv/image_(2).png>)

The more you can focus on what you **achieved**, rather than what you **did**, the more impressive your CV will seem. Including numbers makes your achievements measurable, and makes them much more credible. They also give a sense of scale, showing the hiring manager just how much impact you could have if they choose to hire you.

# Perfect your formatting and structure.

The format and style of your CV can be a massive boost to how professional you seem. Equally, the greatest text content in the world will be ignored entirely if it's delivered in the professional equivalent of rainbow Comic Sans. It really matters how your CV looks at a glance - it would be a massive shame for poor formatting to prevent your actual profile from even being considered.

## Templates

I find that CVs using a [LaTeX template](https://www.overleaf.com/gallery/tagged/cv) usually look the most professional due to the clean typography and rigid formatting. It's harder for things to look "off" in LaTeX, as by default you're given much less control over layout. One of my favourite starting points (and the one I used as a base for my own CV) is [this template by Sourabh Bajaj](https://www.overleaf.com/latex/templates/software-engineer-resume/gqxmqsvsbdjf). Even if you don't know how to use LaTeX, you can get to grips with it relatively quickly: just clone a template using Overleaf, and start replacing the placeholder content with your own. If you want to avoid LaTeX, I'd recommend recreating a similar look and feel in your word processor of choice.

## Fonts

Keep the number of colours used to a minimum. Black-and-white, or tasteful use of a single color, will prevent your CV from looking too busy. Use **bold** to make keywords stand out.

Avoid using a default font like Calibri. I recommend a serif from [Computer Modern](https://www.fontsquirrel.com/fonts/computer-modern) (this is LaTeX's default font), or a professional sans-serif like [Inter](https://fonts.google.com/specimen/Inter). Google Fonts has a large number of free fonts for you to download, but stick to a maximum of two: one for headings and one for body text. [FontPair](https://fontpair.co/) can help you to choose fonts that look good together.

## Structure

The top of your CV should contain your name, email address and phone number. A short, one-sentence objective summary can be helpful, but usually isn't necessary. **Do not include:** photos of yourself, your address, or "star" ratings of your skills. The first two aren't relevant, take up a lot valuable space, and even open you up to discrimination based on your appearance. "Star" ratings of your skills are [extremely subjective, and essentially meaningless](https://workplace.stackexchange.com/a/75700). Anything less than 4 stars is essentially calling attention to your weaknesses - why bother? Use your bullet points to explain the amazing things you've used your skills to do. Humans respond better to stories than statistics anyway.

> Set yourself up to look great from the first word the hiring manager reads.

![/static/cv/image.png](/static/cv/image.png)

The order of the sections on your CV will depend on what you want to bring attention to. If you're a recent graduate, it makes sense to put Education as your first section. If you're an experienced professional, Work Experience is a great headline section. For a freelancer, it might serve you well to lead with Projects. Set yourself up to look great from the first word the hiring manager reads.

In general, you should not include a section for hobbies unless you're sure they're relevant, or they're evidence of great achievements that will show you in a positive light.

# Summary

In summary, writing a great CV means remembering that it is inherently a sales pitch. The best pitches are brief, focused and easy to follow. With that in mind, keep your CV to one page. Always include the results of what you did, with numbers if at all possible. Pick a template and font that looks professional. Use bold to draw the reader's attention to the important words and phrases. In short, sell yourself well.


Thoughts on...