Console

How engineering works at Trunk

How are the teams structured?

We are not a hierarchical organization. Currently, we have two sub-teams, a client team that focuses on C++ work and another team that focuses on anything web based (both backend and frontend).

We think that when stand-ups grow too big it just becomes people standing around talking and no one’s paying attention. We try to organize team stand-ups in small units so that everyone on the call cares about what other people are saying. As the teams continue to grow over time, we’ll continue to shard our working groups so they scale for focus.

Tech stack

• Frontend: Typescript, React, Redux, Next.js
• Backend: Typescript, Node, AWS, CDK, k8s, gRPC
• Observability: Prometheus, Grafana, Kiali, Jaeger
• CI/CD: GitHub Actions
• CLI/Daemon/LSP: C++20, Bazel, LLVM
• VSCode Extension: Typescript
• General: GitHub, Slack, Linear, Slite

What tools do engineers use?

• Machines: Everyone has a Macbook to dev locally on and a remote Linux VM in the cloud
• Editors: VS Code or vim
• Source control: Git & GitHub
• Deployment: GitHub Actions (self-hosted runners)
• Issue tracking: Linear
• Internal documentation: Slite
• Error reporting: Sentry
• Infrastructure: AWS
• Communication: Slack

Can developers pick their own tools?

We believe in all things product in being opinionated but configurable. That product focus carries forward into our internal development practices. We don’t let engineers haphazardly choose their own languages or randomly pick different services. We believe that the more you can move as a team together, the more efficient you are. Our current hardware stack is MacBook M1 machines. Engineers are encouraged to use VS Code, not only to create a unified developer experience but to guarantee our own VS Code extension gets put through its paces during development.

If you decide to go off those rails, we’ll let you do it. Some of our developers use Vim, but I think often the more variation there is in people’s development environments, the more they’re going to end up being less efficient. We have a strong preference, but ultimately developers can choose if they want to.

How does the development process work? What’s the process for working through bugs, features and tech debt?

• Bugs, features, and tech debt are ticketed using Linear
• We work in bi-weekly sprint
• At the start of every sprint, we have a sprint planning meeting. In that meeting
  • We organize and prioritize backlogs.
  • Go through new tickets added during the sprint and discuss
  • Engineers are assigned tickets and they run with it. Sometimes, we lay out a rough sketch of how to solve a particular problem.
• We also balance inbound customer requests and continuously rebalance priorities to ensure customer facing bugs are squashed while maintaining new feature release velocity.

How does code get reviewed, merged, and deployed?

• All code gets code reviewed, no matter whose it is. In reviews, we make sure that
  • Code is legible to other people
  • Tests are written (when appropriate)
  • Code organization is consistent with the rest of the repo
• Merging happens through Trunk Merge, our merge queue. It adds another layer of testing before automatically merging code to ensure there are no logical conflicts between multiple PRs.
• In general, we merge a lot of code. Engineers at Trunk average at least two PRs per day.
• Deployments are all automated as CI jobs via GitHub actions. Some need to be manually kicked off and given a version, and others run automatically via continuous deployment.

What is the QA process?

All of our testing is automated, we have a very high bar for writing unit and integration tests for new code.. We have layers and layers of increasingly high level tests that mostly run as part of CI on every PR. Some of the slower tests run in hourly jobs or nightly jobs.

• Unit tests
• Smoke tests
• Upgrade tests
• Integration tests
• Higher level integration tests: We clone popular repos in containers and run Trunk through its paces on themIf code passes all these tests, it should be good to go. We plan a release roughly every two weeks where we cut off head off of main and ship it.

What are some recent examples of interesting development challenges solved by internal teams as part of building the product?

We have Mac runners that run tests all day long, deployed on MacStadium. They don’t spin up and spin down, which tends to accrue garbage if you have processes not shutting down cleanly.

Recently we upgraded to a new version of LLVM. On Rosetta, the x86 binaries were taking 10 seconds to build instead of 3 seconds. Our tests all basically assume that the system will connect up within three seconds, but that was timing out. These timeouts meant the Trunk daemon kept respawning so we ended up with thousands of them. This crashed the kernel of the OS and the system would reboot. Tracking this down was complicated!

How does on-call work?

The on-call is just a rotation through the service engineering work. Engineers cycle through support tickets during normal working hours.