Software and Applications Engineers
Design, build and maintain software applications and cloud-based services, and facilitate collaboration and integration across different specialist ICT teams.
The night the website slowed down
You are a software engineer on a small team at an online grocery company, and it is the busiest night of the week. It is 5:30pm on a Thursday. Orders are flooding in for the weekend, and customers are starting to complain on social media that the site is slow. You are the engineer on call.
Choose what you would do. Then see what happens. There are often good reasons for each choice.
We're getting dozens of messages from people who can't check out. They say the page just spins. Our phone lines are lighting up. Do we know what's going on yet? What can I tell them?
Tell her it is a problem with the customers' own internet connections
Pia repeats it, and customers are told it is their fault. When it is clearly the site's problem, the company looks dishonest on top of slow.
Tell her you are looking into a slowdown, you do not have a cause yet and you will update her in fifteen minutes
Pia can tell customers something true, but she was hoping for a message she could post. It is honest, though a line she could use right away would have helped more.
Give her a short, honest message to post, say you are working on it and agree a time for the next update
Pia posts the message, and the angry comments soften. You get fifteen minutes of focus to work on the real problem. Communication buys you time.
The database is running at its limit. A change released this afternoon looks like the only thing that is different from yesterday. You can undo it in a minute, but it also fixed a problem for a big customer.
Undo the afternoon change, tell the team and look into the cause afterward
The site recovers within minutes. The big customer's problem returns, which you flag to the team with a plan. Restoring service first, then investigating, is the usual order in a crisis.
Leave the change in and try to tune the database instead
You gain a bit of speed, but the site stays slow for another hour. You were trying to protect the fix, and the customers still suffer.
Add more servers and wait to see whether it improves
It helps a little, and the bill grows. The underlying cause is still there, so the next busy night will bring the same problem.
I think the change I wrote this afternoon caused it. I'm so sorry. I didn't test it with this many orders. Should I just fix it quietly in the morning so we can move on?
Tell him that is fine, and that it is better to say nothing so the team does not worry
He feels relieved, but the lesson is lost and the team learns nothing. The same mistake appears three months later, and nobody knows it has happened before.
Thank him for telling you, share what happened with the team without blame and ask him to help write the fix
The team learns and Tomasz stays involved, but he feels exposed when the story is told. A conversation with him first would have helped.
Thank him, ask what he would change next time, then plan a check for heavy load
Tomasz leads a short talk the next day about testing at scale, and the team adds a check to the release. People stay honest when mistakes become lessons, not punishments.
How it went
The site is back, the team has a new check, and a newer engineer felt safe enough to speak up. Software engineers spend as much time reading, testing and talking as they do writing code, and the best ones care about the people on the other end. Is that the kind of problem-solving you would enjoy?
This is a simplified, fictional situation to help you explore. It is not professional instruction or a test of your ability.
Case study: The Friday release that waited until Monday
A fictional case set in a product team at a fictional online grocery delivery company, with developers, a designer and a product manager.
The situation
Anjali was a software engineer at Fernwood Fresh, an online grocery company. On a Friday afternoon, her team finished a change to how delivery times were shown. The product manager wanted it live before the weekend, which is the company's busiest time. Anjali had run the tests and they all passed, but she had a nagging feeling about how the change would behave with orders that were edited after being placed.
What they did
Anjali told the team she wanted to test that one extra case before releasing, and that it might push the release to Monday. The product manager was annoyed. Anjali spent an hour setting up the case and found that edited orders would show the wrong time. She wrote a fix, tested it, and agreed with the team to release on Monday morning when everyone could watch it.
How it turned out
The release went out on Monday without problems. The product manager admitted the weekend would have been hard if the bug had gone live. Anjali's team agreed to a simple rule: nothing risky goes out on a Friday afternoon, unless there is a very good reason.
What would you do?
- How would you feel about holding up a release when everyone wanted it finished?
- What might a gut feeling about code actually be telling you?
- Why might a good engineer delete code as often as they write it?