Here are 6 practical lessons learned from a production incident, ui-test coverage (using playwright) and code review.

Today we had a small incident in production – one of my teammates found out a feature in one of our component doesn’t work. This is a small retro here for the value of ui-tests (using playwright, but not just) and even tdd in ui-tests.

Debugging the Issue

As noticable from the message above, in the banner component, the remove button stopped working. Quick debugging showed that there was a “typo” in the scss file:

The fix was easy, and manually testing this gave the wanted result – the element was removed with a nice transition animation.

So far, usual case. Can push my change and call it a day. The PR was even approved as it seemed to be working.

Then, my brain started yelling at me – “this bug happened, you should test it!“

How to NOT write a ui test

So… because unit tests covered the functionality (e.g. if an event was emitted, things should happen), and this is clearly a CSS issue, I’ve written a ui-test using our Ultra Marvelous Visual-Regression Behavior-Testing Playwright System.

This test made sure that after clicking the removal button, the element’s height is just like it should be from CSS. 

And it worked (a.k.a. tests are green) but… I forgot what was really supposed to happen!

During the code review, the reviewer mentioned the test should be “remove from DOM” and not “see that height is 0”.

How to write the tests in Playwright?

Nice catch! Let’s just change the test.

Playwright exposes the locator API that keeps track of the count of elements in the DOM with the wanted selector:

const element = await page.locator('vwc-banner');
numberOfBannerElementsNow = await element.count();

How cool is that? Now the test looks like this:

aaaaaaand…. it failed…

Finding the truth thanks to failing tests

The test that was supposed to be my salvation failed.

But why? Manual test showed that the animation worked and we could not see the banner!

The reason is this – the element was never removed from the DOM (expected 0, but got 1 – classic!).

Debugging this I found out that the element was listening to the animationend event, but the css was was using transition for animation!  Hence, animationend will NEVER fire!

So, changing animationend to transitionend made the test pass:

I also needed to change the implementation detail of the preparation phase in the unit tests, but that was just like changing animationend to transitionend in one place.

You can view the full PR here.

Summary and takeaways

Here are the lessons I learned from this incident:You can practice TDD in ui-tests. It is even beneficial (I would have written the test better at onset as well as noticed the animation -> transition issue faster)

There are a few lessons that I’d like to share from this incident.

First of all, it was the first time I saw how one can practice TDD in ui-tests. Until today, I always thought of ui tests as just testing the final styling implementation. This case shows how beneficial that can be: I would have written the test better at onset as well as noticed the animation -> transition issue faster.

Unit tests and ui-tests (e2e or however you call them) are complementary. We have 100% unit tests coverage, but when it comes to test integration with the browser ui-tests are there to give the whole picture.

When fixing a bug, always write a test that covers that bug – don’t let a bug happen twice!!!

Another piece of knowledge is that there is a difference between transition and animation events. Be aware…

As for the playwright API, prefer the playwright locator API instead of the $ api – it has features that make testing much easier and more intuitive.

Finally, tests are good for us! Enjoy them!

Thanks to writing a test and not just sharing a fix, we, as a team, managed to find a bigger bug and prevent the occurrence of said bug in the future.

Tests are good for us! Enjoy them!

Thanks a lot to Miki Ezra Stanger  the kind and thorough review!

Featured Photo by Carl Jorgensen on Unsplash