Yes, sorry for the trouble. We do have automated regression testing but we had a hole in our coverage of this case. We’ve added a test for this so it shouldn’t have an issue going forward.
It’s great that you were able to fix it, but having a hole in your coverage points to a much larger issue that perhaps you guys aren’t addressing? Why was there a hole in the first place? Do you guys perform code reviews? Usually an event like this triggers a CAPA, whereby your company needs to look at it’s processes, determine the root cause, and perhaps make changes to avoid this happening in the future. I’m not sure you guys have critically examined why there was a hole in the first place and what you can do to avoid something like this happening again? Any thoughts on beyond simply putting out this fire?
Thank you for being our customer and for being active on this forum. We take testing extremely seriously and reliability is of utmost importance to us (our engineering culture) and, of course, our customers. We utilize extensive testing at both the unit level and in staging changes to test what will happen within a production environment. Accordingly, we navigate the vast majority of changes to our offering while preventing any noticeable effects, on an almost daily basis when considering the breadth of our offering.
In this incident, we encountered an unanticipated edge case in production that testing did not adequately prevent. This incident was immediately escalated to the highest levels of our org, and a fix was built, underwent testing, and was deployed. I, personally, reviewed the root cause and fix, and our testing processes will be amended to ensure prevention in the future.
Again, apologies for the inconvenience and frustration this caused, and we have implemented the fixes and process changes necessary to address the reason this incident occurred.