If a p-value of 0.05 (5%) is too high, and the inventor of that threshold says it shouldn’t be used without consideration–which it is, all the time–then things we think are significant today just aren’t. We don’t always apply this kind of statistical rigor to decision making, but when we do we’re generally satisfied with this threshold. But we shouldn’t be.
Applied to product development teams, methodology matters a lot. But it can matter so much less than the talent of the individuals on a team. But methods and tools are still championed much more loudly than professional development.
In short, we should be looking for more obvious wins, and recognize that when something is close, but still statistically significant, it’s probably still too close to really move the needle.