Welcome!

WebRTC Summit Authors: Liz McMillan, Elizabeth White, Pat Romanski, Roger Strukhoff, Yeshim Deniz

Related Topics: Java IoT

Java IoT: Blog Feed Post

Is Test Overlap A Necessary Evil?

The overlap is not total because a layer tend to reduce the granularity of the unhappy paths it faces internally

In a recent blog post titled "The Limitations of TDD", Jolt Awards colleague Andrew Binstock shared some reservations Cédric Beust has about TDD.

When a person of extensive experience like Cédric speaks about testing, you pay attention. And I did.

Among the very interesting quotes from Cédric that Andrew has reproduced, the following really struck me:

Another important point is that unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools.

You also bring up another interesting point: overtesting can lead to paralysis. I can imagine reaching a point where you don't want to modify your code because you will have too many tests to update (especially in dynamically typed languages, where you can't use tools that will automate this refactoring for you). The lesson here is to do your best so that your tests don't overlap.

Trust me, as a test-infected developer, I would love to stay in a state of self-delusion and pretend that test-induced paralysis doesn't exist. But that would be a lie: the reality is grimmer than the wonderland of testing I would wish to live in. The reality is that tests both encourage and resist change.

On the one hand, tests encourage and support refactoring: when the behavior of the application should not change but the code needs to be re-organized, tests are a blessing. They give you the courage to dare changing code because of the immediate feedback they give when you've been refactoring a little too aggressively. And this is priceless.

On the other hand, tests resist behavioral changes. Because tests have captured all the nitty-gritty of your application, when comes the time to change its behavior, you will need to invest time to adapt your tests accordingly, and this whether you rework the tests first or not. As Cédric pointed out, in a dynamically typed language, this is immensely painful as development tools are almost useless in assisting you with the required changes. Similarly, if you use mock objects, you are good for going down a deeper Circle of Hell, where more painful and frustrating manual fixes await you.

So, is there any hope out of this love / hate relationship? Knowing that "the only way to go fast is to go well" dumping tests altogether is certainly not an option. Could the solution lies in Cédric's very last words: "do your best so that your tests don't overlap"?

At this point, I don't know yet but I've decided that, as a starting point, I should start to estimate the amount of overlap I'm dealing with in the Erlang game server I'm working on. Interestingly, what I've found could pretty much apply to the vast majority of Java projects I've been previously working on. Maybe it applies to your projects too?
The first thing I've looked at is the testing overlap that exists between two layers of our application:
As you can see, the overlap exists because tests of the upper layer rely on mocks to simulate all the happy paths and most of the unhappy paths of the underlying layer.

The overlap is not total because a layer tend to reduce the granularity of the unhappy paths it faces internally in order to expose the upper layer to a limited amount of bad situations to deal with.

Hence the limited amount of mocked features in the overlap area.

When applied to a typical vertical slice of our system, it looks like this:

This is not too bad. Until the wind of feature change comes blowing on this mock-based card-house of tests, life is peachy.

Until now, the tests I have been looking at were only unit and database ones. If I add our functional tests on top of the overlap diagram, here is what I get:

Now the application container is also tested, plus we get an insane amount of overlap.

But the amount of overlap is not what I want to discuss first: it's the test coverage profile that I want to look at first. Notice how the functional tests explore less unhappy paths as they exercise deeper application layers. This can be explained simply: some unhappy paths are very hard to reproduce via the reduced set of functionalities exposed at the top level, oftentimes because they require a very specific and complex state to be established beforehand or conditions that could only be met in case of low level failures (loss of networking, for example).

It's obviously out of the question to consider dropping functional tests in order to reduce the testing overlap. As Cédric said, they are the only tests that have a true value for the end user of the system. My experience confirms that you can reach a nearly flawless first-time client integration if your functional tests have a coverage profile that is similar to the one in the last figure above.

The only problem lies in the quality of feedback you get from functional testing: because it's impossible to make the gory details of the errors encountered when exploring unhappy paths surface at the uppermost level, your system must have a solid logging strategy that allows you to precisely track issues, should you decide to code using functional tests as your only safety net.

So are the unit tests overlapped by the functional tests the ones that must go? Cédric again gives the answer: if time is short, it's better to focus on the functional tests. Of course, if you have a battery of unit tests in place, keep them.
But, maybe, just maybe, as you move to your next project, consider writing functional tests firsts? That way you would have built first the tests that truly matter and, if time permits, write unit tests as you implement the features expected by the functional tests.

Read the original blog entry...

More Stories By David Dossot

David Dossot has worked as a software engineer and architect for more than 14 years. He is a co-author of Mule in Action and is the project despot of the JCR Transport and a member of the Mule Community Committee. He is the project lead of NxBRE, an open source business rules engine for the .NET platform (selected for O'Reilly's Windows Developer Power Tools). He is also a judge for the Jolt Product Excellence Awards and has written several articles for SD Magazine. He holds a Production Systems Engineering Diploma from ESSTIN.

@WebRTCSummit Stories
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buyers learn their thoughts on their experience.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented a comprehensive view of the numerous testing challenges researchers have faced before arriving at the first release candidate of the WebRTC specifications.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multiple Intel RealSense Depth Camera to scan 3D and build 3D models in real-time, and display as hologram in front of remote participants.
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone innovative products that help customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business and personal computing needs.
SYS-CON Events announced today that Google Cloud has been named “Keynote Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Companies come to Google Cloud to transform their businesses. Google Cloud’s comprehensive portfolio – from infrastructure to apps to devices – helps enterprises innovate faster, scale smarter, stay secure, and do more with data than ever before.
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interoperability. It has lots of potential to address current issues around IoT - security, connectivity and so on - based on P2P technology. It will become a key-component especially in edge computing use cases, in his view.
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal today!
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
22nd International Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, and co-located with the 1st DXWorld Expo will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterprises are using some form of XaaS – software, platform, and infrastructure as a service.
DevOps at Cloud Expo – being held June 5-7, 2018, at the Javits Center in New York, NY – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real results. Among the proven benefits, DevOps is correlated with 20% faster time-to-market, 22% improvement in quality, and 18% reduction in dev and ops costs, according to research firm Vanson-Bourne. It is changing the way IT works, how businesses interact with customers, and how organizations are buying, building, and delivering software.
@DevOpsSummit at Cloud Expo, taking place June 5-7, 2018, at the Javits Center in New York City, NY, is co-located with 22nd Cloud Expo | 1st DXWorld Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
SYS-CON Events announced today that T-Mobile exhibited at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on quality and value. Based in Bellevue, Washington, T-Mobile US provides services through its subsidiaries and operates its flagship brands, T-Mobile and MetroPCS. For more information, visit https://www.t-mobile.com.
SYS-CON Events announced today that Cedexis will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Cedexis is the leader in data-driven enterprise global traffic management. Whether optimizing traffic through datacenters, clouds, CDNs, or any combination, Cedexis solutions drive quality and cost-effectiveness. For more information, please visit https://www.cedexis.com.
SYS-CON Events announced today that Vivint to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. As a leading smart home technology provider, Vivint offers home security, energy management, home automation, local cloud storage, and high-speed Internet solutions to more than one million customers throughout the United States and Canada. The end result is a smart home solution that saves you time and money and ultimately simplifies your life.
SYS-CON Events announced today that Opsani will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Opsani is the leading provider of deployment automation systems for running and scaling traditional enterprise applications on container infrastructure.
SYS-CON Events announced today that Nirmata will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nirmata provides a comprehensive platform, for deploying, operating, and optimizing containerized applications across clouds, powered by Kubernetes. Nirmata empowers enterprise DevOps teams by fully automating the complex operations and management of application containers and its underlying resources. Nirmata not only simplifies deployment and management of Kubernetes clusters but also facilitates delivery and operations of applications by continuously monitoring the application and infrastructure for changes, and auto-tuning the application based on pre-defined policies. Using Nirmata, enterprises can accelerate their journey towards becoming cloud-native.
SYS-CON Events announced today that Opsani to exhibit at SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California. Opsani is creating the next generation of automated continuous deployment tools designed specifically for containers. How is continuous deployment different from continuous integration and continuous delivery? CI/CD tools provide build and test. Continuous Deployment is the means by which qualified changes in software code or architecture are automatically deployed to production as soon as they are ready. Adding continuous deployment to your toolchain is the final step to providing push button deployment for your developers.