The State of Automation Testing A Guide For Industry Experts Beyond 2020 “Quality is never an accident; it is always the result of intelligent effort.” – John Ruskin ~ Content Table ~ 1 Abstract Into Today in General 2 Testing Have you heard of it? 3 QA & QC Who is Test Automation For? 4 Manual testing A brief Introduction 5 Automation testing Properties, Applications, and some insight 6 Looking Ahead Balances must be found ~ Abstract ~ We have entered a time in human history where the quality of human life can be directly related to the quality of the software we use. Just about every aspect of human life has been in some way shape or form impacted by Software & Automation. From the face recognition technology used to unlock their I-phones to the local Walmart having their favorite brand of chicken always in stock; humans and machines are becoming more and more interdependent on one another. Machines need humans to create software and humans need machines to use that software to improve their lives. Better Software means better lives for people. So, how can we make our software better? & How can we assure its quality? Or perhaps, Can our software make our software better? & How well can it be done? This school of thought brought about one of the most radical industries of our time, Testing. ~ Testing ~ What is it and why do we need it? Year on year, The World Quality Report has recognized end user satisfaction as the single most important variable in the lifecycle of a software product. In fact, disenchanting users early on in a software’s lifecycle with too many bugs, glitches or just poor experiences, can lead to bad reputation and even serve as a catalyst to a project's failure. At a high level Testing is running quality checks on software products and maximizing user satisfaction before public deployment. By testing software before releasing it, companies are able to maintain peace of mind, assured that their product is of the highest possible standard. Testing is conducted for a multitude of reasons; + Bug Detection + Maximizing user Experience + Making sure Algorithms are tuned correctly + Verifying Robust System Security Measures + Guaranteeing Compliance with standards + Etc... While all the practical reasons behind why a product needs to be tested is fundamentally important, what is more important still, is understanding how testing can be conducted. Testing consists of many fine grained micro types, but from a high level all micro-types fit into two generalized archetypes Functional & Non-Functional methods. Below is an oversimplified expression in figure 2.1. Figure 2.1 For purposes of clarity and coherence we will not be exploring each method under a microscope, here will briefly define 7 major functional methods within the archetypes; the two “Box methods” , the 4 tiered hierarchy familiarized by traditional computer science and the staple regression testing. + Black Box Testing A method of testing where the software is put to use and only the inputs and outputs are observed. Internal workings are left untouched. + White Box Testing A method of testing where the disparate internal mechanisms are scrutinized via programming + Unit Testing A method of testing that is applied to small portions of code, designed to evaluate single modules. + Integration Testing A method of software testing where individual modules are combined in different variants and scrutinized. + System Testing A method of testing which puts the total product up against different malicious and accidental events, deriving the products level + Acceptance Testing A formal process of verifying whether or not the software satisfies the entire criteria put forth by the customer. + Regression Testing Regression is an unintended change in functionality that takes place spontaneously after a change is applied to the product’s code. Regression testing for it is crucial for further growth and development of a product, because it confirms the presence of absence of expected operations. Above we addressed functional testing; what about Non-functional testing? Non-functional Testing is guaranteeing aspects of a product that deal with the following system properties: security, scalability, flexibility, resusablity, interoperability, reliability, potability and the such, rather than its internal behaviors (functionality). Let's take a closer look at 4 fundamental variants of non-functional testing + Load Testing A method of non-functional testing where demands are exerted on a system and its response is measured. + Compliance Testing A method of testing which ascertains a systems conformity with the requirements such as regulation & technical specification. + Stress Testing A testing method that determines how robust a system is; how far beyond its intended operational efficiencies can it handle and continue to operate. + Reliability Testing A simple straightforward process of identifying if software is capable of performing without any failures in a specific amount of time and environment. Aside from the dualities in testing archetype, Software Testing exists in two generalized categories, Manual testing and Automation testing. While at first glance the two compete against one another, after deeper inspection it becomes clear why it is industry standard to adopt a hybrid testing strategy. Where one lacks the other compliments and vice-versa. QA & QA-Testing Quality Assurance is defined by the International Organization for Standardization as the “ part of quality management focused on providing confidence that quality requirements will be fulfilled”. Within corporate organizational structures, QA is the arm of operations dedicated to the provision of customer satisfactions prior to and long after the release of a product. Automation testing is a very demanding narrow skill that belongs to its professional classification. However, the growing demand for experts with more generalized capabilities has pushed Automation Testing into a broad range of roles. Automation testing is no longer just for testers. In an industry survey conducted by practitest, asking testers what their role is we can see how broad of a skillset testing actually is. A Brief Introduction to ~ Manual Testing ~ Manual testing is the time consuming and tedious process of having a person(s), physically sit in front of a screen, comb through the source code of a software product, line by line, in hopes of debugging it as much as possible. While primitive in its nature, manual testing of software has long been considered to be the most effective form of testing because of the capability of the human mind to derive high-level abstract insight that other forms of intelligence might not pick up on. However, as manual testing does provide its own unique suite of benefits, it also provides hindrances to the overall software deployment process. Three predominant ones; - Slow & Time Consuming - Niche Expertise availability & training - Human error First and foremost, manual testing is terrifically time consuming. Whenever an individual must approach the testing process they must inspect the most granular of system code peculiarities. The finding must be recorded initially by hand and one by one log into a data keeping software. Those records must then be safely stored and transported over to other quality assurance agents. This all said does not even account for the time it would take to acquaint a professional with the products documentation. To get a grasp on how much time is spent on the Testing process we can take a look at a survey (figure 3.1) conducted by Practitest Asking thousands of industry professionals “how much of their time is spent on testing” for the fiscal year of 2019. Second in order, manual testing assumes that professionals with adequate skill sets are in house, or at the very least available on call. The amount of time it can take a new person to read, re-read and digest product documentation could put a serious delay on its deployment, not to mention, lower the guarantees associated with a developer's depth and experience with a product. Should an in-house team exist, their cost of labor would far supersede that of a simple developer; Quality Assurance is a personality type, not just a job title. Moreover, Testing has been a field that has been growing in demand year-on-year while the actual availability pool of quality talent hasn’t followed suit equally. Finally, the two issues above have been identified and overcome. Tremendous success. A quality individual was found or a team was put together. They conducted and completed their examinations. The reports have been filed, stored and transferred. Your software hits the open market and not even 24 hours later, users are threatened with litigation because their devices were compromised. Turns out the professional missed a spot, screwed up a single line in the log files and a half-wit hacker caught something in the code, infected it and boom. 12 months of hard work and resources disintegrated. Human error. It really happens. Conducting manual testing should be done in more rounds than one and by more than one sources. After Manual testing is applied, it is time to transition into the next step. The step that removes the human error and monstrous time costs. ~ Automation Testing ~ Autonomous, automatic quality assurance. We are talking about software for software. We are talking about Automation Testing. Automation testing is a variant of software testing that frees up the human in front of the computer from repetitive tasks and streamlines the quality assurance process. Automation testing is commonly thought to be a complete replacement to manual testing. This is a misconception, rather, Automation testing is meant to accompany manual testing in hopes of shortening test cycles, increasing accuracy, and reducing costs. Automation testing is a topic that has been explored since the pre-y2k era. Software that could scan itself, detect errors in itself and ultimately fix itself has always been a coveted concept for Artificial Intelligence maximallists and technology industrialists alike. Artificial Intelligence in automated testing is unlike AI anywhere else. The name of the game is Catch 22. The goal is to create autonomous intelligent software that tests software. But how do we test the testing software? Software Testing (and software as a whole) is a field that depends on leveraging as many new technologies as possible, from NLP & RPM to ML and of course A.I. Through a study conducted by leading intelligence firms Forbes, The Economist, pwc.com & hbr.org, the impact of artificial intelligence across all industries can be measured as depicted below in figure 4.1. Figure 4.1 With benefits as great as these to come from the introduction of Artificial Intelligence into an environment it comes as a shock that the majority of companies involved in testing struggle with AI. Application The application of Automated testing is optimal for projects of larger sizes and that requires repeatedly testing the same segment. Automated tools are able to produce repeatedly accurate testing. The more redundant a process that must be treated, the more likely it is to be a perfect contender for automation. Below is a simplified testing frequency level chart 4.2. Figure 4.2 The more general and complete the portion of a software product being tested the fewer tests it must undergo; meanwhile where software is at its most granular, (the unit level) the more times a process must be repeated across more segments. Not all testing types work best with automated testing, some testing types are just more suited to alternative implementations. As mentioned above, automation testing is best when implemented in large scale projects with multitudes of small moving parts. This helps put 4 general testing methods into clear focus that should be automated; 1. Positive testing Testing where the system with valid input data in expectations of valid output data. 2. End-to-End tests Testing the general bottleneck points of a system. This testing will help guarantee a clean experience from the beginning users session to the very end. (login → shop → checkout → inventory update) 3. Feature Functionality based Tests Testing of a system’s elements (ex. does the autofill function adequately propose english words). 4. UI Tests While it is good practice to save UI testing until much later in a products phase (simply because it will change so frequently & it is not generally crucial to the operations of a product), properly conducting UI tests through automated approaches will boost user satisfaction with cleanliness. (detects limits of text box characters, pagination and so on.) Understanding that simply because a testing method type can be automated does not at all mean that it should be, we can cut out three (3) testing types that should NOT be automated; 1. Ad Hoc testing Testing a system for Ad Hoc properties will consume massive amounts of time and is conducted during the preparatory and construction phases of a system. (ex. Developing a proprietary hashing algorithm requires deliberate and intense mathematical validations during the process of designing it, then again while building it, and then again before handing it over for use). 2. Failover Negative tests Testing that uses invalid data as inputs in expectations of proper outputs. Generally abstract concepts with negative outputs, such as prices being negative (int vs uint), can and do go unidentified by autonomous agents. 3. Cross Party Environment Tests Many newly arriving software products are heavily dependent on multiple strings of 3rd-party API’s. Testing environments with many dynamically adapting threads (of which the actual software products team has no control over) will constantly detect issues that may (or may not) be valid. To understand automation testing and the role it serves it is important to understand its four (4) crucial properties. Any automation testing tool must possess all four of the following properties in order to provide the necessary functions of an ever changing digital environment: Recording & Replay Scalability Interoperability A.I. & ML + Recording & Replay All automated testing equipment must be able to record the entire testing process that happens, store it safely and be able to replay it at any moment. This is a crucial property because it allows for the manual testers to come in and inspect the replayed activity. + Interoperability Testing software must be able to be integrated across any range of technologies and languages; it must be able to operate with maximum efficiency in any environment. + Scalability Scalability is a general broad term used to ascribe unlimited growth across any vertical. Automation testing software must be able to scale things such as its processing speeds, the amount of code tested in parallel and much more. + Artificial Intelligence & Machine Learning A.I. and ML are at the forefront of fully developed, agile, and highly adept automated testing software. What sets apart the good, the great, and the rest is the level of Artificial Intelligence their products possess. As in with the case of manual testing, while automation testing brings about tremendous benefits, it also comes with its own spectrum of shortcomings. Likewise to manual testing, automated tested is plagued by three predominant ones; - Massive upfront costs - Abstract inadequacy - Team (A.I.) Intelligence Deficiencies One of the leading issues within the automated testing industry is the large costs associated with introduction of automated software. Due to the unique nature of each and every single program (no two applications should have identical source codes), every project and product requires automated testing software that is customized to their needs, this warrants all the more expenses with set-up, testing of the testing software, and talent searching. Not far behind, we are met with a deeply technical issue of the industry. This is by no means a problem only of testing software but software as a whole, at the time being, no form of intelligence is capable of operating on the highly abstract planes that humans are capable of. Therefore, errors, bugs and other issues that might arise go undetected because there have not been previously vetted as being problematic. Finally we have the notorious A.I. uncertainties & team intelligence deficiencies. As mentioned above in the Artificial intelligence sliver portion in Automation testing, A.I. & ML are the drivers of explosive software development. Its techniques are being used by every industry giant and participants of the software industry that do not apply A.I. are handicapping themselves. ~ Looking Ahead ~ Heading beyond 2020, testing as a field will continue to grow and continue to be one of the fundamental roles in the software development industry. Automated testing will not (should not) completely erase manual testing. However, it is safe to say that manual testers that understand how to implement automated testing will be much more of a valuable resource than those who do not. We know that entirely relying on one form of testing is sub-optimal & that a healthy testing scheme lies somewhere on the manual ←→ automated spectrum. Hybrid is the answer. As testing continues to grow and develop so will the tools associated with it and its applications. Circa 2020, almost every field of technology is and will be dependent on the role that automation testing will play in its growth.