“It’s a brand new protocol, and it’s taking all the best ideas that have been out there for a while and trying to standardize it through the W3C,” said David Burns, head of open source at BrowserStack (a browser testing company that is on the WebDriver BiDi working group) and chair of the Browser Testing and Tools Working Group at W3C, which is the group responsible for the WebDriver and WebDriver BiDi specifications. 

The original WebDriver protocol, or WebDriver Classic, is a “remote control interface that enables introspection and control of user agents,” according to its W3C definition. Essentially, it provides a way to remotely control the behavior of web browsers so that applications can be tested in them. 

However, this protocol only offers one-way communication, meaning that the client sends a request to the server, and the server can reply only to that one request, explained Puja Jagani, team lead at BrowserStack and a key code committer for the WebDriver BiDi project.

“The server cannot initiate communication with the client but can only respond. So if something of interest happens in the browsers it cannot communicate back to the client unless the client asks for it,” explained Jagani.

The BiDi in WebDriver BiDi stands for bidirectional communication, meaning that it actually allows events in the browser to stream back to the controlling software.

According to Jagani, because browsers are event-driven, it’s helpful for the browser to be able to share events back to the client when something interesting happens. 

For instance, with this new protocol, users can subscribe to the events created when a network request is sent to or from the browser, which enables them to monitor (or modify) all outgoing requests and incoming responses.

An example of this in action involves an application that is pointing to a production database in the cloud. When testing that application, WebDriver BiDi could be used to modify outgoing requests to point to a test database so that the production database isn’t flooded with test data.

“This is only possible with bidirectional communication. It is not possible without the W3C BiDi protocol,” said Jagani.

CDP vs WebDriver

The Chrome DevTools Protocol (CDP) and WebDriver Classic have historically been often compared because they are both low-level tools — tools that execute remote commands outside of the browser, such as opening multiple tabs or simulating device mode, Jecelyn Yeen, senior developer relations engineer for Chrome, and Maksim Sadym, software engineer at Google, explained in a blog post. 

High-level tools, by contrast, are those that execute commands within the browser. Examples of these include Puppeteer, Cypress, and TestCafe.

CDP does enable bidirectional communication, but it’s limited for testing purposes because it only works for Chromium-based browsers, like Google Chrome, and wouldn’t work in Firefox or Safari. According to Yeen and Sadym, “WebDriver BiDi aims to combine the best aspects of WebDriver ‘Classic’ and CDP.”

However, BrowserStack’s Burns emphasized that this new protocol isn’t intended to replace CDP, but rather it’s a new testing and automation protocol entirely. “CDP is always going to be there on Chromium browsers,” he said.

It already has browser support 

CDP’s creator, Google, is heavily involved in developing and supporting WebDriver BiDi, as is Mozilla. “We are glad that Mozilla and Google have come and helped us get it to that point where it’s standardized and now everyone can benefit from it,” Burns said. He added that Apple isn’t quite there yet, and it’s not clear at the moment when support for WebDriver BiDi will be available in WebKit-based browsers. 

“Sometimes standards can move at a glacial pace, and part of that is for good reason. It involves creating the collaboration points and getting consensus — and sometimes consensus can be really hard, especially where Google, Mozilla, and Apple, they have their own ideas of what makes something better, and so getting that can be really, really slow to implement,” Burns explained. 

Testing automation tools and testing companies have also started supporting it

In addition to the browsers needing to support it, another piece of the puzzle is getting the testing automation tools and testing providers on board. Fortunately, the automation tools Selenium and WebDriverIO, as well as the testing companies BrowserStack, SauceLabs, and LambdaTest, are all part of the WebDriver BiDi Working Group. 

WebdriverIO and Selenium already have some support for the new protocol, and BrowserStack supports it too. Selenium itself is also updating its entire implementation from WebDriver to WebDriver BiDi. Burns explained that retrofitting the classic version of WebDriver to BiDi is the last major piece of the process, and is expected to be complete within the next year. 

“It’s a volunteer-driven project, so this happens when everyone’s bandwidth and time matches, so it gets done in like spurts or chugs of work, right? But I think that’s how it is for open source development in general,” said Jagani, who is also a member of the Selenium Technical Leadership Committee.

She noted that by Selenium 5 (the current version is 4.24), the goal is to have at least the high-level APIs done, which cover a number of use cases, like giving the user the ability to listen to console logs and the ability to do basic authentication for their website, to name a couple.

Once Selenium 5 is out, the next goal will be to start transitioning commands one by one from WebDriver Classic to WebDriver BiDi. “Hopefully, by Selenium 6, we are BiDi only,” she said. She did go on to explain that it’s a long process with many external variables. Browsers are still in the process of implementing it, and once BiDi is in the stable version of the browser, that’s when Selenium comes in and can start implementing it. After that, there’s still a period where users will need to use it and give feedback so that Selenium can ensure its implementation is resilient.

Jagani said that the user experience should remain the same once Selenium is switched over to BiDi, and there won’t be a big breaking change. 

“That’s what Selenium tries to do — even from Selenium 3 to 4 — we try to make sure it’s a seamless integration with minimal breaking changes,” she said. “Selenium is very big on backwards compatibility as much as possible, or at least ensuring that we’re deprecating things as required so you know we are going to be removing it and giving sufficient warnings. That experience for users using WebDriver Classic would remain the same, because eventually it’ll be the same APIs, just using BiDi under the hood.”

To take advantage of the new advanced capabilities that BiDi brings, there will be newer APIs available, which will be similar to the ones users are already familiar with. 

The post WebDriver BiDi offers the best of both worlds in browser automation tools appeared first on SD Times.

The Shift in Perception: A Story from Typemock

In the early days of AI-driven unit testing, Typemock encountered significant skepticism. When we first introduced the idea that our tools could automate unit tests using AI, many people didn’t believe us. The concept seemed too futuristic, too advanced to be real.

Back then, the focus was primarily on whether AI could truly understand and generate meaningful tests. The idea that AI could autonomously create and execute unit tests was met with doubt and curiosity. But as AI technology advanced and Typemock continued to innovate, the conversation started to change.

Fast forward to today, and the questions we receive are vastly different. Instead of asking whether AI-driven unit tests are possible, the first question on everyone’s mind is: “Is the code sent to the cloud?” This shift in perception highlights a significant change in priorities. Security and data privacy have become the primary concerns, reflecting the growing awareness of the risks associated with cloud-based AI solutions.

RELATED: Addressing AI bias in AI-driven software testing

This story underscores the evolving landscape of AI-driven testing. As the technology has become more accepted and widespread, the focus has shifted from disbelief in its capabilities to a deep concern for how it handles sensitive data. At Typemock, we’ve adapted to this shift by ensuring that our AI-driven tools not only deliver powerful testing capabilities but also prioritize data security at every level.

The Risk of Intellectual Property (IP) Leakage

1. Exposure to Hackers: Proprietary data, if not adequately secured, can become a target for hackers. This could lead to severe consequences, such as financial losses, reputational damage, and even security vulnerabilities in the software being developed.
2. Cloud Vulnerabilities: AI-driven tools that operate in cloud environments are particularly susceptible to security breaches. While cloud services offer scalability and convenience, they also increase the risk of unauthorized access to sensitive IP, making robust security measures essential.
3. Data Sharing Risks: In environments where data is shared across multiple teams or external partners, there is an increased risk of IP leakage. Ensuring that IP is adequately protected in these scenarios is critical to maintaining the integrity of proprietary information.

The Permanence of Data in AI Models

1. Inability to Unlearn: Once AI models are trained with specific data, they retain that information indefinitely. This creates challenges in situations where sensitive data needs to be removed, as the model’s decisions continue to be influenced by the now “forgotten” data.
2. Data Persistence: Even after data is deleted from storage, its influence remains embedded in the AI model’s learned behaviors. This makes it difficult to comply with privacy regulations like the GDPR’s “right to be forgotten,” as the data’s impact is still present in the AI’s functionality.
3. Risk of Unintentional Data Exposure: Because AI models integrate learned data into their decision-making processes, there is a risk that the model could inadvertently expose or reflect sensitive information through its outputs. This could lead to unintended disclosure of proprietary or personal data.

Best Practices for Ensuring Data Privacy and Security in AI-Driven Testing

Protecting Intellectual Property

To mitigate the risks of IP leakage in AI-driven testing, organizations must adopt stringent security measures:

• On-Premises AI Processing: Implement AI-driven testing tools that can be run on-premises rather than in the cloud. This approach keeps sensitive data and proprietary code within the organization’s secure environment, reducing the risk of external breaches.
• Encryption and Access Control: Ensure that all data, especially proprietary code, is encrypted both in transit and at rest. Additionally, implement strict access controls to ensure that only authorized personnel can access sensitive information.
• Regular Security Audits: Conduct frequent security audits to identify and address potential vulnerabilities in the system. These audits should focus on both the AI tools themselves and the environments in which they operate.

Protecting Code Structure with Identifier Obfuscation

1. Code Obfuscation: By systematically altering variable names, function names, and other identifiers to generic or randomized labels, organizations can protect sensitive IP while allowing AI to analyze the code’s structure. This ensures that the logic and architecture of the code remain intact without exposing critical details.
2. Balancing Security and Functionality: It’s essential to maintain a balance between security and the AI’s ability to perform its tasks. Obfuscation should be implemented in a way that protects sensitive information while still enabling the AI to effectively conduct its analysis and testing.
3. Preventing Reverse Engineering: Obfuscation techniques help prevent reverse engineering of code by making it more difficult for malicious actors to decipher the original structure and intent of the code. This adds an additional layer of security, safeguarding intellectual property from potential threats.

The Future of Data Privacy and Security in AI-Driven Testing

Shifting Perspectives on Data Sharing

While concerns about IP leakage and data permanence are significant today, there is a growing shift in how people perceive data sharing. Just as people now share everything online, often too loosely in my opinion, there is a gradual acceptance of data sharing in AI-driven contexts, provided it is done securely and transparently.

• Greater Awareness and Education: In the future, as people become more educated about the risks and benefits of AI, the fear surrounding data privacy may diminish. However, this will also require continued advancements in AI security measures to maintain trust.
• Innovative Security Solutions: The evolution of AI technology will likely bring new security solutions that can better address concerns about data permanence and IP leakage. These solutions will help balance the benefits of AI-driven testing with the need for robust data protection.

Typemock’s Commitment to Data Privacy and Security

At Typemock, data privacy and security are top priorities. Typemock’s AI-driven testing tools are designed with robust security features to protect sensitive data at every stage of the testing process:

• On-Premises Processing: Typemock offers AI-driven testing solutions that can be deployed on-premises, ensuring that your sensitive data remains within your secure environment.
• Advanced Encryption and Control: Our tools utilize advanced encryption methods and strict access controls to safeguard your data at all times.
• Code Obfuscation: Typemock supports techniques like code obfuscation to ensure that AI tools can analyze code structures without exposing sensitive IP.
• Ongoing Innovation: We are continuously innovating to address the emerging challenges of AI-driven testing, including the development of new techniques for managing data permanence and preventing IP leakage.

Data privacy and security are paramount in AI-driven testing, where the risks of IP leakage, data permanence, and code exposure present significant challenges. By adopting best practices, leveraging on-premises AI processing, and using techniques like code obfuscation, organizations can effectively manage these risks. Typemock’s dedication to these principles ensures that their AI tools deliver both powerful testing capabilities and peace of mind.

The post Data privacy and security in AI-driven testing appeared first on SD Times.

In addition to coding, a developer’s time is spent on:

• planning and design
• meetings and collaboration
• testing and debugging
• code reviews
• documentation
• learning and training

The good news is that AI can impact every aspect of a developer’s day, as well as the other members of the development team.

The first two items are closely related as most of the time spent in meetings is focused on planning and design. When it comes to documents and specifications, Agile practices espouse a philosophy of “just enough, just in time.” This philosophy impacts the quality of written documentation. Often developers are given a rough idea of the requirements and then end up in a slew of meetings to ensure the vague descriptions are understood. It would be better and more efficient to start with clear, complete specifications. 

Generative AI is very good at creating requirements that include all the detail a developer needs to implement a feature. It is also good at summarizing information. By using AI to generate complete requirements, development teams can reduce the amount of meeting time and collaboration required. The developer doesn’t even have to read the full specifications. They can read a summary and then ask the AI to clarify any questions. In fact, these specifications can feed directly into the code copilot to automate the first iteration of code generation.

Testing also benefits from better documentation and Generative AI as well. Just as copilots can generate code from clear specifications, they can also generate the tests directly as well. Since the tests are generated from the same specs, they are available to validate the functionality of the code when it is complete. In fact, AI can execute the tests and summarize these results, which saves time for both the developer and the test engineer. 

This brings us to code reviews. When most of the code is generated by AI, code reviews by humans will become even more important. In fact, developers will likely spend a higher percentage of their time reviewing AI generated code, since the other activities of their day will be minimized. 

Much of the drudgery of code reviews can and should be eliminated however, whether the code is generated by humans or AI. Static code analysis tools can ensure the code is written according to the company standards and properly documented. Static security analysis and open source compliance tools can ensure that most of the well-known vulnerabilities are eliminated with no human intervention. As a result, the human code reviewer can focus on the high-level design. AI can help with review as well, providing the developer with an overview and guided tour of how the code was created, which further eliminates some of the documentation work that the developer is normally required to do. 

The proper architecture of the code can be ensured through better specifications up front as well. When a human developer writes code, much of the architectural decisions are in their head as they are writing the code. For very complex projects, they may produce some rough documentation, but for most of their work, they wing it. With the advent of copilots, there is a need to specify the high-level design of code in a way that is easily created by the architect and understood by AI. This should be a growing area of innovation over the next year. 

The net result will be a shift in the way developers spend their day. A higher percentage of their time will be spent on architectural design and code review relative to what they do today. The code review will be easier and less time consuming due to the AI assist. Functional testing will be easier since a test script will be provided before the code is even produced and less time will be spent in meetings and writing documentation. 

The net impact on ROI will be an increase in value delivery to the business with a reduced investment cost per feature. There will be an improvement in the quality of changes delivered and reduced risk of down time. This improved ROI is not due solely to the adoption of a code generation copilot, but to the use of AI in all aspects of the development process, especially in the front-end planning phase. Improved ROI comes not only from the productivity improvement of developers, but from improvements in the business analyst and testers as well. 

To measure the impact of Generative AI on the development process, start with the DORA metrics that we all know and love. The two most important metrics for developers are Lead Time to Change and Change Failure Rate. If changes require half the time to make it to production at the same or better failure rate, then you have doubled your developer’s productivity. 

When you streamline the development process, you should be able to see an improvement in Change Failure Rate. This improvement in velocity should also mean that your team has more time available to iterate on their designs before release, resulting in more usable software and happier users. It’s difficult to measure the true value of happy users, but it should improve productivity across the board.

You may also like…

Q&A: Evaluating the ROI of AI implementation

Discerning reality from the hype around AI

The post How to maximize your ROI for AI in software development appeared first on SD Times.

If you can generate five or even 10 times the amount of code you previously could, that’s also five to 10  times more code that needs to be tested. 

“Many CFOs right now are looking at $30 per month per developer to go get them a GitHub Copilot or similar product,” said Jim Scheibmeir, senior director analyst at Gartner. “And I feel like we’ve kind of forgotten that frequently a bottleneck in software development is not the writing of code, but the testing of code. We’re gonna make developers so much more productive, which includes making them more productive at writing defects.”

Unlike AI-assisted dev tools where developers want to write more code, the goal with AI-assisted testing tools is to enable less testing. For instance, according to Scheibmeir, things like test impact analysis tools can create a testing strategy that is properly sized for the actual code change that is being pushed, so that only the tests that need to be run are run, rather than just running every test you have for every change. 

“These tools provide focus for testers,” he said. “And it’s so very difficult to give testers focus today. There’s this feeling like we must go test all of the things and yet we’re always crunched on time.”

Arthur Hicken, chief evangelist at Parasoft, agrees that we’ve already reached a point where test suites are taking hours, or even days, to complete, and using generative AI to help optimize test coverage can help with that.  “You can put together with AI these days a pretty good estimation of what you need to do to validate a change,” he said.

Generative AI helping with test generation, management, and more

Beyond helping testers test less, AI is creeping into other aspects of the process to make it more efficient end to end. For instance, Madhup Mishra, SVP at SmartBear, says that generative AI can now be used to create the tests themselves. “The tester can actually express their software test in simple English, and AI can actually create the automated test on their behalf,” he said. 

“Behind the scenes, GenAI should be understanding the context of the test, understanding what’s happening on the screen, and they can actually come up with a recommended test that actually solves the user’s problem without the user having to do a lot more,” he said.

Scheibmeir explained that the idea of making test generation easier had already been explored by low-code and no-code tools with their intuitive drag-and-drop interfaces, and generative AI is now taking it to that next level. 

And according to Eli Lopian, CEO of Typemock, AI is really good at exploring edge cases and may come up with scenarios that a developer might have missed. He believes that it can understand complex interactions in the codebase that the tester might not see, which can result in better coverage. 

AI can also help with generation of test data, such as usernames, addresses, PIN codes, phone numbers, etc. According to Mishra, generating test data can often be a lengthy, time-consuming process because testers have to think up all the possible variations, such as the characters that can go in a name or the country codes that come before phone numbers. 

“Generative AI can create all the different combinations of test data that you can ultimately use to be able to test all the corner cases,” Mishra explained. 

Another potential opportunity is using AI in test management. Companies often have a repository of all the different tests they have created, and AI can sort through all that and make suggestions on which to use. This allows testers to utilize what they’ve already created and free up more of their time to create new tests they need, explained Mishra. 

Parasoft’s Hicken added that AI could sort through older tests and validate if they are still going to work. For instance, if a test is capturing today’s date, then that test won’t work tomorrow. 

AI might make testing more accessible, but won’t eliminate need for it

Together, all of these AI enhancements are helping organizations take more responsibility for software quality themselves, where in the past they might have outsourced testing, Scheibmeir said. 

Similar to the citizen developer movement, the capabilities for testing that are now available make it easier for anyone to run a test, so it doesn’t require such specialized skills like it once did. 

“The hype and capabilities that generative AI are offering have brought some of these organizations back to the table of should we own more of that testing ourselves, more of that test automation ourselves,” Scheibmeir said. 

However, it’s still important to keep in mind that AI does have its drawbacks. According to Lopian, one of the biggest downsides is that AI doesn’t understand the emotion that software is supposed to give you. 

“AI is going to find it difficult to understand when you’re testing something and you want to see, is the button in the right place so that the flow is good? I don’t think that AI would be as good as humans in that kind of area,” he said.

It’s also important to remember that AI won’t replace testers, and testers will still need to keep an eye on it for now to ensure all the right coverage and the right tests are happening. Lopian likened it to a “clever intern” that you still need to keep an eye on to make sure they’re doing things correctly. 

AI’s impact on development skills will drive need for quality to shift further left

Another important consideration is the potential that if developers rely too heavily on generative AI, their development skills might atrophy, Mishra cautioned. 

“How many times have you gotten an Uber and realized the Uber driver knows nothing about where you’re going, they’re just blindly following the direction of the GPS, right? So that’s going to happen to development, and QA needs to sort of come up to speed on making sure that quality is embedded right from the design phase, all the way to how that application code will behave in production and observing it,” he said.  

Hicken agrees, likening it to how no one memorizes phone numbers anymore because our phones can store it all. 

“If I was a young person wanting to have a good long-term career, I would be careful not to lean on this crutch too much,” he said.

This isn’t to say that developers will totally forget how to do their jobs and that in 20, 30 years no one will know how to create software without the help of AI, but rather that there will emerge a new class of “casual developers,” which will be different from citizen developers.

Hicken believes this will lead to a more stratified developer community where you’ve got the “OG coders” who know how the computer works and how to talk to it, and also casual developers who know how to ask the computer questions — prompt engineers. 

“I think we are going to have to better define the people that are creating and managing our software, with roles and titles that help us understand what they’re capable of,” he said. “Because if you just say software engineer, that person needs to actually understand the computer. And if you say developer, it might be that they don’t need to understand the computer.”

You may also like…

The evolution and future of AI-driven testing: Ensuring quality and addressing bias

RAG is the next exciting advancement for LLMs

The post Generative AI development requires a different approach to testing appeared first on SD Times.

On the most recent episode of our weekly podcast, What the Dev?, we spoke with Arthur Hicken, chief evangelist at the testing company Parasoft, about all of that and whether we’ll learn from the incident. 

Here’s an edited and abridged version of that conversation:

AH: I think that is the key topic right now: lessons not learned — not that it’s been long enough for us to prove that we haven’t learned anything. But sometimes I think, “Oh, this is going to be the one or we’re going to get better, we’re going to do things better.” And then other times, I look back at statements from Dijkstra in the 70s and go, maybe we’re not gonna learn now. My favorite Dijkstra quote is “if debugging is the act of removing bugs from software, then programming is the act of putting them in.” And it’s a good, funny statement, but I think it’s also key to one of the important things that went wrong with CrowdStrike. 

We have this mentality now, and there’s a lot of different names for it — fail fast, run fast, break fast —  that certainly makes sense in a prototyping era, or in a place where nothing matters when failure happens. Obviously, it matters. Even with a video game, you can lose a ton of money, right? But you generally don’t kill people when a video game is broken because it did a bad update. 

David Rubinstein, editor-in-chief of SD Times: You talk about how we keep having these catastrophic failures, and we keep not learning from them. But aren’t they all a little different in certain ways, like you had Log4j that you thought would be the thing that oh, people are now definitely going to pay more attention now. And then we get CrowdStrike, but they’re not all the same type of problem?

AH: Yeah, that is true, I would say, Log4j was kind of insidious, partly because we didn’t recognize how many people use this thing. Logging is one of those less worried about topics. I think there is a similarity in Log4j and in CrowdStrike, and that is we have become complacent where software is built without an understanding of what the rigors are for quality, right? With Log4j, we didn’t know who built it, for what purpose, and what it was suitable for. And with CrowdStrike, perhaps they hadn’t really thought about what if your antivirus software makes your computer go belly up on you? And what if that computer is doing scheduling for hospitals or 911 services or things like that? 

And so, what we’ve seen is that safety critical systems are being impacted by software that never thought about it. And one of the things to think about is, can we learn something from how we build safety critical software or what I like to call good software? Software meant to be reliable, robust, meant to operate under bad conditions. 

I think that’s a really interesting point. Would it have hurt CrowdStrike to have built their software to better standards? And the answer is it wouldn’t. And I posit that if they were building better software, speed would not be impacted negatively and they’d spend less time testing and finding things.

DR: You’re talking about safety critical, you know, back in the day that seemed to be the purview of what they were calling embedded systems that really couldn’t fail. They were running planes and medical devices and things that really were life and death. So is it possible that maybe some of those principles could be carried over into today’s software development? Or is it that you needed to have those specific RTOSs to ensure that kind of thing?

AH: There’s certainly something to be said for a proper hardware and software stack. But even in the absence of that, you have your standard laptop with no OS of choice on it and you can still build software that is robust. I have a little slide up on my other monitor from a joint webinar with CERT a couple of years ago, and one of the studies that we used there is that 64% of vulnerabilities in NIST are programming errors. And 51% of those are what they like to call classic errors. I look at what we just saw in CrowdStrike as a classic error. A buffer overflow, reading null pointers on initialized things, integer overflows, these are what they call classic errors. 

And they obviously had an effect.  We don’t have full visibility into what went wrong, right? We get what they tell us. But it appears that there’s a buffer overflow that was caused by reading a config file, and one can argue about the effort and performance impact of protecting against buffer overflows, like paying attention to every piece of data. On the other hand, how long has that buffer overflow been sitting in that code? To me a piece of code that’s responding to an arbitrary configuration file is something you have to check. You just have to check this. 

The question that keeps me up at night, like if I was on the team at CrowdStrike, is okay, we find it, we fix it, then it’s like, where else is this exact problem? Are we going to go and look and find six other or 60 other or 600 other potential bugs sitting in the code only exposed because of an external input?

DR: How much of this comes down to technical debt, where you have these things that linger in the code that never get cleaned up, and things are just kind of built on top of them? And now we’re in an environment where if a developer is actually looking to eliminate that and not writing new code, they’re seen as not being productive. How much of that is feeding into these problems that we’re having?

AH: That’s a problem with our current common belief about what technical debt is, right? I mean the original metaphor is solid, the idea that stupid things you’re doing or things that you failed to do now will come back to haunt you in the future. But simply running some kind of static analyzer and calling every undealt with issue technical debt is not helpful. And not every tool can find buffer overflows that don’t yet exist. There are certainly static analyzers that can look for design patterns that would allow or enforce design patterns that would disallow buffer overflow. In other words, looking for the existence of a size check. And those are the kinds of things that when people are dealing with technical debt, they tend to call false positives. Good design patterns are almost always viewed as false positives by developers. 

So again, it’s that we have to change the way we think, we have to build better software. Dodge said back in, I think it was the 1920s, you can’t test quality into a product. And the mentality in the software industry is if we just test it a little more, we can somehow find the bugs. There are some things that are very difficult to protect against. Buffer overflow, integer overflow, uninitialized memory, null pointer dereferencing, these are not rocket science.

You may also like…

Lessons learned from CrowdStrike outages on releasing software updates

Software testing’s chaotic conundrum: Navigating the Three-Body Problem of speed, quality, and cost

Q&A: Solving the issue of stale feature flags

The post Q&A: Lessons NOT learned from CrowdStrike and other incidents appeared first on SD Times.

With advancements in AI, particularly in machine learning and natural language processing, testing tools have become more sophisticated. AI-driven tools can now learn from previous tests, predict potential defects, and adapt to new testing environments with minimal human intervention. Typemock has been at the forefront of this evolution, continuously innovating to incorporate AI into its testing solutions.

RELATED: Addressing AI bias in AI-driven software testing

Typemock’s AI Enhancements

Typemock has developed AI-driven tools that significantly enhance efficiency, accuracy, and test coverage. By leveraging machine learning algorithms, these tools can automatically generate test cases, optimize testing processes, and identify potential issues before they become critical problems. This not only saves time but also ensures a higher level of software quality.

I believe AI in testing is not just about automation; it’s about intelligent automation. We harness the power of AI to enhance, not replace, the expertise of unit testers. 

Difference Between Automated Testing and AI-Driven Testing

Automated testing involves tools that execute pre-written test scripts automatically without human intervention during the test execution phase. These tools are designed to perform repetitive tasks, check for expected outcomes, and report any deviations. Automated testing improves efficiency but relies on pre-written tests.

AI-driven testing, on the other hand, involves the use of AI technologies to both create and execute tests. AI can analyze code, learn from previous test cases, generate new test scenarios, and adapt to changes in the application. This approach not only automates the execution but also the creation and optimization of tests, making the process more dynamic and intelligent.

While AI has the capability to generate numerous tests, many of these can be duplicates or unnecessary. With the right tooling, AI-driven testing tools can create only the essential tests and execute only those that need to be run. The danger of indiscriminately generating and running tests lies in the potential to create many redundant tests, which can waste time and resources. Typemock’s AI tools are designed to optimize test generation, ensuring efficiency and relevance in the testing process.

While traditional automated testing tools run predefined tests, AI-driven testing tools go a step further by authoring those tests, continuously learning and adapting to provide more comprehensive and effective testing.

Addressing AI Bias in Testing

AI bias occurs when an AI system produces prejudiced results due to erroneous assumptions in the machine learning process. This can lead to unfair and inaccurate testing outcomes, which is a significant concern in software development. 

To ensure that AI-driven testing tools generate accurate and relevant tests, it is essential to utilize the right tools that can detect and mitigate bias:

• Code Coverage Analysis: Use code coverage tools to verify that AI-generated tests cover all necessary parts of the codebase. This helps identify any areas that may be under-tested or over-tested due to bias.
• Bias Detection Tools: Implement specialized tools designed to detect bias in AI models. These tools can analyze the patterns in test generation and identify any biases that could lead to the creation of incorrect tests.
• Feedback and Monitoring Systems: Establish systems that allow continuous monitoring and feedback on the AI’s performance in generating tests. This helps in early detection of any biased behavior.

Ensuring that the tests generated by AI are effective and accurate is crucial. Here are methods to validate the AI-generated tests:

• Test Validation Frameworks: Use frameworks that can automatically validate the AI-generated tests against known correct outcomes. These frameworks help ensure that the tests are not only syntactically correct but also logically valid.
• Error Injection Testing: Introduce controlled errors into the system and verify that the AI-generated tests can detect these errors. This helps ensure the robustness and accuracy of the tests.
• Manual Spot Checks: Conduct random spot checks on a subset of the AI-generated tests to manually verify their accuracy and relevance. This helps catch any potential issues that automated tools might miss.

How Can Humans Review Thousands of Tests They Didn’t Write?

Reviewing a large number of AI-generated tests can be daunting for human testers, making it feel similar to working with legacy code. Here are strategies to manage this process:

• Clustering and Prioritization: Use AI tools to cluster similar tests together and prioritize them based on risk or importance. This helps testers focus on the most critical tests first, making the review process more manageable.
• Automated Review Tools: Leverage automated review tools that can scan AI-generated tests for common errors or anomalies. These tools can flag potential issues for human review, reducing the workload on testers.
• Collaborative Review Platforms: Implement collaborative platforms where multiple testers can work together to review and validate AI-generated tests. This distributed approach can make the task more manageable and ensure thorough coverage.
• Interactive Dashboards: Use interactive dashboards that provide insights and summaries of the AI-generated tests. These dashboards can highlight areas that require attention and allow testers to quickly navigate through the tests.

By employing these tools and strategies, your team can ensure that AI-driven test generation remains accurate and relevant, while also making the review process manageable for human testers. This approach helps maintain high standards of quality and efficiency in the testing process.

Ensuring Quality in AI-Driven Tests

Some best practices for high-quality AI testing include:

• Use Advanced Tools: Leverage tools like code coverage analysis and AI to identify and eliminate duplicate or unnecessary tests. This helps create a more efficient and effective testing process.
• Human-AI Collaboration: Foster an environment where human testers and AI tools work together, leveraging each other’s strengths.
• Robust Security Measures: Implement strict security protocols to protect sensitive data, especially when using AI tools.
• Bias Monitoring and Mitigation: Regularly check for and address any biases in AI outputs to ensure fair testing results.

The key to high-quality AI-driven testing is not just in the technology, but in how we integrate it with human expertise and ethical practices.

The technology behind AI-driven testing is designed to shorten the time from idea to reality. This rapid development cycle allows for quicker innovation and deployment of software solutions.

The future will see self-healing tests and self-healing code. Self-healing tests can automatically detect and correct issues in test scripts, ensuring continuous and uninterrupted testing. Similarly, self-healing code can identify and fix bugs in real-time, reducing downtime and improving software reliability.

Increasing Complexity of Software

As we manage to simplify the process of creating code, it paradoxically leads to the development of more complex software. This increasing complexity requires new paradigms and tools, as current ones will not be sufficient. For example, the algorithms used in new software, particularly AI algorithms, might not be fully understood even by their developers. This will necessitate innovative approaches to testing and fixing software.

This growing complexity will necessitate the development of new tools and methodologies to test and understand AI-driven applications. Ensuring these complex systems run as expected will be a significant focus of future testing innovations.

To address security and privacy concerns, future AI testing tools will increasingly run locally rather than relying on cloud-based solutions. This approach ensures that sensitive data and proprietary code remain secure and within the control of the organization, while still leveraging the powerful capabilities of AI.

You may also like…

Software testing’s chaotic conundrum: Navigating the Three-Body Problem of speed, quality, and cost

Report: How mobile testing strategies are embracing AI

The post The evolution and future of AI-driven testing: Ensuring quality and addressing bias appeared first on SD Times.

The cause? An update that was pushed by CrowdStrike to Windows machines that triggered a logic error causing the device to get the Blue Screen of Death (BSOD). Even though CrowdStrike pulled the update fairly quickly, the computers had to be updated individually by IT teams, leading to a lengthy recovery process.

While we don’t know what specifically CrowdStrike’s testing process looked like, there are a number of basic steps that companies releasing software should be doing, explained Dr. Justin Cappos, professor of computer science and engineering at NYU. “I’m not gonna say they didn’t do any testing, because I don’t know … Fundamentally, while we have to wait for a little more detail to see what controls existed and why they weren’t effective, it’s clear that somehow they had massive problems here,” said Cappos.  

He says that one thing companies should be doing is rolling out major updates gradually. Paul Davis, field CISO at JFrog, agrees, noting that whenever he’s led security for companies, any major updates to the software would have been deployed slowly and the impact would be carefully monitored. 

He said that issues were first reported in Australia, and in his past experiences, they would keep a particularly close eye on users in that country after an update because Australia’s workday starts so much earlier than the rest of the world. If there was a problem there, the rollout would be immediately stopped before it had the chance to impact other countries later on. 

“In CrowdStrike’s situation, they would have been able to reduce the impact if they had time to block the distribution of the errant file if they had seen it earlier, but until we see the timeline, we can only guess,” he said. 

Cappos said that all software development teams also need a way to roll back systems to a previously good state when issues are discovered. 

“And whether that’s something that every vendor should have to figure out for themselves or Microsoft should provide a common good platform, we can maybe debate that, but it’s clear there was a huge failure here,” he said. 

Claire Vo, chief product officer at LaunchDarkly, agrees, adding: “Your ability to contain, identify, and remediate software issues is what makes the difference between a minor mishap and a major, brand-impacting event.” She believes that software bugs are inevitable and everyone should be operating under the assumption that they could happen.

She recommends software development teams decouple deployments from releases, do progressive rolluts, use flags that can power runtime fixes, and automate monitoring so that your team can “contain the blast radius of any issues.” 

Marcus Merrell, principal test strategist at Sauce Labs, also believes that companies need to assess the potential risk of any software release they’re planning. 

“The equation is simple: what is the risk of not shipping a code versus the risk of shutting down the world,” he said. “The vulnerabilities fixed in this update were pretty minor by comparison to ‘planes don’t work anymore’, and will likely have the knock-on effect of people not trusting auto-updates or security firms full stop, at least for a while.”

Despite what went wrong last week, Cappos says this isn’t a reason to not regularly update software, as software updates are crucial to keeping systems secure. 

“Software updates themselves are essential,” he said. “This is not a cautionary tale against software updates … Do take this as a cautionary tale about vendors needing to do better software supply chain QA. There are tons of things out there, many are free and open source, many are used widely within industry. This is not a problem that no one knows how to solve. This is just an issue where an organization has taken inadequate steps to handle this and brought a lot of attention to a really important issue that I hope gets fixed in a good way.”

You may also like…

Software testing’s chaotic conundrum: Navigating the Three-Body Problem of speed, quality, and cost

The secret to better products? Let engineers drive vision

The post Lessons learned from CrowdStrike outages on releasing software updates appeared first on SD Times.

Qodana is a static code analysis tool that integrates into JetBrains’ IDEs, allowing issues to be addressed directly within the IDE.

“Since launching the cloud version of Qodana last summer, we kept receiving requests for a self-hosted version. Following successful Beta tests with some of our clients, we’re now launching the first release of Qodana Self-Hosted, allowing you to manage, maintain, and upgrade Qodana entirely on your end,” JetBrains wrote in a blog post. 

Qodana Self-Hosted helps companies ensure that their proprietary or sensitive information stays within the company’s infrastructure, offers them greater control over data control and processing, and can have better performance due to the ability to optimize the solution to the organization’s specific performance needs. 

The self-hosted version currently only supports AWS, but JetBrains plans to add more hosting options in future versions. Pricing will start at $40 per developer per month, but organizations that request a demo by August 31, 2024 will receive a 40% discount off one year of Qodana Self-Hosted. 

Other recent improvements to Qodana include an early access program for the C and C++ linter, the ability to create custom inspections with FlexInspect, support for Unity analysis, and the addition of .NET analysis to the Community version.

You may also like…

JetBrains’ code quality platform Qodana reaches general availability

The importance of prevention: How shifting left, static analysis and unit testing create better code quality

The post JetBrains now allows Qodana to be self-hosted appeared first on SD Times.

Jtest now includes test templates in Unit Test Assistant, which is a feature that uses AI to generate a suite of tests. With the new Jtest release, testers get more control over the structure of their test classes and can specify common configurations that their tests require.

Jtest can also now run test impact analysis right from within the IDE. Whenever a code change is made, Jtest will identify and execute tests and provide feedback to the developer on the impact of their modifications.

“With the new Jtest release, developers get real-time insights into which tests are impacted by their code changes,” Igor Kirilenko, chief product officer at Parasoft, told SD Times. “While you are still modifying your code, Jtest automatically runs the relevant tests and delivers instant feedback. This groundbreaking feature not only saves time but also ensures that potential bugs are caught and fixed before they ever reach the build pipeline.”

In Jtest and dotTEST, an integration with OpenAI/Azure OpenAI Service provides AI-generated fixes for flow analysis violations. 

Jtest and dotTEST also now support the latest version of the Common Weakness Enumeration (CWE) list, 4.14. Additionally, both have improved out-of-the-box static analysis test configurations.

And finally, DTP’s integration with OpenAI/Azure OpenAI Service speeds up remediation of security vulnerabilities by matching security rule violations to known vulnerabilities, and then coming up with a probability score of the likelihood of each being a real vulnerability or a false positive. 

“Developers often face significant cognitive load when triaging static analysis violations, particularly those related to security,” Jeehong Min, technical product manager at Parasoft, told SD Times. “Each security rule comes with its own learning curve, requiring time to understand its nuances. To assist developers, Parasoft DTP offers recommendations powered by pre-trained machine learning models and models that learn from the development team’s triage behavior. The ultimate goal is to help developers make informed decisions when triaging and remediating static analysis violations.”

You may also like…

The human side of automation: Reclaiming work-life balance

Reducing the total cost of ownership of software testing

The post Parasoft’s latest release offers several new automated features for testing Java, C#, .NET apps appeared first on SD Times.

The new offering enables developers to test sensor perception and other related AI software in a realistic test environment before they deploy their solutions in the real world. 

It can be used to simulate different activities, like whether a robotic arm is operating correctly, an airport luggage carousel is running, a tree branch is blocking the road, a factory conveyor belt is in motion, or a person is nearby. 

“Developing safe and reliable autonomous machines powered by generative physical AI requires training and testing in physically based virtual worlds,” said Rev Lebaredian, vice president of Omniverse and simulation technology at NVIDIA. “NVIDIA Omniverse Cloud Sensor RTX microservices will enable developers to easily build large-scale digital twins of factories, cities and even Earth — helping accelerate the next wave of AI.”

The new offering is built on the OpenUSD framework and it utilizes NVIDIA RTX ray tracing and neural-rendering technologies. 

Early access to Omniverse Cloud Sensor RTX will be available later this year, and developers can sign up here. NVIDIA has given access to only two companies so far: Foretellix and MathWorks. 

You may also like…

• NVIDIA GTC kicks off with announcement of new Blackwell GPUs for AI uses
• Four questions to answer before scaling a product
• Safe AI development: Integrating explainability and monitoring from the start

The post NVIDIA Omniverse Cloud Sensor RTX simulates physical sensors so that developers can test autonomous machines appeared first on SD Times.