Discover key insights to accelerate software delivery | State of Software Quality Report 2024

GET YOUR REPORT

Frequently Asked Questions

Will my data be used in training Katalon's generative AI models that can potentially cause data leakage for sensitive data?

+

No, Katalon leverages OpenAI's GPT models to generate test scripts (StudioAssist) and manual test cases (Jira plugin). According to OpenAI's API data usage policies, "OpenAI will not use data submitted by customers via your API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data." Katalon does not opt-in to share data to train GPT models, ensuring that your data remains confidential and secure.

How the data is governed?

+

Data governance in Katalon StudioAssist ensures that no user data is used to train AI models. The data is processed with strict adherence to data protection regulations like GDPR and CCPA. Katalon does not store, transfer, or process sensitive data​​​​.

Where is the data / input is stored?

+

Data input by users in StudioAssist is processed through OpenAI's GPT models. This data is transmitted securely to OpenAI for processing. Katalon does not retain any user data used for generating or explaining code through StudioAssist​​​​.

What certification do we hold with OpenAI?

+

Katalon adheres to OpenAI's API data usage policies, which include not using customer data to train or improve their models unless explicitly opted-in. While specific certifications with OpenAI are not mentioned, compliance with these policies ensures data security and privacy​​​​.

Does Katalon guarantee the accuracy of the generated outcomes, eg. test scripts, test cases, data analytic insights?

+

Katalon leverages generative AI models to generate code and tests. Even the best generative AI models such as OpenAI's GPT (behind ChatGPT) have inherent limitations. These limitations arise from several factors:

  • Inaccuracy in Complex Scenarios: Generative models may sometimes produce code that looks syntactically correct but may not function as intended, especially in complex scenarios requiring deep domain knowledge.
  • Lack of Context Awareness: The AI model might not fully understand the specific context of your project, leading to potential misalignment with project requirements or coding standards.
  • Knowledge Cut-off Date: The information embedded in the model is frozen at a certain point in time, which means that the generated code may not include the latest techniques, methodologies, or keywords.
  • Potential Bias or Inconsistency: Models like ChatGPT are trained on diverse datasets and may inadvertently introduce biases or inconsistencies that are not suitable for your particular application.

For these reasons, it is highly advisable not to rely solely on Katalon's generated code and tests. While the tool can significantly accelerate development and testing processes, human oversight is essential. It is strongly recommended to review and revise the generated results carefully before integrating them into your project to ensure they meet the specific needs and quality standards.

Katalon is committed to enhancing the capabilities of its AI-driven features, but the collaboration between human expertise and AI remains vital for achieving the best possible outcomes.

Given that GPT's knowledge cut-off was September 2021, will my generated code include the latest Katalon keywords?

+

In the current version of StudioAssist, the newly released keywords after GPT's knowledge cut-off will not be included in generated code. This is a known limitation. However, we are committed to improving this aspect, and the ability to include new keywords will be part of future releases.

Does Katalon StudioAssist generate code based on my current coding style?

+

Currently, StudioAssist does not have the ability to leverage your existing coding style for generating new code. It adheres to a standardized coding style. We recognize the importance of personalized code generation and are actively working on incorporating this feature in later releases.

Does Katalon StudioAssist generate code using my test objects and variables?

+

At present, StudioAssist is not capable of automatically recognizing the test object library, existing test cases, variables, or test data. You need to specify what should be used in the comments for code generation. We understand the importance of this functionality and are planning enhancements in future releases to make the process more intuitive and aligned with your specific requirements.

Click