Using Test Studio Ultimate, I am testing an application that sends a variety of emails that require user interaction with the Email contents (clicking links, for example). Unfortunately Gmail is an absolute mess in the DOM, and trying to find anything by ID/Class is virtually useless. At this point, the best solution I have come to is using XPath to locate elements.
When running through a bunch of iterations of my test (user setup, email notification + interaction, back to web app) the Gmail part will fail about 1/10 times on the XPath lookup. The main issue seems to be with how Test Studio reads and converts the HTML to XML, then parses it to try and find the specified XPath query. In the test failure log, I don't get a full log dump. It seems to be several hundred lines of logs from existing tests, then it dumps a HUGE (~2000-8000 lines) XML content from the Gmail page into the log, and then it fails out each level of testing.
The 2 primary functions I'm trying to handle are:
- Get email contents based on the subject
- Click the link generated, which is stored in an email message
First off, is there a way to work around the error in Test Studio? Most of the time it works properly, so I'm not sure how to troubleshoot the intermittent failure.
Secondly, is there a better way than XPath to try and locate page elements in Gmail? (Eventually the dev team will implement other ways of handling the email contents, but that will be months down the road.)
I don't have logs at the moment; I will save them the next time I see a failure.
7 Answers, 1 is accepted
Thank you for the detailed description.
I have thoroughly read the information you have shared and since the tests fail intermittently this might be a timing issue. Have you included enough wait steps and execution delays to build a stable test for smooth execution?
You could also double check if there isn't anything dynamic and different in the failure cases - a banner or an advertisement which does (not) appear when the test fails.
Also I would like to know if it is mandatory to use Gmail as a web mail client. As far as I understand it is important to validate the emails sent from the application under test. So you could use any other web mail account which is more stable in its behavior.
Please collect the failure details of a failed test and send these over the current thread for further investigation. Alternatively the application log generated during a failing test could also be useful.
Thanks in advance for the cooperation!
Regards,
Elena Tsvetkova
Progress Telerik
Rank 1
Thanks for the response. I will make sure to get the logs next time it fails, as I got pulled away to another task and haven't revisited it yet.
I am fairly certain it's not a timing issue, as I have an element wait, a forcible 5+ seconds wait, and another element wait after the fact. Also, the place where it most commonly fails is while the page is fully loaded and I Have looked up 2 other elements by simpler methods (non-XPath).
Gmail is not mandatory, but it does simplify a lot of other procedures as we are a Google Apps (G Suite) shop, so we have a lot of flexibility with how we manage our business email system for testing. I am talking to the team about an alternative, and there will likely be a custom test-email recipient service created down the road (several months). For the meantime, the only alternative I can see would be to pass emails through a client like Thunderbird, but I have not tested anything like that before.
Rank 1
I have an error log now, but it is quite confusing. The element it fails at, "GmailMessagesSpan", is a standard span near the top of the page. I'm using the following logic in Find Element:
- tagname (is exactly) "a"
- title (contains) "Google Account: <FilteredOut>"
- [child find] > tagname (is exactly) "span"
This doesn't involve any XPath, neither does the step after nor the step before. Step 42 is a Find Element lookup in Test Studio, but the failure information below shows a failure while waiting because "No FindParams defined". When it gets to "Using FindParam:" I see my custom XPath logic is there from Step 15 (which passed). How is Step 42 (a wait step using Test Studio "Find Element" logic) getting confused by Step 15, a custom coded step where I use XPath to locate an item on the page?
First is the CodedStep 15, after that is the full error text. I can give more information if needed, but I don't think the problem is my XPath logic:
Step 15 ClickSubscriptionEmail()
[CodedStep(@"Call ClickEmailBySubject() Using Subscription Text")]public void ClickSubscriptionEmail(){ try { subName = GetExtractedValue("MainSubName").ToString(); } catch (Exception ex) { Log.WriteLine(ex.ToString()); } subName = GetExtractedValue("MainSubName").ToString(); string emailSubject = "Please sign [DEMO USE ONLY] Subscription Agreement " + subName; ClickEmailBySubject(emailSubject, true);}
Failure Log:
'8/28/2017 12:40:44 PM' - 'Fail' : 42. Wait for element 'GmailMessagesSpan' 'is' visible.------------------------------------------------------------Failure Information: ~~~~~~~~~~~~~~~Unexpected error while waiting on condition. Error: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Exception: Refresh() failed due to an unexpected exception. ---> ArtOfTest.WebAii.Exceptions.FindException: Find Details:- No FindParams defined.- Start Element: [Element: 'html:0']- Search log during Find:Searching for element starting at '[Element: 'html:0']' Using FindParam: [Find logic: Use 'XPath' where xpath is '//span/b[text()='Please sign [DEMO USE ONLY] CLIENT NAME HERE']/parent::*/parent::*/preceding-sibling::*/div/div[@title='Subscription']' and (none) ]Valid Xml used for XPath identification:----------------------------------------
----------------------------------------
<?xml version="1.0"?>
<html __absIndex="0" class="aAX" lang="en">
<!-- 5000+ lines of HTML/XPath are contained here, along with the majority of Gmail's "Style" tags, so I filtered them out. I can attach it separately if needed, but the XPath and HTML are all valid and the element exists. -->
</html>
----------------------------------------
Element found is: '[Element: 'div:764']'
Attributes validation PASSED!
Element identification and validation succeeded!
at ArtOfTest.WebAii.ObjectModel.Element.Refresh()
--- End of inner exception stack trace ---
at ArtOfTest.WebAii.ObjectModel.Element.Refresh()
at ArtOfTest.Common.Design.Extensibility.Descriptors.VerificationDescriptor.EnsureTargetsAreCurrent()
at ArtOfTest.WebAii.Design.IntrinsicTranslators.Descriptors.VisibleVerificationDescriptor.ExtractData(IAutomationHost targetBrowser, DescriptorValueStore dataStore)
at ArtOfTest.Common.Design.Extensibility.Descriptors.VerificationDescriptor.<PerformWait>b__30_0(IAutomationHost hst, DescriptorValueStore values)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Delegate.DynamicInvokeImpl(Object[] args)
at ArtOfTest.Common.WaitAsync._worker_DoWork[T,V](Object waitParam)
------------------------------------------------------------
'8/28/2017 12:40:44 PM' - Detected a failure. Step is marked 'ContinueOnFailure=False' aborting test execution.
------------------------------------------------------------
'8/28/2017 12:40:44 PM' - Overall Result: Fail
<?xml version="1.0"?><html __absIndex="0" class="aAX" lang="en">
Rank 1
I had a thought. During this test I URL-navigate to the Gmail inbox page so it loads fresh, rather than using click/XPath navigation. This happens right before the "wait" element the fails, so I am going to put in an Execution Delay and an ActiveBrowser.RefreshDomTree() right before this failing step and see what happens.
I'm trying this because of the failure line "Refresh() failed due to an unexpected exception."
If this fixes it, I don't know why I would need to manually do these extra steps just for loading Gmail again.
Thank you for the details.
It seems that the find expressions for the elements are mixed somehow. Would it be possible to provide us a sample test to inspect? If we could also execute it successfully this will be really useful.
Please let me know your opinion on the above topic. Thanks!
Regards,
Elena Tsvetkova
Progress Telerik
Rank 1
Hi Elena,
Sorry for the delay. I was working on another issue.
I have refactored the Gmail tests to use Basic HTML mode (a lot less complex). The issue was still occurring, even with every coded step and XPath lookup removed. This time I think I found the problem.
First I load Gmail (logged in already), and then I Verify/Wait for an element. Those steps right after the navigation were the ones occasionally failing, so I put an Execution Delay step immediately after the navigation. Before, I got failures about 15% of the time. Now, I run the same stripped down test 20 times in a row (Test As Step duplicated over and over) and I see no failures at all.
Hopefully that solves it, but I'm wondering what the scenario was that causes it in the first place.The error I was seeing in the stripped down test (no coded steps, no XPath) was:
InnerException:
System.Exception: Refresh() failed due to an unexpected exception. ---> ArtOfTest.WebAii.Exceptions.FindException: Find Details:
- No FindParams defined.
- Start Element: [Element: 'html:0']
- Search log during Find:
Finding all elements using:
[Find logic: Use 'AttributesOnly' where (class=onyx-spinner) ]
// ^--- THIS CLASS IS FROM OUR WEB APP, NOT LOCATED ANYWHERE IN THE GMAIL TEST
Found '3' elements that match.
at ArtOfTest.WebAii.ObjectModel.Element.Refresh()
--- End of inner exception stack trace ---
at ArtOfTest.WebAii.ObjectModel.Element.Refresh()
at ArtOfTest.Common.Design.Extensibility.Descriptors.VerificationDescriptor.EnsureTargetsAreCurrent()
at ArtOfTest.WebAii.Design.IntrinsicTranslators.Descriptors.VisibleVerificationDescriptor.ExtractData(IAutomationHost targetBrowser, DescriptorValueStore dataStore)
at ArtOfTest.WebAii.Design.Execution.ExecutionEngine.ExecuteStep(Int32 order)
Thank you for getting back to me. I am pleased to know that inserting an execution delay has resolved the issue.
Probably the execution was faster than expected and the DOM tree was not fully loaded to face the reported misbehavior. Though to drill down what the exact cause for that is would require research in depth on our end and there is no estimation I could provide.
Thank you for your understanding and do not hesitate to contact us again of any difficulties faced.
Regards,
Elena Tsvetkova
Progress Telerik