Title: Watir: Retrieve all dynamic HTML elements that match an attribute?
Tags: ruby;web-scraping;watir
Question: I am trying to scrape dynamic content with Watir and I am stuck.
Basically, I know that I can use
browser.element(css: ".some_class").wait_until_present
in order to scrape only when "some_class" is loaded.
The problem is that it is only giving me the first element having this class name and I want all of them.
I also know I can use
browser.spans(css: ".some_class")
in order to collect ALL the classes having this name, the problem is that I can't combine it with "wait_until_present" (it gives me an error). And spans on his own is not working because the content is not loaded yet, the page is using javascript
Is there a way to combine both? That means waiting for the class_name to be loaded AND select all the elements matching this class name, not just the first one?
I've been stuck for ages...
Thanks a lot for your help
Comment: Can you please try this problem through debug mode? Stop once after clicking the button which leads here and leave some.time gap and continue to execute the rest of the statement and let's sree whether it returns all the value.


Here is another answer: There currently isn't anything in Watir for waiting for a collection of elements (though I had been recently thinking about adding something). For now, you just have to manually wait for an element to appears and then get the collection.
The simplest one is to call both of your lines:
browser.element(css: ".some_class").wait_until_present
browser.spans(css: ".some_class")
If you wanted to one-liner it, you could use #tap:
browser.spans(css: ".some_class").tap { |c| c[0].wait_until_present }
#=&gt; Watir75b5:f2c1:f5d7:b954:8dc3:cee9:baf:a8c0SpanCollection
Note that if you are just checking the class name, you might want to avoid writing the CSS-selector. Not only is it easier to read without it, it won't be as performant.
browser.spans(class: "some_class").tap { |c| c[0].wait_until_present }
Comment for this answer: @Rajagopalan, nothing is impossible with the right perspective. Watir could know how many elements should exist - if the user tells it. Similar to Capybara, Watir could implement something like `browser.spans(minimum: 3)`, which would wait until there are at least 3 spans.
Comment for this answer: @Rajagopalan, while there some situations where a user might not know how many elements to expect, there are plenty of times they will. For example, I have a page that loads a table asynchronously. The table is there or it is not - ie I would want to wait until there is at least 1 row.
Comment for this answer: @Jessicascn, any chance you can create or share a page that reproduces the issue?
Comment for this answer: @JustinKo I think this would be as simple as aliasing `#any?`to `#present?` and then including `Waitable`, @Jessicascn can you raise an issue on Github with this feature request? https://github.com/watir/watir/issues/new. Also, Collections are now lazy loaded which means nothing is located until you try to use it. Just doing `my_collection.last` won't attempt to locate the element. `my_collection.last.exists?` will locate the element (or click or attribute_value, or send_keys, etc)
Comment for this answer: We can never wait for collection because how many elements in a collection will be determined after loading. So that's impossible.
Comment for this answer: User can't know know how many elements even before it loads , that would be meaningless.
Comment for this answer: How does page load involves Java script ? Can you explain me a bit? Web driver will not leave the control to next statement in program before it completely loads the page
Comment for this answer: Okay, I can envision that now
Comment for this answer: Thanks a lot for your answer! Unfortunately I've tried that but it doesn't work. The spans gives me a collection, but when I iterate on it it gives me nothing back. I've tried to select the last guy of the collection and it is saying "located : false", which makes me think it didn't wait the page to be loaded in order to use the spans.

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 1, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2018:00:06/21/18', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<ruby><web-scraping><watir>', 'Title': 'Watir: Retrieve all dynamic HTML elements that match an attribute?', 'ViewCount': 298, 'id': 50973338, 'user_AboutMe': None, 'user_AccountId': 11758079.0, 'user_DisplayName': 'Jessicascn', 'user_Reputation': 141.0, 'user_Views': 48.0, 'user_id': 8605123.0}}