Hmmm... My opinion on this mixed. Competition is a good thing, but at the same I can see lots of opportunity for collaboration. With so many testing tools around, it would be fun to do some kind of "Testing Tools Summit" to share ideas and compare notes. Maybe host it in conjuction with a "Python Web Frameworks Summit". :-)
Do you think Selenium is just too complicated to get going or do you think it's just lacking documentation? What are your top 3 three things you'd like to see (or see fixed or see changed) in Selenium, that weren't there?
-Jason "The guy who wrote Selenium" Huggins.
I don't have any real problem with Selenium. In part I did this because it amused me, but also the ability to do automated/non-browser evaluation of the script is important. Like I said, it wouldn't be that hard to write a non-browser Selenium interpreter (maybe that even exists), but I just wrote it in this direction.
Well, as long as I'm on the topic, this is what I like less about Selenium:
- There's a lot of up-front complexity that isn't very important, and little things (like how do you start your own test suite) are not as blindingly obvious as they should be.
- Last time I looked at it -- and this was in discussion, so it may have been resolved -- there wasn't many ways to create higher level abstractions of your scripts, to keep the line count down and cut out repetetive parts. (Note that the featureset provided by my code is, if anything, worse on that regard)
- The names are really long.
- The table is a bit too constrained. Everything must take 0-2 arguments.
- Interactive development seems like it would be hard -- I am not likely to type Selenium commands out easily. Interactive creation of Twill in the browser seems quite reasonable (not implemented, of course).
But all that said, none of those were the reasons I wrote this, nor are they the kind of substantive reasons I would or wouldn't choose Selenium.
I think both Twill and Selenium are little too terse in its syntax. Instead of this (twill syntax):# Not too exciting... go /examples/twill_js/test.html fv 0 age 10 fv 0 name Ian echo Get ready for the submit! submit 0 notfind NOTHING follow back code 200
I'd rather have a HyperTalk-inspired syntax like the following:-- Not too exciting go to /examples/twill_js/test.html type 10 into field with name "age" type "Ian" into field with name "name" print "Get ready for the submit" click "submit-me" assert page is not null go back assert response_code is 200
With that tangent aside, I'm thinking that the logical next steps for Selenium are:
- To make the HTML table-tag aspect of Selenium tests optional. Split on whitespace instead, like Twill.
- More fully embracing browser extensions like Selenium Recorder... Browser extensions let Selenium run "outside" of the security sandbox Selenium usually lives in. And being "outside" that box would let users do interesting things like testing or automating any site on the net (not just the the local host where Selenium was installed.) The key words for this are "cross site scripting" and "same origin policy"... and it trips up lots of new users of Selenium.
Anyway, 'tis interesting times to be a web developer and software tester. Lots of cool stuff is getting created these days in this space.
thanks for your positive comments on twill's syntax: I make it up as I go, and (thus far) have mostly been just happy when something works. I do try to keep the syntax simple as opposed to easy, because one of the major use cases for twill is its use by non-programmers; what programmers find easy is often not so simple. Note also that the twill syntax is very closely linked to Python code; for example,:go here
translates directly to:go('here')
This is a very convenient feature for Python programmers and several users seem to be using it this way.
Also, let me say: twill aims low. I want a simple, solid way to script and test Web sites in a fairly black-box-ish manner. It honestly shouldn't be too much work to get there, and I bet the future will be in recorders and/or test automation technology rather than twill-ish languages. After all, who cares what language the test uses, if the computer can write it for you?
Finally -- Grig Gheorghiu and I have proposed an Agile Development tutorial for PyCon. As you no doubt know, Grig is a huge proponent of Selenium. We will certainly feature both twill and Selenium in our tutorial. No doubt our febrile imaginations will come up with plans for some orcish melange of twill and Selenium, and I have a hard time resisting a good hack...
p.s. "--" for comments is ungodly! long live '#'!
p.p.s. just kidding ;).
I bet the future will be in recorders and/or test automation technology rather than twill-ish languages. After all, who cares what language the test uses, if the computer can write it for you?
I disagree, I think recordings can never understand what matters and what doesn't, and the result is too fragile. And once you've done it, if it's not expressed as some at least vaguely sensible language, then it is uneditable. That language might be in markup, or might not, but I think it really has to be a "language", not just a literal recording. Recorders are nice, still, they just aren't complete by themselves.
"Recorders are nice, still, they just aren't complete by themselves."
I agree. Test recorders are kind of like of Ruby on Rails's generators and scaffolding... They both get knocked alot, however, they are very useful when you consider them "the starting point" and not the final production-ready result. Test recorders lower the barrier to entry into the testing framework's language or API--- that, and the kids love 'em. :-) My past experience with macro or test recording is with SQA Robot by Rational (now IBM) and with Microsoft Excel's built in macro utility. In both cases, I used recording to generate a simple, single working script... then went on to edit that raw test into a far more robust and "programmed, not recorded" test suite. The recorders gave me working code from step one, which I then manipulated as needed.
Well, maybe it's a matter of degree. At some point I doubt there will be a need for language innovation -- but recorders can get much better...
We'll see ;).
Quoting Benji York from gmane.comp.web.zope.devel:
The rendering framework is pluggable and it's pretty easy to add new rendering options for other testing systems (twill, etc.).
The testrecorder is currently in "beta" form, but is very handy, so if anyone wants to use/market/improve/add tests for it, go ahead. :) Get it via Subversion at svn://svn.zope.org/repos/main/zope.testrecorder/trunk
It is very cool.