Last modified: 2014-09-25 15:02:38 UTC
Rspec expectation in this piece of code (from font_selection_steps.rb[1]) fails because of bug #56081. Then(/^the active content font must be the same as font prior to the preview$/) do pending('bug #56081') do on(PanelPage).get_content_font.should == @original_content_font end end We have marked the expectation as pending, so the scenario (and Jenkins job that runs it) do not fail because of a known bug. When the bug is fixed, and the expectation no longer fails, it would be great if the scenario then failed, so we would be alerted that the scenario no longer needs to be pending. 1: https://github.com/wikimedia/mediawiki-extensions-UniversalLanguageSelector/blob/master/tests/browser/features/step_definitions/font_selection_steps.rb
The documentation says I am expecting the correct behavior: https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/pending.rb?source=cc#L50-L57
Cucumber output if the step is not marked pending: $ bundle exec cucumber features/font_selection.feature:23 Using the default profile... ............F- (::) failed steps (::) expected: "sans-serif" got: "OpenDyslexic,sans-serif" (using ==) (RSpec::Expectations::ExpectationNotMetError) ./features/step_definitions/font_selection_steps.rb:15:in `/^the active content font must be the same as font prior to the preview$/' features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview' Failing Scenarios: cucumber features/font_selection.feature:23 # Scenario: Discarding live preview of content font 1 scenario (1 failed) 9 steps (1 failed, 1 skipped, 7 passed) 1m12.158s
If the step is marked as pending, the default Cucumber formatter (progress) does not give a lot of information: $ bundle exec cucumber features/font_selection.feature:23 Using the default profile... ............P- (::) pending steps (::) features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview' 1 scenario (1 pending) 9 steps (1 skipped, 1 pending, 7 passed) 0m55.161s
But formatter called "pretty" gives more information (... 'bug #56081 (Cucumber::Pending)' ...): $ bundle exec cucumber -f pretty features/font_selection.feature:23 Using the default profile... @commons.wikimedia.beta.wmflabs.org @login @reset-preferences-after Feature: Font selection In order to have better using experience, As a reader and writer, I want to change or disable the fonts for interface and content. In addition the user is provided live preview feature: changes are applied immediately when selection is made. Changes can either be applied or discarded for easy testing. Background: # features/font_selection.feature:12 Given I am logged in # features/step_definitions/common_steps.rb:8 And I set "German" as the interface language # features/step_definitions/common_steps.rb:12 And I open ULS # features/step_definitions/font_selection_steps.rb:1 And I open display settings # features/step_definitions/font_selection_steps.rb:5 When I open fonts panel of language settings # features/step_definitions/font_selection_steps.rb:9 Scenario: Discarding live preview of content font # features/font_selection.feature:23 When I select "OpenDyslexic" font for the content language for the live preview # features/step_definitions/panel_steps.rb:38 And I close the panel to discard the changes # features/step_definitions/panel_steps.rb:43 Then the active content font must be the same as font prior to the preview # features/step_definitions/font_selection_steps.rb:13 bug #56081 (Cucumber::Pending) ./features/step_definitions/font_selection_steps.rb:14:in `/^the active content font must be the same as font prior to the preview$/' features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview' And the selected content font must be "system" # features/step_definitions/font_selection_steps.rb:19 1 scenario (1 pending) 9 steps (1 skipped, 1 pending, 7 passed) 1m3.638s
But if change the expectation to the example from rspec documentation (so the expectation inside the pending block no longer fails: pending('bug #56081') do true.should be(true) end I was expecting the scenario to fail, but it did not fail: $ bundle exec cucumber -f pretty features/font_selection.feature:23 Using the default profile... @commons.wikimedia.beta.wmflabs.org @login @reset-preferences-after Feature: Font selection In order to have better using experience, As a reader and writer, I want to change or disable the fonts for interface and content. In addition the user is provided live preview feature: changes are applied immediately when selection is made. Changes can either be applied or discarded for easy testing. Background: # features/font_selection.feature:12 Given I am logged in # features/step_definitions/common_steps.rb:8 And I set "German" as the interface language # features/step_definitions/common_steps.rb:12 And I open ULS # features/step_definitions/font_selection_steps.rb:1 And I open display settings # features/step_definitions/font_selection_steps.rb:5 When I open fonts panel of language settings # features/step_definitions/font_selection_steps.rb:9 Scenario: Discarding live preview of content font # features/font_selection.feature:23 When I select "OpenDyslexic" font for the content language for the live preview # features/step_definitions/panel_steps.rb:38 And I close the panel to discard the changes # features/step_definitions/panel_steps.rb:43 Then the active content font must be the same as font prior to the preview # features/step_definitions/font_selection_steps.rb:13 Expected pending 'bug #56081' to fail. No Error was raised. No longer pending? (Cucumber::Pending) ./features/step_definitions/font_selection_steps.rb:14:in `/^the active content font must be the same as font prior to the preview$/' features/font_selection.feature:26:in `Then the active content font must be the same as font prior to the preview' And the selected content font must be "system" # features/step_definitions/font_selection_steps.rb:19 1 scenario (1 pending) 9 steps (1 skipped, 1 pending, 7 passed) 0m59.764s
I think the problem is that we are using Cucumber::Pending instead of RSpec::Pending, and looks like Cucumber does not behave like RSpec.
I am not sure how to call RSpec#pending from Cucumber, I have tried this: RSpec::Core::Pending.pending('bug #56081') but no luck: uninitialized constant RSpec::Core (NameError)
Dan, any idea on how to make this happen?
We could reimplement `#pending` in a world helper module and rescue/re-raise the `Cucumber::Pending` exception as something else. module MediawikiSelenium::Pending class Failure < StandardError; end def pending(*args) super rescue Cucumber::Pending => e if block_given? raise Failure, e.message else raise e end end end World(MediawikiSelenium::Pending)
I think just my Ruby-fu is failing me. See comments 6 and 7. RSpec::Pending already does what I want, but looks like I am calling Cucumber::Pending here: pending('bug #56081') do true.should be(true) end How do I make sure RSpec::Pending is called. See comment 7 on what I have tried.
I'm not sure that cucumber even uses `rspec-core`. It might just use the matchers. Either way, I think implementing our own helper is more inline with the Cucumber Way—not that I think the Cucumber Way is great, but it's better not to wrestle with it too much. If you don't want to override the behavior of `#pending` globally, then we can use a different name (`#pending_or_fail`, `#temporarily_pending`, etc.).
As far as I understand it, Cucumber does use rspec, that is why we explicitly use rspec-expectations[2]. 1: https://github.com/wikimedia/mediawiki-selenium/blob/master/mediawiki_selenium.gemspec#L26
Right, but rspec-expectations is separate from rspec-core which is what implements `describe`, `it`, `pending`, etc.
I see, that was what I misunderstood. Thanks. :) Ok then, so we could do three things: #1 patch Cucumber::Pending #2 require RSpec::Core::Pending and use it #3 report a bug for Cucumber::Pending saying it does not behave as RSpec::Core::Pending, which is confusing
I think I'd still rather just reimplement pending for our own Cucumber World (see comment 9) over monkey patching Cucumber::Pending (#1) or adding another dependency (#2).
(In reply to Dan Duvall from comment #15) > I think I'd still rather just reimplement pending for our own Cucumber World > (see comment 9) over monkey patching Cucumber::Pending (#1) or adding > another dependency (#2). Sounds good to me! :)
Change 160024 had a related patch set uploaded by Dduvall: Stricter pending behavior for falsely passing steps https://gerrit.wikimedia.org/r/160024
Change 160024 merged by jenkins-bot: Stricter pending behavior for falsely passing steps https://gerrit.wikimedia.org/r/160024
Released in 0.4.0.
Thanks Dan, I have tested this and it works great! :)
