Bug 57983

I disabled all media tests: http://trac.webkit.org/changeset/83100

Guys, first we should try to find what cause these random crashes. If we found and fixed it, we can enable media tests. But we shouldn't kill the reputation and the trustiness of the Qt bot with these crazy crashy flakey tests.

The flakey I could find in today's history: media/remove-from-document.html -> crashed media/video-preload.html -> crashed media/video-plays-past-end-of-test.html -> crashed media/video-src.html -> crashed media/video-volume.html -> crashed That looks pretty random :(

Created attachment 98158 [details] Skip only failing media tests I downloaded the Qt Webkit VM to analyze the flakey media tests. I modified the Skipped file to skip only those tests that failed outright on the VM. I then ran the layout tests many times and did not see any of the media tests fail. With this set of tests I did not see any flakiness. I'm not sure if this bug was logged before or after the buildbot was upgraded to Qt Mobility 1.2 so it is not clear to me what version of Qt Mobility was in place when the flakiness was observerd. In other words, I could not reproduce the flakiness in order to find a fix for it but I could find a stable set of tests.

Comment on attachment 98158 [details] Skip only failing media tests Rejecting attachment 98158 [details] from review queue. nancy.piedra@nokia.com does not have reviewer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py. - If you do not have reviewer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags. - If you have reviewer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed). The commit-queue restarts itself every 2 hours. After restart the commit-queue will correctly respect your reviewer rights.

Comment on attachment 98158 [details] Skip only failing media tests Rejecting attachment 98158 [details] from commit-queue. nancy.piedra@nokia.com does not have committer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py. - If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags. - If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed). The commit-queue restarts itself every 2 hours. After restart the commit-queue will correctly respect your committer rights.

Sorry, meant to do review? commit-queue? but hit + instead.

I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list.

(In reply to comment #8) > I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list. Yep I think we should give a try.

Comment on attachment 98158 [details] Skip only failing media tests View in context: https://bugs.webkit.org/attachment.cgi?id=98158&action=review > LayoutTests/ChangeLog:6 > + Skipped the Qt media tests that are failing on the Qt Webkit VM Extra line here.

Comment on attachment 98158 [details] Skip only failing media tests $ Tools/Scripts/run-webkit-tests media media/video-preload.html -> failed media/video-seek-no-src-exception.html -> timed out media/video-seekable.html -> failed media/video-source-inserted.html -> timed out media/video-source-moved.html -> timed out media/video-src-change.html -> timed out media/video-src-remove.html -> failed media/video-src-set.html -> timed out media/video-src.html -> timed out media/video-volume.html -> failed and then run-webkit-tests waits forever ... :( r- now, it needs more investigation

media/adopt-node-crash.html -> timed out

media/video-source-none-supported.html -> timed out media/video-src-invalid-remove.html -> failed

Ossy - Are you running the tests on the VM? If not, what is the configuration of the system you are running the tests on.

media/video-source-removed.html -> crashed media/before-load-member-access.html -> timed out media/video-src-source.html -> timed out Media tests are still flakey. :-S Unskipping flakey tests won't solve the bug, we shouldn't enable them until we find and fix the bug.

(In reply to comment #14) > Ossy - Are you running the tests on the VM? If not, what is the configuration of the system you are running the tests on. I ran tests on the buildbot, not on the VM. But they have same configuration.

I tried to find flakey tests with: $ run-tests-in-xvfb.sh media --iterations 100

OK. I will try what you have done and see if I get similar results. If I do, I will continue to debug.

I ran the following on the VM and saw no tests fail or time out: run-tests-in-xvfb.sh media --iterations 100 I will keep trying to see if I can reproduce what you're seeing on the buildbot.

(In reply to comment #19) > I ran the following on the VM and saw no tests fail or time out: > run-tests-in-xvfb.sh media --iterations 100 > > I will keep trying to see if I can reproduce what you're seeing on the buildbot. Very strange .... I try to check it on the VM too.

(In reply to comment #20) > (In reply to comment #19) > > I ran the following on the VM and saw no tests fail or time out: > > run-tests-in-xvfb.sh media --iterations 100 > > > > I will keep trying to see if I can reproduce what you're seeing on the buildbot. > > Very strange .... I try to check it on the VM too. That's exactly what I keep saying. The VM does not behave the same way as the bots. It also seems to behave differently whether you run with VirtualBox or VMWare, it sounds unbelievable but it is.

I ran it again and now I am seeing some failures.

Created attachment 98369 [details] Skip only failing media tests (2nd try) Ossy - I commented out a few more tests. Could you try this? If it doesn't work on the buildbot then I will look into the flakiness more.

Comment on attachment 98369 [details] Skip only failing media tests (2nd try) r- again, because I got flakey fails and timeouts media/video-single-valid-source.html media/adopt-node-crash.html media/video-source-inserted.html media/video-src.html media/before-load-member-access.html media/broken-video.html media/constructors.html media/controls-css-overload.html

The Qt port was removed from the project, so this bug is not valid anymore.