Bug 192161

Summary: webkitpy: Sort tests by associated device type
Product: WebKit Reporter: Jonathan Bedard <jbedard>
Component: Tools / TestsAssignee: Jonathan Bedard <jbedard>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, ews-watchlist, glenn, lforschler, rniwa, slewis, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=192409
https://bugs.webkit.org/show_bug.cgi?id=192470
https://bugs.webkit.org/show_bug.cgi?id=192486
Attachments:
Description Flags
Patch
none
Patch
none
Patch
none
Archive of layout-test-results from ews101 for mac-sierra
none
Patch
none
Patch for landing none

Jonathan Bedard
Reported 2018-11-29 09:59:05 PST
There are a number of circumstances where we want to run a specific test on a specific device type. Our current technique for doing this requires each port defining CUSTOM_DEVICE_CLASSES. This approach is problematic because it doesn't work if a simulator is already booted or when running on-device testing. Instead, we should check which device types are available.
Attachments
Patch (19.51 KB, patch)
2018-12-03 17:03 PST, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (21.11 KB, patch)
2018-12-04 10:28 PST, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch (25.76 KB, patch)
2018-12-04 16:09 PST, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Archive of layout-test-results from ews101 for mac-sierra (2.45 MB, application/zip)
2018-12-05 00:10 PST, EWS Watchlist 		no flags
Details
Patch (26.36 KB, patch)
2018-12-05 08:41 PST, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Patch for landing (26.39 KB, patch)
2018-12-05 10:52 PST, Jonathan Bedard 		no flags
DetailsFormatted DiffDiff
Show Obsolete (4) View All Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2018-11-29 10:02:52 PST
<rdar://problem/46345392>
Jonathan Bedard
Comment 2 2018-12-03 13:50:05 PST
My original description of this bug isn't accurate. I think what makes more sense, given the goals of <https://bugs.webkit.org/show_bug.cgi?id=192162>, is to generate a mapping of device-types and test lists then iterate through each device type, checking if the current port supports that device type. Once <https://bugs.webkit.org/show_bug.cgi?id=192162> lands, we'll be able to get ride of the CUSTOM_DEVICE_CLASSES.
Jonathan Bedard
Comment 3 2018-12-03 14:42:33 PST
(In reply to Jonathan Bedard from comment #2) > ... > <https://bugs.webkit.org/show_bug.cgi?id=192162> lands, we'll be able to get > ride of the CUSTOM_DEVICE_CLASSES. ...be able to get RID of the CUSTOM_DEVICE_CLASSES.
Jonathan Bedard
Comment 4 2018-12-03 17:03:58 PST
Created attachment 356439 [details] Patch
Jonathan Bedard
Comment 5 2018-12-04 08:44:02 PST
Comment on attachment 356439 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=356439&action=review > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:259 > + while index < len(device_type_order): Consider the example: [iPhone SE, iPhone 7, iPhone, iPad]. In this example, we don't want 'iPhone' tests to boot their own simulator if they don't have to. What this code will do is run the iPhone tests with the iPhone SE, because iPhone matches iPhone SE, then remove both iPhone SE and iPhone from the list. [iPhone 7, iPad] will be left. Then, we will boot an iPhone 7. iPad does not match iPhone 7, so after the iPhone 7 tests, we will boot an iPad.
Jonathan Bedard
Comment 6 2018-12-04 10:28:33 PST
Created attachment 356514 [details] Patch
Jonathan Bedard
Comment 7 2018-12-04 16:09:25 PST
Created attachment 356550 [details] Patch
EWS Watchlist
Comment 8 2018-12-05 00:10:12 PST
Comment on attachment 356550 [details] Patch Attachment 356550 [details] did not pass mac-ews (mac): Output: https://webkit-queues.webkit.org/results/10275026 New failing tests: http/tests/misc/resource-timing-resolution.html
EWS Watchlist
Comment 9 2018-12-05 00:10:13 PST
Created attachment 356587 [details] Archive of layout-test-results from ews101 for mac-sierra The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews101 Port: mac-sierra Platform: Mac OS X 10.12.6
Jonathan Bedard
Comment 10 2018-12-05 08:41:32 PST
Created attachment 356608 [details] Patch
Lucas Forschler
Comment 11 2018-12-05 09:58:17 PST
Comment on attachment 356608 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=356608&action=review > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:269 > + # Remove devices types from device_type_order once they have been fulfilled to me, fulfilled implies completed, with no work to be done... but isn't this logic adding it to the list/queue? I think it might be better to replace fulfilled with 'claimed | queued | recorded | scheduled?
Jonathan Bedard
Comment 12 2018-12-05 10:04:49 PST
(In reply to Lucas Forschler from comment #11) > Comment on attachment 356608 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=356608&action=review > > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:269 > > + # Remove devices types from device_type_order once they have been fulfilled > > to me, fulfilled implies completed, with no work to be done... but isn't > this logic adding it to the list/queue? > > I think it might be better to replace fulfilled with 'claimed | queued | > recorded | scheduled? Going with: # Remove devices types from device_type_order once tests associated with that type have been claimed.
Jonathan Bedard
Comment 13 2018-12-05 10:52:33 PST
Created attachment 356618 [details] Patch for landing
WebKit Commit Bot
Comment 14 2018-12-05 11:30:54 PST
Comment on attachment 356618 [details] Patch for landing Clearing flags on attachment: 356618 Committed r238903: <https://trac.webkit.org/changeset/238903>
WebKit Commit Bot
Comment 15 2018-12-05 11:30:56 PST
All reviewed patches have been landed. Closing bug.
Note You need to log in before you can comment on or make changes to this bug.