9801-9850 of 10000 results (96ms)
2020-02-10 ยง
12:56 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 989c9f8: Revert "Revert "Remove handler deleted from the MachineVision extension"" (duration: 00m 58s) [production]
12:51 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 989c9f8: Revert "Revert "Remove handler deleted from the MachineVision extension"" (duration: 00m 59s) [production]
12:49 <urbanecm@deploy1001> Finished scap: SWAT: 799224f: 137a40e (T241242; T243974) (duration: 20m 18s) [production]
12:30 <vgutierrez> depool cp5004 and reimage as buster - T242093 [production]
12:29 <vgutierrez> pooling cp5005 with buster - T242093 [production]
12:28 <urbanecm@deploy1001> Started scap: SWAT: 799224f: 137a40e (T241242; T243974) [production]
12:23 <vgutierrez> pooling ncredir1001 with buster - T243391 [production]
12:18 <_joe_> running puppet, scap pull on mwdebug1001 [production]
12:17 <vgutierrez> upload trafficserver 8.0.5-1wm15 to apt.wm.o (buster) - T244538 [production]
12:08 <vgutierrez@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
12:08 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:06 <vgutierrez> testing ats 8.0.5-1-wm15 on cp4032 - T244538 [production]
12:06 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: SWAT: 014405a: Add throttle rules for OSU Editathon and workshop for cawiki, remove expired ones (T244608, T244645) (duration: 01m 03s) [production]
11:57 <vgutierrez> depool ncredir1001 and reimage as buster - T243391 [production]
11:57 <vgutierrez> pooling ncredir1002 with buster - T243391 [production]
11:43 <vgutierrez> pooling cp4027 with buster - T242093 [production]
11:38 <vgutierrez> depool ncredir1002 and reimage as buster - T243391 [production]
11:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:29 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:22 <vgutierrez> depooling cp5011 and cp5005 & reimage as buster - T242093 [production]
11:07 <vgutierrez> depool cp4027 & reimage as buster - T242093 [production]
11:07 <vgutierrez> pooling ncredir2001 with buster - T243391 [production]
11:03 <vgutierrez> pooling cp4028 with buster - T242093 [production]
10:50 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:48 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:47 <godog> remove old logs from /var/log/swift on swift hsots [production]
10:31 <vgutierrez> depool ncredir2001 and reimage as buster - T243391 [production]
10:26 <vgutierrez> depool cp4028 & reimage as buster - T242093 [production]
10:14 <moritzm> installing sudo security updates for buster [production]
08:53 <vgutierrez> pooling cp4029 with buster - T242093 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight from 1 to 5 for db1107 - T242702', diff saved to https://phabricator.wikimedia.org/P10364 and previous config saved to /var/cache/conftool/dbconfig/20200210-084446-marostegui.json [production]
08:43 <vgutierrez> pooling ncredir2002 with buster - T243391 [production]
08:34 <effie> rolling restart php-fpm on labweb[1001-1002].wikimedia.org,mw*.eqiad.wmnet,scandium.eqiad.wmnet, wtp[1025-1048].eqiad.wmnet [production]
08:32 <effie> update php-apcu on eqiad - T236800 [production]
08:29 <effie> rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet [production]
08:23 <effie> update php-apcu on codfw - T236800 [production]
07:58 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:56 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:54 <moritzm> updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) [production]
07:29 <moritzm> updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) [production]
07:09 <elukey> restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json [production]
06:55 <vgutierrez> depool ncredir2002 and reimage as buster - T243391 [production]
06:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json [production]
06:47 <vgutierrez> depool cp4029 & reimage as buster - T242093 [production]
06:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json [production]
06:39 <marostegui> Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 [production]
06:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json [production]