|
2019-10-24
ยง
|
| 14:58 |
<bblack> |
cr3-esams - change fallback static route for high-traffic2 to lvs3006 |
[production] |
| 14:58 |
<bblack> |
cr2-esams - change fallback static route for high-traffic2 to lvs3006 |
[production] |
| 14:47 |
<effie> |
run puppet on all canaries and codfw - T229792 |
[production] |
| 14:42 |
<ema@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 14:40 |
<effie> |
Remove hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from all canaries and codfw - T229792 |
[production] |
| 14:40 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:26 |
<bblack> |
lvs3006 (upload, becoming active) - manual pybal med s/90/0/ (will take over from lvs3002, intended permanently). |
[production] |
| 14:23 |
<bblack> |
lvs3006 (upload, inactive) - manual pybal med s/100/90/ (preferred to lvs3004 for fallback from lvs3002) |
[production] |
| 14:22 |
<effie> |
enable puppet on mw app canaries |
[production] |
| 14:16 |
<ema> |
power-cycle cp3056, stuck rebooting into d-i T233242 |
[production] |
| 13:59 |
<ema> |
pool cp3060 T233242 |
[production] |
| 13:36 |
<bblack> |
re-pooling esams in dns |
[production] |
| 13:34 |
<effie> |
enable puppet on mwdebug* |
[production] |
| 13:25 |
<XioNoX> |
enable transit4/6 on cr2-knams |
[production] |
| 13:24 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=varnish-be,name=cp30[56].* |
[production] |
| 13:24 |
<bblack@cumin1001> |
conftool action : set/weight=100; selector: name=cp30[56].*,service=varnish-be |
[production] |
| 13:23 |
<bblack@cumin1001> |
conftool action : set/weight=1; selector: name=cp30[56].*,cluster=cache_text,service=varnish-fe |
[production] |
| 13:22 |
<bblack@cumin1001> |
conftool action : set/weight=1; selector: name=cp30[56].*,cluster=cache_text,service=nginx |
[production] |
| 13:22 |
<bblack@cumin1001> |
conftool action : set/weight=1; selector: name=cp30[56].*,cluster=cache_upload,service=varnish-fe |
[production] |
| 13:22 |
<bblack@cumin1001> |
conftool action : set/weight=1; selector: name=cp30[56].*,cluster=cache_upload,service=nginx |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3063.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3051.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3059.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3061.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3057.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3065.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3055.esams.wmnet |
[production] |
| 13:18 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: service=ats-be,name=cp3053.esams.wmnet |
[production] |
| 13:17 |
<ema> |
set ats-be weights on new esams upload nodes T233242 |
[production] |
| 13:06 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.3 |
[production] |
| 12:56 |
<effie> |
purge hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from mw* canaries - T229792 |
[production] |
| 12:42 |
<ema@puppetmaster1001> |
conftool action : set/weight=100; selector: name=cp3060.esams.wmnet,service=varnish-be |
[production] |
| 12:33 |
<effie> |
Stopping puppet on all hosts including the hhvm class (C:hhvm) - 544864 - T229792 |
[production] |
| 12:25 |
<ema> |
cp3060: powercycle -- NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [charon:1226] T233242 |
[production] |
| 12:14 |
<bblack> |
depool esams in geodns |
[production] |
| 12:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2092 after analyze table', diff saved to https://phabricator.wikimedia.org/P9468 and previous config saved to /var/cache/conftool/dbconfig/20191024-120812-marostegui.json |
[production] |
| 12:06 |
<XioNoX> |
shutdown cr1-esams - cr2-knams link |
[production] |
| 12:00 |
<XioNoX> |
shutdown transit BGP sessions on cr2-knams |
[production] |
| 11:40 |
<Urbanecm> |
EU SWAT done |
[production] |
| 11:35 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 3a5cb68: Permission changes of move-rootuserpages assignment at commonswiki (T236359) (duration: 01m 00s) |
[production] |
| 11:33 |
<ema@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:31 |
<ema@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:30 |
<Urbanecm> |
Run mwscript namespaceDupes.php --wiki=commonswiki --add-prefix=FIXME --fix (T236352) |
[production] |
| 11:28 |
<vgutierrez@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 11:26 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:26 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: e079956: Add CAT as alias for NS_CATEGORY at commonswiki (T236352) (duration: 01m 00s) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized dblists/commonsuploads.dblist: SWAT: 2d66deb: Restrict uploads on azwiki (T236307) (duration: 01m 03s) |
[production] |
| 11:15 |
<mlitn@deploy1001> |
Synchronized php-1.35.0-wmf.3/extensions/WikibaseMediaInfo: Also use custom PrefetchingTermLookup in SingleEntitySourceServices (duration: 01m 01s) |
[production] |
| 11:13 |
<mlitn@deploy1001> |
Synchronized php-1.35.0-wmf.3/extensions/Wikibase: Allow defining entity-type-specific PrefetchingTermLookup (duration: 01m 06s) |
[production] |
| 11:08 |
<aborrero@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |