|
2021-06-29
ยง
|
| 21:42 |
<maryum> |
deployed updated security patch for T285190 to wmf.11 |
[production] |
| 21:31 |
<sbassett> |
Reverted and deployed updated security patch for T285190 to wmf.12 |
[production] |
| 21:29 |
<sbassett> |
Reverted and deployed updated security patch for T285190 to wmf.11 |
[production] |
| 21:19 |
<sbassett> |
Deployed updated security patch for T285190 to wmf.11 |
[production] |
| 20:55 |
<dancy> |
Deleted all CDB files on beta so they'll be recreated on the next scap sync-world run |
[production] |
| 20:26 |
<dancy> |
Reverting to scap 3.17.1-1+0~20210419163335.8~1.gbpa6b2e0 in beta |
[production] |
| 19:43 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1020.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:41 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1020.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:35 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1019.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:33 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1019.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:28 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1018.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:26 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1018.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:17 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1017.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:14 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1017.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:07 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.12 |
[production] |
| 18:57 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:55 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:34 |
<Krinkle> |
krinkle@mwmaint2002.codfw: mwscript purgeParserCache.php --wiki=aawiki --age=1814400 --msleep 200 --tag pc3 |
[production] |
| 18:28 |
<Krinkle> |
krinkle@mwmaint2002.codfw: mwscript purgeParserCache.php --wiki=aawiki --age=1814400 --msleep 200 --tag pc2 |
[production] |
| 18:21 |
<Krinkle> |
krinkle@mwmaint2002.codfw: mwscript purgeParserCache.php --wiki=aawiki --age=1814400 --msleep 200 --tag pc1 |
[production] |
| 18:11 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:09 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:07 |
<brennen@deploy1002> |
Pruned MediaWiki: 1.37.0-wmf.7 (duration: 04m 00s) |
[production] |
| 17:59 |
<urbanecm> |
Start server-side upload of ~2.5G of JPG files (T282755) |
[production] |
| 17:52 |
<brennen@deploy1002> |
Finished scap: testwikis wikis to 1.37.0-wmf.12 (duration: 57m 11s) |
[production] |
| 16:55 |
<ryankemper> |
T281327 `[Cirrus -> codfw]` Current banned nodes are`elastic2043` and `elastic2045`; `elastic2043` can be unbanned after a re-image, and `elastic2045` can be unbanned in ~30 minutes after shards rebalance (had heavy shards scheduled) |
[production] |
| 16:55 |
<brennen@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.12 |
[production] |
| 16:45 |
<brennen> |
1.37.0-wmf.12 was branched at 3703c3194b590a1fcccb485245022eac369d2b69 for T281153 |
[production] |
| 16:28 |
<ebernhardson> |
temporarily ban elastic2045 from production-search-codfw |
[production] |
| 15:43 |
<dcausse> |
unbanning elastic2054 |
[production] |
| 15:30 |
<dcausse> |
restarting blazegraph on wdqs1012 |
[production] |
| 15:17 |
<effie> |
pool mw2383 back |
[production] |
| 15:15 |
<mutante> |
[mwlog2002:~] $ sudo systemctl start mw-log-cleanup |
[production] |
| 15:06 |
<dcausse> |
banning elastic2054 |
[production] |
| 14:53 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw227[1-2].codfw.wmnet,service=canary |
[production] |
| 14:52 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw227[8-9].codfw.wmnet,service=canary |
[production] |
| 14:52 |
<dzahn@cumin1001> |
conftool action : set/weight=1; selector: name=mw225[1-2].codfw.wmnet,service=canary |
[production] |
| 14:52 |
<effie> |
depool mw2383 as it is misbehaving |
[production] |
| 14:47 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |
| 14:47 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-update-tendril |
[production] |
| 14:47 |
<dzahn@cumin1001> |
conftool action : set/weight=25; selector: name=mw226[1-2].codfw.wmnet |
[production] |
| 14:47 |
<dzahn@cumin1001> |
conftool action : set/weight=25; selector: name=mw2290.codfw.wmnet |
[production] |
| 14:46 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 14:46 |
<dzahn@cumin1001> |
conftool action : set/weight=25; selector: name=mw22[7-8][0-9].codfw.wmnet |
[production] |
| 14:45 |
<dzahn@cumin1001> |
conftool action : set/weight=25; selector: name=mw225[1-8].codfw.wmnet |
[production] |
| 14:44 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 14:44 |
<dzahn@cumin1001> |
conftool action : set/weight=25; selector: name=mw225[1-8].codfw.wmnet,service=api_appserver |
[production] |
| 14:43 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters (exit_code=0) |
[production] |
| 14:38 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters |
[production] |
| 14:38 |
<_joe_> |
restarting pohp-fpm on mw2383 |
[production] |