|
2021-06-29
§
|
| 14:22 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions |
[production] |
| 14:22 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) |
[production] |
| 14:22 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki |
[production] |
| 14:22 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) |
[production] |
| 14:21 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly |
[production] |
| 14:21 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) |
[production] |
| 14:21 |
<jayme@cumin1001> |
MediaWiki read-only period starts at: 2021-06-29 14:21:26.671853 |
[production] |
| 14:21 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly |
[production] |
| 14:15 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) |
[production] |
| 14:15 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance |
[production] |
| 14:13 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 44 hosts with reason: DC switchover |
[production] |
| 14:13 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 44 hosts with reason: DC switchover |
[production] |
| 14:12 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
| 14:11 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
| 14:10 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
| 14:09 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
| 14:08 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) |
[production] |
| 14:02 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
| 14:01 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
| 14:01 |
<jayme@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
| 13:51 |
<otto@deploy1002> |
Started deploy [analytics/refinery@edc31a2] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@edc31a2] |
[production] |
| 13:49 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@edc31a2] (thin): Regular analytics weekly train THIN [analytics/refinery@edc31a2] (duration: 00m 07s) |
[production] |
| 13:49 |
<otto@deploy1002> |
Started deploy [analytics/refinery@edc31a2] (thin): Regular analytics weekly train THIN [analytics/refinery@edc31a2] |
[production] |
| 13:49 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@edc31a2]: Regular analytics weekly train [analytics/refinery@COMMIT_HASH] (duration: 17m 42s) |
[production] |
| 13:35 |
<volker-e@deploy1002> |
Finished deploy [design/style-guide@e97fccb]: Deploy design/style-guide: e97fccb styles: Add internationalization and accessibility note labels and treatments (#476) (duration: 00m 07s) |
[production] |
| 13:34 |
<volker-e@deploy1002> |
Started deploy [design/style-guide@e97fccb]: Deploy design/style-guide: e97fccb styles: Add internationalization and accessibility note labels and treatments (#476) |
[production] |
| 13:31 |
<otto@deploy1002> |
Started deploy [analytics/refinery@edc31a2]: Regular analytics weekly train [analytics/refinery@COMMIT_HASH] |
[production] |
| 11:54 |
<phuedx@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:702095|vector: Finish enabling language switcher treatment A/B test on fawiki (T269093)]] (duration: 00m 56s) |
[production] |
| 11:38 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/Wikibase/repo/: Backport: [[gerrit:702018|Use EntityLookup backed TermLookup for Rdf PropertyStubs (T285634)]], Part II (duration: 00m 58s) |
[production] |
| 11:36 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/Wikibase/repo/includes/Rdf/PropertyStubRdfBuilder.php: Backport: [[gerrit:702018|Use EntityLookup backed TermLookup for Rdf PropertyStubs (T285634)]], Part I (duration: 00m 56s) |
[production] |
| 11:35 |
<ladsgroup@deploy1002> |
sync-file aborted: Backport: [[gerrit:702018|Use EntityLookup backed TermLookup for Rdf PropertyStubs (T285634)]] (duration: 00m 10s) |
[production] |
| 10:30 |
<moritzm> |
cleanup now unused nginx mods and former deps (various X11 libs and libxslt) on acmechief* after switch towards nginx-light T164456 |
[production] |
| 09:27 |
<moritzm> |
installing nettle security updates on buster |
[production] |
| 08:47 |
<elukey> |
repool mw13[55,84] after debugging - T285634 |
[production] |
| 08:46 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1384.eqiad.wmnet |
[production] |
| 08:46 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1355.eqiad.wmnet |
[production] |
| 08:43 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=elastic1039.eqiad.wmnet |
[production] |
| 08:25 |
<elukey> |
cumin 'A:mw-eqiad' '/usr/local/sbin/restart-php7.2-fpm' -b 2 -s 30 - T285634 |
[production] |
| 08:21 |
<elukey> |
depool mw1355 (mw appserver) for debugging - T285634 |
[production] |
| 08:21 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1355.eqiad.wmnet |
[production] |
| 08:12 |
<hashar> |
Upgrading Jenkins on contint2001 / contint1001 and restarting CI Jenkins # T285531 |
[production] |
| 08:03 |
<hashar> |
Upgraded Jenkins on releases1002 / releases2002 # T285531 |
[production] |
| 08:02 |
<hashar> |
Upgraded Jenkins on releases1002 / releases2002 |
[production] |
| 07:49 |
<godog> |
remove 20G migration data /root/prometheus from prometheus4001 - T243057 |
[production] |
| 07:48 |
<godog> |
remove old /root/prometheus data from prometheus4001 |
[production] |
| 07:05 |
<moritzm> |
upgrading bullseye early installs to the latest state of testing T275873 |
[production] |
| 06:46 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.11/includes/MediaWiki.php: Add statsd action timing metric T284274 (duration: 00m 58s) |
[production] |
| 02:47 |
<cdanis> |
✔️ cdanis@cumin2001.codfw.wmnet ~ 🕥🍺 sudo cumin -b16 'A:cp-upload and A:codfw' 'run-puppet-agent -q' |
[production] |
| 02:34 |
<ryankemper> |
T285643 Banned `elastic1039` from all 3 elasticsearch clusters and set `elastic1039.eqiad.wmnet` to failed in netbox |
[production] |
| 02:27 |
<cdanis> |
✔️ cdanis@cumin2001.codfw.wmnet ~ 🕥🍺 sudo cumin -b16 'A:cp-upload' 'run-puppet-agent -q' |
[production] |