|
2021-06-29
§
|
| 07:48 |
<godog> |
remove old /root/prometheus data from prometheus4001 |
[production] |
| 07:05 |
<moritzm> |
upgrading bullseye early installs to the latest state of testing T275873 |
[production] |
| 06:46 |
<tstarling@deploy1002> |
Synchronized php-1.37.0-wmf.11/includes/MediaWiki.php: Add statsd action timing metric T284274 (duration: 00m 58s) |
[production] |
| 02:47 |
<cdanis> |
✔️ cdanis@cumin2001.codfw.wmnet ~ 🕥🍺 sudo cumin -b16 'A:cp-upload and A:codfw' 'run-puppet-agent -q' |
[production] |
| 02:34 |
<ryankemper> |
T285643 Banned `elastic1039` from all 3 elasticsearch clusters and set `elastic1039.eqiad.wmnet` to failed in netbox |
[production] |
| 02:27 |
<cdanis> |
✔️ cdanis@cumin2001.codfw.wmnet ~ 🕥🍺 sudo cumin -b16 'A:cp-upload' 'run-puppet-agent -q' |
[production] |
| 02:25 |
<eileen> |
civicrm revision changed from 927ab7cff7 to 789c92d13b, config revision is 1739c53fcb |
[production] |
| 02:04 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@0e916b1]: 0.3.75 (duration: 08m 40s) |
[production] |
| 01:57 |
<ryankemper> |
[WDQS Deploy] Tests passing following deploy of `0.3.75` on canary `wdqs1003`; proceeding to rest of fleet |
[production] |
| 01:56 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@0e916b1]: 0.3.75 |
[production] |
| 01:50 |
<ryankemper> |
[WDQS Deploy] Gearing up for deploy of wdqs `0.3.75`. Pre-deploy tests passing on canary `wdqs1003` |
[production] |
| 00:25 |
<Krinkle> |
krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761 |
[production] |
|
2021-06-28
§
|
| 23:07 |
<urbanecm> |
Evening B&C window done |
[production] |
| 23:05 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 5ec855d14b31a9392274c2bfe2e21e2ad44986bc: Enable Parsoid inspired media structure on test wikis (T51097) (duration: 00m 59s) |
[production] |
| 22:51 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |
| 22:51 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-update-tendril |
[production] |
| 22:50 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 22:48 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 22:48 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters (exit_code=0) |
[production] |
| 22:44 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters |
[production] |
| 22:43 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
| 22:43 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 22:43 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners (exit_code=0) |
[production] |
| 22:43 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners |
[production] |
| 22:43 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) |
[production] |
| 22:43 |
<legoktm@cumin1001> |
[DRY-RUN] MediaWiki read-only period ends at: 2021-06-28 22:43:04.512602 |
[production] |
| 22:43 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite |
[production] |
| 22:43 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) |
[production] |
| 22:42 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite |
[production] |
| 22:42 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) |
[production] |
| 22:42 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions |
[production] |
| 22:42 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) |
[production] |
| 22:42 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki |
[production] |
| 22:42 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) |
[production] |
| 22:41 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly |
[production] |
| 22:41 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) |
[production] |
| 22:41 |
<legoktm@cumin1001> |
[DRY-RUN] MediaWiki read-only period starts at: 2021-06-28 22:41:41.222740 |
[production] |
| 22:41 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly |
[production] |
| 22:40 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) |
[production] |
| 22:40 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance |
[production] |
| 22:40 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
| 22:38 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
| 22:38 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) |
[production] |
| 22:32 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
| 22:32 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
| 22:32 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
| 22:31 |
<legoktm> |
starting DC switchover live test, which will "switch" us from codfw -> eqiad |
[production] |
| 22:28 |
<eileen> |
civicrm revision changed from 9d1203fb28 to 927ab7cff7, config revision is 1739c53fcb |
[production] |
| 22:09 |
<legoktm> |
live-hacked spicerack on cumin1001 to ignore x2, see https://phabricator.wikimedia.org/T285519#7182377 |
[production] |
| 21:55 |
<Krinkle> |
krinkle@mwmaint1002: purgeParserCache.php --tag pc2, ref T282761 |
[production] |