|
2020-09-15
§
|
| 09:04 |
<gehel> |
restart elasticsearch on elastic2029 (high GC |
[production] |
| 09:01 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:59 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 08:58 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 08:53 |
<elukey> |
roll restart druid zookeeper clusters for openjdk upgrades |
[production] |
| 08:53 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:13 |
<marostegui> |
Stop MySQL on labsdb1010 for PDU maintenance T261456 |
[production] |
| 08:05 |
<liw@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_498180604" --store-class=LCStoreCDB --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 11m 10s) |
[production] |
| 08:04 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 08:02 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:01 |
<akosiaris> |
T187984 migration script on otrs1001 proceeding as expected. Still in step 31/44, but that's what we saw in the test migration |
[production] |
| 07:54 |
<liw@deploy1001> |
Started scap: testwikis to 1.36.0-wmf.9 |
[production] |
| 07:24 |
<godog> |
swift codfw add ms-be2057 at object weight 100 - T261633 |
[production] |
| 07:19 |
<elukey> |
roll restart druid cluster to pick up openjdk updates |
[production] |
| 07:19 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 07:16 |
<XioNoX> |
pre-configure SGIX port on cr2-eqsin |
[production] |
| 06:57 |
<liw> |
1.36.0-wmf.9 was branched at 7269b6b57b6f79646b96ece818d2f2d38e0d2ea6 for T257977 |
[production] |
| 06:08 |
<marostegui> |
Stop mysql on es2011 to clone es2028 |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2011 to clone es2028', diff saved to https://phabricator.wikimedia.org/P12585 and previous config saved to /var/cache/conftool/dbconfig/20200915-060623-marostegui.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set es2012 as es1 codfw master T261717', diff saved to https://phabricator.wikimedia.org/P12584 and previous config saved to /var/cache/conftool/dbconfig/20200915-060508-marostegui.json |
[production] |
| 05:33 |
<marostegui> |
Depool labsdb1010 for PDU maintenance |
[production] |
| 05:10 |
<marostegui> |
Restart sanitarium hosts on eqiad and codfw T262832 |
[production] |
|
2020-09-14
§
|
| 22:59 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:59 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:49 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:49 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:45 |
<mholloway-shell@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 21:34 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:32 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:30 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'enable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:24 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'disable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:05 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:03 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:38 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:36 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<cdanis@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a588eb0c6 T262087 modify wgEventStreams to reference NEL schema (duration: 00m 56s) |
[production] |
| 19:00 |
<Urbanecm> |
Morning B&C done |
[production] |
| 18:57 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a5d56edc7460ac43492f9c04cff86c1b03e56fa4: e2f47980c371b52b1b66957f2bff2266745ab00a: Enable Special:Investigate on eswiki (T262436) (duration: 00m 56s) |
[production] |
| 18:49 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:47 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:38 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 7d1939323cc3ea5dacf67a43d4d359c114203a66: Remove investigate from $wgAvailableRights (T260175) (duration: 00m 56s) |
[production] |
| 18:32 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: d2fa6533a594c8544342954eae19a4a0f7baeff0: Remove the investigate right from testwiki and frwiki (T260175) (duration: 00m 56s) |
[production] |
| 18:30 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/EventStreamConfig/includes/: a4c86089371319ae5a3bb6053c4a9b3e83130286: Default to using API json formatversion=2 (T251609) (duration: 00m 57s) |
[production] |
| 18:21 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 27ba5a1da1fb00e721cfa82dd4cd1fbac2541053: add new parse* servers to $wgLinterSubmitterWhitelist (T247441) (duration: 00m 56s) |
[production] |
| 18:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: 720e6cbfe1800fe32dc65c209240ba08706dbb17: flaggedrevs: Move setting of wgFlaggedRevsAutopromote and wgFlaggedRevsAutoconfirm out of wgExtensionFunctions (T237191) (duration: 00m 56s) |
[production] |
| 18:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 699f5e8c2a50f35e98850ea32f7847d183600351: Add logo Wordmark and Tagline for hywiki (T259985) (duration: 00m 55s) |
[production] |
| 18:08 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/: 699f5e8c2a50f35e98850ea32f7847d183600351: Add logo Wordmark and Tagline for hywiki (T259985) (duration: 00m 56s) |
[production] |
| 17:51 |
<mutante> |
all new parse* parsoid hardware pooled now and set to active in netbox, deploy in 10 min will add to $wgLinterSubmitterWhitelist (T247441) |
[production] |
| 17:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse20[1-2][0-9].codfw.wmnet |
[production] |