|
2020-09-15
§
|
| 11:17 |
<effie> |
roll out scap 3.15.0-1 to all - T261234 |
[production] |
| 11:12 |
<XioNoX> |
mass update SCS SNMP community in LibreNMS - T246890 |
[production] |
| 10:58 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 10:56 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:54 |
<XioNoX> |
mass update PDU SNMP community in LibreNMS - T246890 |
[production] |
| 10:48 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 10:36 |
<moritzm> |
uploaded libxml2 2.9.1+dfsg1-5+deb8u8+wmf1 for jessie-wikimedia |
[production] |
| 10:33 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 10:22 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "testwikiswikis to 1.36.0-wmf.9" |
[production] |
| 10:12 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 09:22 |
<marostegui> |
Stop MySQL on s5 and s8 eqiad primary master - lag will show up on labsdb hosts T261455 |
[production] |
| 09:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:13 |
<oblivian@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 09:08 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 09:05 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 09:05 |
<oblivian@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 09:04 |
<gehel> |
restart elasticsearch on elastic2029 (high GC |
[production] |
| 09:01 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:59 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) |
[production] |
| 08:58 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 08:53 |
<elukey> |
roll restart druid zookeeper clusters for openjdk upgrades |
[production] |
| 08:53 |
<elukey@cumin1001> |
START - Cookbook sre.zookeeper.roll-restart-zookeeper |
[production] |
| 08:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:13 |
<marostegui> |
Stop MySQL on labsdb1010 for PDU maintenance T261456 |
[production] |
| 08:05 |
<liw@deploy1001> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_498180604" --store-class=LCStoreCDB --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 11m 10s) |
[production] |
| 08:04 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 08:02 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) |
[production] |
| 08:01 |
<akosiaris> |
T187984 migration script on otrs1001 proceeding as expected. Still in step 31/44, but that's what we saw in the test migration |
[production] |
| 07:54 |
<liw@deploy1001> |
Started scap: testwikis to 1.36.0-wmf.9 |
[production] |
| 07:24 |
<godog> |
swift codfw add ms-be2057 at object weight 100 - T261633 |
[production] |
| 07:19 |
<elukey> |
roll restart druid cluster to pick up openjdk updates |
[production] |
| 07:19 |
<elukey@cumin1001> |
START - Cookbook sre.druid.roll-restart-workers |
[production] |
| 07:16 |
<XioNoX> |
pre-configure SGIX port on cr2-eqsin |
[production] |
| 06:57 |
<liw> |
1.36.0-wmf.9 was branched at 7269b6b57b6f79646b96ece818d2f2d38e0d2ea6 for T257977 |
[production] |
| 06:08 |
<marostegui> |
Stop mysql on es2011 to clone es2028 |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es2011 to clone es2028', diff saved to https://phabricator.wikimedia.org/P12585 and previous config saved to /var/cache/conftool/dbconfig/20200915-060623-marostegui.json |
[production] |
| 06:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set es2012 as es1 codfw master T261717', diff saved to https://phabricator.wikimedia.org/P12584 and previous config saved to /var/cache/conftool/dbconfig/20200915-060508-marostegui.json |
[production] |
| 05:33 |
<marostegui> |
Depool labsdb1010 for PDU maintenance |
[production] |
| 05:10 |
<marostegui> |
Restart sanitarium hosts on eqiad and codfw T262832 |
[production] |
|
2020-09-14
§
|
| 22:59 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:59 |
<mholloway-shell@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:49 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . |
[production] |
| 22:49 |
<mholloway-shell@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 22:45 |
<mholloway-shell@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 21:34 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:32 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:30 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'enable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:24 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'disable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:05 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:03 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |