|
2020-09-22
ยง
|
| 16:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:00 |
<robh> |
running dell epsa test on down host mw1360 per T262151 |
[production] |
| 14:34 |
<moritzm> |
installing nginx security updates on buster |
[production] |
| 14:33 |
<shdubsh> |
restart apache on prometheus nodes to pick up new ext endpoint |
[production] |
| 14:24 |
<ema> |
upload libvmod-re2 1.5.3-1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 14:24 |
<papaul> |
rebooting ms-be2019 |
[production] |
| 14:15 |
<XioNoX> |
upgrade FNM on netflow2001 - T257035 |
[production] |
| 14:12 |
<jayme> |
running ipvsadm -D -t 10.2.1.19:1970; ipvsadm -D -t 10.2.1.21:24766 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet - T255868 T255877 |
[production] |
| 14:12 |
<jayme> |
running ipvsadm -D -t 10.2.2.19:1970; ipvsadm -D -t 10.2.2.21:24766 on lvs1016.eqiad.wmnet,lvs1015.eqiad.wmnet - T255868 T255877 |
[production] |
| 14:11 |
<jayme> |
restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255868 T255877 |
[production] |
| 14:10 |
<XioNoX> |
upgrade FNM on netflow5001 - T257035 |
[production] |
| 14:09 |
<jayme> |
restarting pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255868 T255877 |
[production] |
| 14:09 |
<shdubsh> |
restart statsv on webperf[1-2]001 to route metrics through statsd-exporter |
[production] |
| 14:09 |
<XioNoX> |
upgrade FNM on netflow1001 - T257035 |
[production] |
| 14:06 |
<XioNoX> |
upgrade FNM on netflow3001 - T257035 |
[production] |
| 14:05 |
<jayme> |
running puppet on lvs servers - T255868 T255877 |
[production] |
| 14:03 |
<hnowlan@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 14:02 |
<hnowlan> |
roll-restarting restbase codfw for java updates |
[production] |
| 13:59 |
<XioNoX> |
add fastnetmon_1.1.7 to buster-wikimedia repo - T257035 |
[production] |
| 13:55 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 13:55 |
<ema> |
upload varnish-modules 0.15.0-1+wmf1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 13:49 |
<marostegui> |
Deploy MCR change on db2098:3313 - T238966 |
[production] |
| 13:44 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:39 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:35 |
<ema> |
upload libvmod-netmapper 1.8-1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 12:54 |
<ema> |
upload varnishkafka 1.1.0-1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 12:11 |
<moritzm> |
installing python3.7 security updates on Buster |
[production] |
| 12:09 |
<moritzm> |
installing bundler updates on buster |
[production] |
| 11:59 |
<Urbanecm> |
Reset password for SUL User:Freibo |
[production] |
| 11:58 |
<Lucas_WMDE> |
EU backport&config window done |
[production] |
| 11:56 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwmaint2001:~$ mwscript namespaceDupes.php trwikisource --fix | tee T263358.fix # 1350 to fix, 1350 resolvable, 0 deleted |
[production] |
| 11:55 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@mwmaint2001:~$ mwscript namespaceDupes.php trwikisource | tee T263358.dryrun # 1350 to fix, 1350 resolvable, 0 deleted |
[production] |
| 11:54 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:628598|Create Portal and Portal_talk namespaces on trwikisource, and fix an incorrect alias (T263358)]] (duration: 00m 57s) |
[production] |
| 11:47 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:628521|Removing Wikipedia store link from enwiki (T262329)]] (duration: 00m 57s) |
[production] |
| 11:37 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:628515|Set timezone for wikis of the CWIRP to Europe/Rome (T263123)]] (duration: 00m 59s) |
[production] |
| 11:35 |
<hnowlan@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 11:35 |
<hnowlan> |
roll-restarting restbase eqiad for java updates |
[production] |
| 11:25 |
<ema> |
upload varnish 6.0.6-1wm1 to buster-wikimedia component/varnish6 T261632 |
[production] |
| 11:24 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) |
[production] |
| 11:13 |
<moritzm> |
installing intel-microcode 3.20200616.1 on Buster baremetal servers (compared to to current installed packages this reverts microcode changes for some Skylake CPUs we don't use |
[production] |
| 11:00 |
<moritzm> |
installing intel-microcode 3.20200616.1 on Stretch baremetal servers (compared to to current installed packages this reverts microcode changes for some Skylake CPUs we don't use |
[production] |
| 10:51 |
<XioNoX> |
Add policy-options for primary IXPs to all routers - T262517 |
[production] |
| 10:48 |
<hnowlan@cumin1001> |
START - Cookbook sre.cassandra.roll-restart |
[production] |
| 10:48 |
<hnowlan> |
roll-restarting sessionstore for java security updates |
[production] |
| 10:44 |
<moritzm> |
installing bacula security updates on stretch |
[production] |
| 10:33 |
<moritzm> |
installing remaining libx11 security updates |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2034 (re)pooling @ 100%: Slowly repool es2034 T261717 ', diff saved to https://phabricator.wikimedia.org/P12733 and previous config saved to /var/cache/conftool/dbconfig/20200922-101342-root.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2033 (re)pooling @ 100%: Slowly repool es2033 T261717 ', diff saved to https://phabricator.wikimedia.org/P12732 and previous config saved to /var/cache/conftool/dbconfig/20200922-101324-root.json |
[production] |
| 10:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 100%: Slowly es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12731 and previous config saved to /var/cache/conftool/dbconfig/20200922-101308-root.json |
[production] |
| 10:00 |
<kormat> |
deploying schema change to s2 in eqiad. labsdb will have s2 lag until this finishes. T259831 |
[production] |