|
2021-01-18
§
|
| 08:13 |
<elukey> |
clean up old archiva debs and upload 2.2.4-3 to buster-wikimedia - T272082 |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1138 (re)pooling @ 100%: After restarting for kernel upgraed', diff saved to https://phabricator.wikimedia.org/P13786 and previous config saved to /var/cache/conftool/dbconfig/20210118-080122-root.json |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1138 (re)pooling @ 75%: After restarting for kernel upgraed', diff saved to https://phabricator.wikimedia.org/P13785 and previous config saved to /var/cache/conftool/dbconfig/20210118-074618-root.json |
[production] |
| 07:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1138 (re)pooling @ 50%: After restarting for kernel upgraed', diff saved to https://phabricator.wikimedia.org/P13784 and previous config saved to /var/cache/conftool/dbconfig/20210118-073115-root.json |
[production] |
| 07:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1138 (re)pooling @ 25%: After restarting for kernel upgraed', diff saved to https://phabricator.wikimedia.org/P13783 and previous config saved to /var/cache/conftool/dbconfig/20210118-071611-root.json |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1138 for kernel upgrade', diff saved to https://phabricator.wikimedia.org/P13782 and previous config saved to /var/cache/conftool/dbconfig/20210118-065312-marostegui.json |
[production] |
| 06:35 |
<marostegui> |
Reboot dbproxy2001, dbproxy2002, dbproxy2003 for kernel upgrade |
[production] |
| 06:22 |
<marostegui> |
Reboot db1154 and db1155 for kernel upgrade |
[production] |
|
2021-01-15
§
|
| 23:59 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:57 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:22 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti5002.eqsin.wmnet with reason: REIMAGE |
[production] |
| 21:19 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti5002.eqsin.wmnet with reason: REIMAGE |
[production] |
| 20:39 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (3/3; T272075) (duration: 00m 55s) |
[production] |
| 20:37 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/wikipedia-tagline-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (2/3; T272075) (duration: 00m 55s) |
[production] |
| 20:36 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/wikipedia-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (1/3; T272075) (duration: 00m 58s) |
[production] |
| 20:21 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/wikipedia-fr-20.svg: 66e6be391ecfde7ca0604146ab978987ce472b5c: Set anniversary logo for frwiki (1/3; T272075) (duration: 01m 54s) |
[production] |
| 17:17 |
<legoktm> |
legoktm@contint2001:~$ sudo systemctl reload apache2 # for T272159 |
[production] |
| 16:17 |
<bstorm> |
canceled downtime for maintain-dbusers on labstore1004 T272127 |
[production] |
| 15:30 |
<elukey> |
restart archiva to apply hot-fix for T272082 |
[production] |
| 15:17 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1002.wikimedia.org |
[production] |
| 15:14 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1002.wikimedia.org |
[production] |
| 15:11 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1001.wikimedia.org |
[production] |
| 15:07 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1001.wikimedia.org |
[production] |
| 15:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moscovium.eqiad.wmnet |
[production] |
| 15:01 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host moscovium.eqiad.wmnet |
[production] |
| 14:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2003.wikimedia.org |
[production] |
| 14:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2003.wikimedia.org |
[production] |
| 14:29 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2004.wikimedia.org |
[production] |
| 14:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2004.wikimedia.org |
[production] |
| 11:30 |
<jynus> |
rolling restart of eqiad source backup dbs |
[production] |
| 11:19 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1002.eqiad.wmnet |
[production] |
| 11:15 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-conf1002.eqiad.wmnet |
[production] |
| 11:11 |
<XioNoX> |
update cloud-in4 firewall rules |
[production] |
| 11:06 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2036.codfw.wmnet |
[production] |
| 10:59 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc2036.codfw.wmnet |
[production] |
| 10:58 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1003.eqiad.wmnet |
[production] |
| 10:56 |
<jiji@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host mc2036.codfw.wmnet |
[production] |
| 10:55 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc2036.codfw.wmnet |
[production] |
| 10:53 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-conf1003.eqiad.wmnet |
[production] |
| 10:53 |
<vgutierrez> |
re-enable puppet on acme-chief clients |
[production] |
| 10:53 |
<jynus> |
rolling restart of dbprov2* hosts |
[production] |
| 10:52 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1001.eqiad.wmnet |
[production] |
| 10:52 |
<_joe_> |
rebuilding the docker images coredns,nutcracker,prometheus-statsd-exporter,service-checker,wmfdebug to use wikimedia-buster as a base |
[production] |
| 10:51 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1001.eqiad.wmnet |
[production] |
| 10:48 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host acmechief1001.eqiad.wmnet |
[production] |
| 10:46 |
<vgutierrez> |
disable puppet on acme-chief clients |
[production] |
| 10:45 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host an-conf1001.eqiad.wmnet |
[production] |