2021-06-07
§
|
10:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1004.wikimedia.org |
[production] |
10:34 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1157.eqiad.wmnet with reason: REIMAGE |
[production] |
10:32 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1003.wikimedia.org |
[production] |
10:30 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica1003.wikimedia.org |
[production] |
10:28 |
<kormat> |
reimaging db1157 T283131 |
[production] |
10:24 |
<moritzm> |
remove now obsolete nginx mods and dependencies on htmldumper1001 T164456 |
[production] |
10:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2006.wikimedia.org |
[production] |
10:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2006.wikimedia.org |
[production] |
10:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2005.wikimedia.org |
[production] |
10:10 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2005.wikimedia.org |
[production] |
10:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host theemin.codfw.wmnet |
[production] |
10:08 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db1157 depooling: reimage to buster T283131', diff saved to https://phabricator.wikimedia.org/P16311 and previous config saved to /var/cache/conftool/dbconfig/20210607-100822-kormat.json |
[production] |
09:55 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host theemin.codfw.wmnet |
[production] |
09:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
09:48 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet |
[production] |
09:43 |
<moritzm> |
upgrading bullseye hosts to latest packages in testing |
[production] |
09:39 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1002.eqiad.wmnet |
[production] |
09:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host failoid1002.eqiad.wmnet |
[production] |
09:19 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2002.codfw.wmnet |
[production] |
09:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host failoid2002.codfw.wmnet |
[production] |
09:03 |
<moritzm> |
installing imagemagick security updates on stretch |
[production] |
06:05 |
<marostegui> |
Upgrade mysql on dbstore1003 T283235 |
[production] |
05:57 |
<marostegui> |
Stop dbstore1004 to clone dbstore1007 T283125 |
[production] |
05:37 |
<marostegui> |
Depool clouddb1020 (s5, s8) for upgrade |
[production] |
05:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE |
[production] |
05:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE |
[production] |
04:48 |
<marostegui> |
Depool clouddb1019:3314 (long running alter table) |
[production] |
2021-06-04
§
|
22:08 |
<cwhite@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh4001.wikimedia.org |
[production] |
21:51 |
<cwhite@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host doh4001.wikimedia.org |
[production] |
20:59 |
<bblack> |
repool cp1087 - T278729 |
[production] |
20:11 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE |
[production] |
20:09 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE |
[production] |
19:06 |
<bblack> |
depool cp1087 - T278729 |
[production] |
18:21 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
17:36 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
17:33 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
17:33 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
17:33 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
17:28 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE |
[production] |
17:25 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE |
[production] |
15:25 |
<topranks> |
Adding 1:1 NAT configuration for fran2001 / analytics.codfw.wikimedia.org to pfw3-codfw (backup site) |
[production] |
14:47 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: I434d9cfa29d84f (duration: 00m 56s) |
[production] |
14:46 |
<krinkle@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/extension.json: Iea41ab8599ffae (duration: 00m 56s) |
[production] |
14:44 |
<krinkle@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/includes/: Iea41ab8599ffae (duration: 00m 59s) |
[production] |
14:41 |
<krinkle@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) |
[production] |