|
2021-06-07
§
|
| 10:24 |
<moritzm> |
remove now obsolete nginx mods and dependencies on htmldumper1001 T164456 |
[production] |
| 10:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2006.wikimedia.org |
[production] |
| 10:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2006.wikimedia.org |
[production] |
| 10:13 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2005.wikimedia.org |
[production] |
| 10:10 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ldap-replica2005.wikimedia.org |
[production] |
| 10:08 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host theemin.codfw.wmnet |
[production] |
| 10:08 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'db1157 depooling: reimage to buster T283131', diff saved to https://phabricator.wikimedia.org/P16311 and previous config saved to /var/cache/conftool/dbconfig/20210607-100822-kormat.json |
[production] |
| 09:55 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host theemin.codfw.wmnet |
[production] |
| 09:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet |
[production] |
| 09:48 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet |
[production] |
| 09:43 |
<moritzm> |
upgrading bullseye hosts to latest packages in testing |
[production] |
| 09:39 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1002.eqiad.wmnet |
[production] |
| 09:36 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host failoid1002.eqiad.wmnet |
[production] |
| 09:19 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2002.codfw.wmnet |
[production] |
| 09:17 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host failoid2002.codfw.wmnet |
[production] |
| 09:03 |
<moritzm> |
installing imagemagick security updates on stretch |
[production] |
| 06:05 |
<marostegui> |
Upgrade mysql on dbstore1003 T283235 |
[production] |
| 05:57 |
<marostegui> |
Stop dbstore1004 to clone dbstore1007 T283125 |
[production] |
| 05:37 |
<marostegui> |
Depool clouddb1020 (s5, s8) for upgrade |
[production] |
| 05:29 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:27 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE |
[production] |
| 04:48 |
<marostegui> |
Depool clouddb1019:3314 (long running alter table) |
[production] |
|
2021-06-04
§
|
| 22:08 |
<cwhite@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh4001.wikimedia.org |
[production] |
| 21:51 |
<cwhite@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host doh4001.wikimedia.org |
[production] |
| 20:59 |
<bblack> |
repool cp1087 - T278729 |
[production] |
| 20:11 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:09 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:06 |
<bblack> |
depool cp1087 - T278729 |
[production] |
| 18:21 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 17:36 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 17:33 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 17:33 |
<razzi@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 17:33 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 17:28 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:25 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE |
[production] |
| 15:25 |
<topranks> |
Adding 1:1 NAT configuration for fran2001 / analytics.codfw.wikimedia.org to pfw3-codfw (backup site) |
[production] |
| 14:47 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: I434d9cfa29d84f (duration: 00m 56s) |
[production] |
| 14:46 |
<krinkle@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/extension.json: Iea41ab8599ffae (duration: 00m 56s) |
[production] |
| 14:44 |
<krinkle@deploy1002> |
Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/includes/: Iea41ab8599ffae (duration: 00m 59s) |
[production] |
| 14:41 |
<krinkle@deploy1002> |
Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) |
[production] |
| 13:39 |
<Krinkle> |
mwmaint1002: Running purge_parsercache_now.php on pc1008, server 3/4, ref T282761 |
[production] |
| 13:33 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:46 |
<marostegui> |
Upgrade mysql on clouddb1016 T283235 |
[production] |
| 12:27 |
<marostegui> |
Upgrade mysql on clouddb1015 T283235 |
[production] |
| 11:20 |
<jbond> |
upload debmonitor-client_0.3.0-1+deb10u3_all.deb to apt |
[production] |