4001-4050 of 10000 results (47ms)
2021-06-07 §
10:34 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1157.eqiad.wmnet with reason: REIMAGE [production]
10:32 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1003.wikimedia.org [production]
10:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica1003.wikimedia.org [production]
10:28 <kormat> reimaging db1157 T283131 [production]
10:24 <moritzm> remove now obsolete nginx mods and dependencies on htmldumper1001 T164456 [production]
10:21 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2006.wikimedia.org [production]
10:19 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica2006.wikimedia.org [production]
10:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2005.wikimedia.org [production]
10:10 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ldap-replica2005.wikimedia.org [production]
10:08 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host theemin.codfw.wmnet [production]
10:08 <kormat@cumin1001> dbctl commit (dc=all): 'db1157 depooling: reimage to buster T283131', diff saved to https://phabricator.wikimedia.org/P16311 and previous config saved to /var/cache/conftool/dbconfig/20210607-100822-kormat.json [production]
09:55 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host theemin.codfw.wmnet [production]
09:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet [production]
09:48 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1002.eqiad.wmnet [production]
09:43 <moritzm> upgrading bullseye hosts to latest packages in testing [production]
09:39 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1002.eqiad.wmnet [production]
09:36 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid1002.eqiad.wmnet [production]
09:19 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2002.codfw.wmnet [production]
09:17 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid2002.codfw.wmnet [production]
09:03 <moritzm> installing imagemagick security updates on stretch [production]
06:05 <marostegui> Upgrade mysql on dbstore1003 T283235 [production]
05:57 <marostegui> Stop dbstore1004 to clone dbstore1007 T283125 [production]
05:37 <marostegui> Depool clouddb1020 (s5, s8) for upgrade [production]
05:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE [production]
05:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE [production]
04:48 <marostegui> Depool clouddb1019:3314 (long running alter table) [production]
2021-06-05 §
16:16 <Amir1> deleting all private archives of mm2. All are inaccessible now (T282303) [production]
15:21 <Amir1> delete mbox files of group D and E in mm2 (T282303) [production]
14:35 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:21 <mutante> backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host) [production]
00:18 <mutante> backup1001 systemctl reload bacula-dir fails [production]
2021-06-04 §
22:08 <cwhite@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh4001.wikimedia.org [production]
21:51 <cwhite@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh4001.wikimedia.org [production]
20:59 <bblack> repool cp1087 - T278729 [production]
20:11 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
20:09 <bblack@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
19:06 <bblack> depool cp1087 - T278729 [production]
18:21 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
17:36 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:33 <razzi@cumin1001> END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
17:25 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
15:25 <topranks> Adding 1:1 NAT configuration for fran2001 / analytics.codfw.wikimedia.org to pfw3-codfw (backup site) [production]
14:47 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: I434d9cfa29d84f (duration: 00m 56s) [production]
14:46 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/extension.json: Iea41ab8599ffae (duration: 00m 56s) [production]
14:44 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/includes/: Iea41ab8599ffae (duration: 00m 59s) [production]
14:41 <krinkle@deploy1002> Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
13:39 <Krinkle> mwmaint1002: Running purge_parsercache_now.php on pc1008, server 3/4, ref T282761 [production]