3201-3250 of 10000 results (46ms)
2021-06-07 §
09:03 <moritzm> installing imagemagick security updates on stretch [production]
06:05 <marostegui> Upgrade mysql on dbstore1003 T283235 [production]
05:57 <marostegui> Stop dbstore1004 to clone dbstore1007 T283125 [production]
05:37 <marostegui> Depool clouddb1020 (s5, s8) for upgrade [production]
05:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE [production]
05:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: REIMAGE [production]
04:48 <marostegui> Depool clouddb1019:3314 (long running alter table) [production]
2021-06-05 §
16:16 <Amir1> deleting all private archives of mm2. All are inaccessible now (T282303) [production]
15:21 <Amir1> delete mbox files of group D and E in mm2 (T282303) [production]
14:35 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:21 <mutante> backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host) [production]
00:18 <mutante> backup1001 systemctl reload bacula-dir fails [production]
2021-06-04 §
22:08 <cwhite@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh4001.wikimedia.org [production]
21:51 <cwhite@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh4001.wikimedia.org [production]
20:59 <bblack> repool cp1087 - T278729 [production]
20:11 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
20:09 <bblack@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp1087.eqiad.wmnet with reason: REIMAGE [production]
19:06 <bblack> depool cp1087 - T278729 [production]
18:21 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
17:36 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:33 <razzi@cumin1001> END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) [production]
17:33 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart [production]
17:28 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
17:25 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE [production]
15:25 <topranks> Adding 1:1 NAT configuration for fran2001 / analytics.codfw.wikimedia.org to pfw3-codfw (backup site) [production]
14:47 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: I434d9cfa29d84f (duration: 00m 56s) [production]
14:46 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/extension.json: Iea41ab8599ffae (duration: 00m 56s) [production]
14:44 <krinkle@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/DiscussionTools/includes/: Iea41ab8599ffae (duration: 00m 59s) [production]
14:41 <krinkle@deploy1002> Scap failed!: 9/9 canaries failed their endpoint checks(https://en.wikipedia.org) [production]
13:39 <Krinkle> mwmaint1002: Running purge_parsercache_now.php on pc1008, server 3/4, ref T282761 [production]
13:33 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
12:46 <marostegui> Upgrade mysql on clouddb1016 T283235 [production]
12:27 <marostegui> Upgrade mysql on clouddb1015 T283235 [production]
11:20 <jbond> upload debmonitor-client_0.3.0-1+deb10u3_all.deb to apt [production]
10:59 <topranks> Running homer for Gerrit 698162: Set up BGP peering to doh5001 in eqsin, triggering DoH /24 announcement there. [production]
09:47 <ema> pool cp1087 T278729 [production]
09:37 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1003.eqiad.wmnet [production]
09:33 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people1003.eqiad.wmnet [production]
09:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2002.codfw.wmnet [production]
09:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host people2002.codfw.wmnet [production]
09:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16304 and previous config saved to /var/cache/conftool/dbconfig/20210604-091742-root.json [production]
09:06 <ema> reboot cp1087 T278729 [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 75%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16303 and previous config saved to /var/cache/conftool/dbconfig/20210604-090239-root.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 50%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16302 and previous config saved to /var/cache/conftool/dbconfig/20210604-084735-root.json [production]
08:33 <marostegui> Upgrade db1110 T283235 [production]
08:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 25%: Repool db1110', diff saved to https://phabricator.wikimedia.org/P16301 and previous config saved to /var/cache/conftool/dbconfig/20210604-083232-root.json [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110', diff saved to https://phabricator.wikimedia.org/P16300 and previous config saved to /var/cache/conftool/dbconfig/20210604-082956-marostegui.json [production]
08:20 <godog> upgrade karma to 0.86-1 [production]
07:38 <jynus> stop and upgrade db1150 T283235 [production]