301-350 of 10000 results (34ms)
2021-02-16 §
09:40 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'staging' . [production]
09:37 <marostegui@cumin1001> dbctl commit (dc=all): 'db1092 (re)pooling @ 20%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14368 and previous config saved to /var/cache/conftool/dbconfig/20210216-093716-root.json [production]
09:28 <marostegui> Failover m2-master from dbproxy1013 to dbproxy1015 [production]
09:22 <marostegui@cumin1001> dbctl commit (dc=all): 'db1092 (re)pooling @ 10%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14367 and previous config saved to /var/cache/conftool/dbconfig/20210216-092213-root.json [production]
08:37 <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 [production]
08:30 <marostegui> Deploy schema change on s6 codfw - T273359 [production]
07:40 <dcausse> restarting blazegraph on wdqs1013 [production]
07:32 <elukey> restart hadoop daemons on an-worker1099 after reconfiguring a new disk [analytics]
07:27 <marostegui> Reboot dbproxy1021 for kernel upgrade [production]
07:21 <marostegui> Reboot dbproxy1012, 1015, 1016, 1017 for kernel upgrade [production]
07:18 <marostegui> Reboot dbproxy2* for kernel upgrade [production]
06:58 <elukey> restart hdfs/yarn daemons on an-worker1097 to exclude a failed disk [analytics]
06:49 <marostegui> Reboot pc2010 pc2009 pc2008 pc2007 for kernel upgrade [production]
06:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 to clone db1172 T258361', diff saved to https://phabricator.wikimedia.org/P14365 and previous config saved to /var/cache/conftool/dbconfig/20210216-064602-marostegui.json [production]
06:43 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
06:37 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1093 from dbctl T273955', diff saved to https://phabricator.wikimedia.org/P14364 and previous config saved to /var/cache/conftool/dbconfig/20210216-063250-marostegui.json [production]
04:18 <James_F> Manually updated doc1001 via https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation#Updating_the_doc.wikimedia.org_site [releng]
04:17 <jforrester@deploy1001> Finished deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. (duration: 00m 17s) [production]
04:17 <jforrester@deploy1001> Started deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. [production]
04:00 <James_F> Zuul: Add Tim Abdullin from S&F to CI allow list [releng]
2021-02-15 §
21:33 <eileen> civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1 [production]
20:38 <mforns> running hdfs fsck to troubleshoot corrupt blocks [analytics]
19:24 <Reedy> stash'd patch saved to quarry-web-01.quarry.eqiad1.wikimedia.cloud:/root/T274815.patch T274815 [quarry]
19:22 <Reedy> T274815 filed with the login failure traceback [quarry]
19:21 <Reedy> re-enabled puppet on quarry-web-01.quarry.eqiad1.wikimedia.cloud as it had been disabled for a week [quarry]
19:20 <Reedy> `git stash` framawiki changes as it was breaking login [quarry]
19:18 <Reedy> `git stash` framawiki changes as it was breaking login [tools.quarry]
17:28 <elukey> restart hdfs namenodes on the main cluster to pick up new racking changes (worker nodes from the backup cluster) [analytics]
16:46 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage1002.eqiad.wmnet [production]
16:39 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestage1002.eqiad.wmnet [production]
16:33 <volans> restarted netbox on netbox1001 [production]
16:32 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage1001.eqiad.wmnet [production]
16:27 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestage1001.eqiad.wmnet [production]
16:26 <jayme> rolled back linkrecommendation helm releases to the most recent revision running chart verion linkrecommendation-0.0.4 on clusters codfw and eqiad (cc: kostajh) [production]
16:25 <arturo> [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers (T272963) [admin]
16:22 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mwdebug1002.eqiad.wmnet [production]
16:18 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet [production]
16:14 <hoo> Updated the Wikidata property suggester with data from the 2021-02-01 JSON dump (with pre-applied T132839 workarounds) [production]
16:12 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestage2002.codfw.wmnet [production]
16:12 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2001.codfw.wmnet [production]
16:09 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2003-dev.codfw.wmnet [production]
16:07 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubestage2001.codfw.wmnet [production]
16:05 <aborrero@cumin2001> START - Cookbook sre.hosts.reboot-single for host cloudnet2003-dev.codfw.wmnet [production]
15:58 <hashar> Successfully published image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.1 # T209953 [releng]
15:58 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host schema2004.codfw.wmnet [production]
15:53 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host schema2004.codfw.wmnet [production]
15:51 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host schema2003.codfw.wmnet [production]
15:48 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host schema2003.codfw.wmnet [production]
15:48 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]