1-50 of 10000 results (28ms)
2020-06-18 ยง
22:30 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 56s) [production]
21:14 <volans> start check-homer-diff.service on cumin2001 after merging the fix r/606526 [production]
20:17 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 57s) [production]
19:44 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on group1 wikis - T249261 (duration: 00m 57s) [production]
18:53 <wkandek@cumin1001> conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet [production]
18:35 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
18:32 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:16 <wkandek@cumin1001> conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet [production]
17:14 <wkandek@cumin1001> conftool action : set/pooled=yes; selector: name=mw2339.codfw.wmnet [production]
17:12 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2339.codfw.wmnet [production]
16:51 <maryum> reindex suspended until deployment of code [production]
16:49 <hnowlan> Shut off non-dockerised deployment-prep instance of changeprop [production]
16:15 <maryum> reindexing French wiki in Elasticsearch [production]
15:37 <Reedy> creatd bot_passwords tables on officeiwki and otrs_wikiwiki T254925 T246489 [production]
15:34 <moritzm> installing harfbuzz security updates [production]
15:23 <moritzm> installing Ruby 2.1 security updates [production]
15:15 <moritzm> installing python-django security updates (packaged buster version) [production]
15:04 <moritzm> installing bind updates on jessie (client side tools/libs) [production]
14:19 <marostegui@cumin2001> dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11591 and previous config saved to /var/cache/conftool/dbconfig/20200618-141941-marostegui.json [production]
14:14 <moritzm> failover ganeti master in codfw to ganeti2021 [production]
14:03 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P11590 and previous config saved to /var/cache/conftool/dbconfig/20200618-140352-marostegui.json [production]
14:02 <marostegui@cumin2001> dbctl commit (dc=all): 'Repool db1075', diff saved to https://phabricator.wikimedia.org/P11589 and previous config saved to /var/cache/conftool/dbconfig/20200618-140203-marostegui.json [production]
13:53 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:53 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:52 <akosiaris> restart logstash2005 for applying an increased ganeti migration_downtime of 10k [production]
13:47 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
13:43 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
13:40 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
13:34 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
13:11 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
13:05 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
12:52 <marostegui@cumin2001> dbctl commit (dc=all): 'Depool db1075 for schema change', diff saved to https://phabricator.wikimedia.org/P11586 and previous config saved to /var/cache/conftool/dbconfig/20200618-125216-marostegui.json [production]
12:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove weight from es5 master as es1024 is fully repooled now', diff saved to https://phabricator.wikimedia.org/P11585 and previous config saved to /var/cache/conftool/dbconfig/20200618-124801-marostegui.json [production]
12:23 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:20 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:04 <kormat> reimaging db1077 for final test T251768 [production]
11:51 <jbond@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 01m 00s) [production]
11:06 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
11:00 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
10:34 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:34 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
09:48 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
09:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2076', diff saved to https://phabricator.wikimedia.org/P11583 and previous config saved to /var/cache/conftool/dbconfig/20200618-094001-marostegui.json [production]
09:39 <akosiaris> update wikifeeds to latest chart version in codfw [production]
09:39 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
09:38 <marostegui@cumin2001> dbctl commit (dc=all): 'Repool es2022', diff saved to https://phabricator.wikimedia.org/P11582 and previous config saved to /var/cache/conftool/dbconfig/20200618-093803-marostegui.json [production]
09:38 <akosiaris> uncordon kubernetes20{07..14} and kubernetes10{07..14}. Nodes are now fully put in rotation and ready to receive production traffic [production]
09:34 <marostegui> Deploy schema change on s3 codfw master (this will create lag on codfw) - T250066 [production]
09:31 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]