5301-5350 of 10000 results (27ms)
2021-07-28 §
07:20 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on 40 hosts with reason: T287559 [production]
07:20 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 6 hosts with reason: T287559 [production]
07:20 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on 6 hosts with reason: T287559 [production]
07:07 <godog> remove cloud*/syslog.log from centrallog2001 - T287559 [production]
07:06 <godog> remove node_pinger.prom from node-pinger hosts [production]
06:42 <godog> remove obsolete user.log.manual-rotation from centrallog1001 to free disk space [production]
02:43 <TimStarling> on mwmaint2002 fixing T286273 broken files using eval.php [production]
2021-07-27 §
23:53 <thcipriani@deploy1002> Synchronized php-1.37.0-wmf.16/skins/Vector: Backport: [[gerrit:708220|Restore print, links, table and message box styles (T278896)]] (duration: 01m 07s) [production]
23:15 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:708152|Enable user links on office + test wikis (T287391)]] (duration: 02m 00s) [production]
20:44 <ryankemper> [WDQS] Returning `wdqs` dns discovery to the expected status of `(eqiad, codfw) = (depooled, pooled)`: `sudo confctl --object-type discovery select 'dnsdisc=wdqs,name=eqiad' set/pooled=false` [production]
20:44 <legoktm> legoktm@wtp1025:~$ sudo systemctl restart php7.2-fpm # restart php-fpm, opcache hit ratio was warning [production]
20:43 <ryankemper@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=eqiad [production]
20:35 <twentyafterfour@deploy1002> Pruned MediaWiki: 1.37.0-wmf.14 (duration: 03m 12s) [production]
20:25 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.16 [production]
20:02 <twentyafterfour@deploy1002> Finished scap: testwikis wikis to 1.37.0-wmf.16 (duration: 36m 06s) [production]
19:26 <twentyafterfour@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.16 [production]
18:49 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate EchoMail and EchoInteraction to EventGate - T287210 (duration: 02m 28s) [production]
18:17 <twentyafterfour> MediaWiki branch `1.37.0-wmf.16` prepped and patched in preparation for the upcoming deployment window. [production]
17:50 <legoktm@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
17:47 <legoktm@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
17:28 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2022.codfw.wmnet [production]
17:18 <herron@cumin1001> START - Cookbook sre.hosts.decommission for hosts logstash2022.codfw.wmnet [production]
17:17 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2021.codfw.wmnet [production]
16:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1003.eqiad.wmnet [production]
16:49 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1003.eqiad.wmnet [production]
16:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1004.eqiad.wmnet [production]
16:47 <herron@cumin1001> START - Cookbook sre.hosts.decommission for hosts logstash2021.codfw.wmnet [production]
16:44 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2020.codfw.wmnet [production]
16:43 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1004.eqiad.wmnet [production]
16:42 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1002.eqiad.wmnet [production]
16:37 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1002.eqiad.wmnet [production]
16:34 <herron@cumin1001> START - Cookbook sre.hosts.decommission for hosts logstash2020.codfw.wmnet [production]
16:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1001.eqiad.wmnet [production]
16:21 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve1001.eqiad.wmnet [production]
16:14 <dcausse@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . [production]
15:42 <elukey> add disk_template drbd back to ml-serve-ctrl100[12] vms after performance testing - T287238 [production]
15:22 <dcausse> cirrus: reindexing 823 wikis in elastic@[eqiad, codfw and cloudelastic] to apply new mapping (weighted_tags) T147505 [production]
15:22 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2004.codfw.wmnet [production]
15:17 <mmandere> pool lvs1014.eqiad.wmnet - T286061 [production]
15:16 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve2004.codfw.wmnet [production]
15:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2003.codfw.wmnet [production]
15:11 <marostegui> Move m1-master from dbproxy1012 to dbproxy1014 T286061 [production]
15:11 <mmandere> pool authdns1001.wikimedia.org - T286061 [production]
15:10 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve2003.codfw.wmnet [production]
15:09 <mmandere> pool cp10[79-82].eqiad.wmnet - T286061 [production]
15:05 <jmm@puppetmaster1001> conftool action : set/pooled=yes; selector: name=ldap-replica1003.wikimedia.org [production]
14:57 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2002.codfw.wmnet [production]
14:55 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs1014.eqiad.wmnet with reason: Eqiad row B maintenance [production]
14:55 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs1014.eqiad.wmnet with reason: Eqiad row B maintenance [production]
14:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1162 T287230', diff saved to https://phabricator.wikimedia.org/P16917 and previous config saved to /var/cache/conftool/dbconfig/20210727-145352-marostegui.json [production]