3301-3350 of 10000 results (33ms)
2020-09-07 §
12:59 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
12:43 <kormat@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) [production]
12:42 <kormat@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
12:29 <marostegui> Upgrade and reboot db2094 and db2095 (sanitarium hosts in codfw) [production]
12:18 <gehel> restarting elasticsearch on elastic2029 (high GC) [production]
12:01 <volans> restart uwsgi on debmonitor1002 to test db reconnection [production]
11:58 <marostegui> Reboot pc1008 for upgrade [production]
11:36 <Urbanecm> EU B&C done [production]
11:30 <urbanecm@deploy1001> Synchronized docroot/noc/index.html: bbfe2ce61014f616d89bc0c21a380c15777b62e3: noc: Remove link to outdated blog (T259978) (duration: 00m 57s) [production]
11:27 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: ff9f1042529bd332effc0fcd18db70f417c2e939: Update help URL (T256623) (duration: 00m 56s) [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7b512d3a27c4c33949389cbbe7823cc534fbff9a: [hewiktionary] Enable wikilove (T262181) (duration: 00m 57s) [production]
11:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 35224f43f1c461d42da5c963bb60d28fbe1992ee: [eswiki] Create an `abusefilter` user group (T262174; 2/2) (duration: 00m 57s) [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/abusefilter.php: 35224f43f1c461d42da5c963bb60d28fbe1992ee: [eswiki] Create an `abusefilter` user group (T262174; 1/2) (duration: 01m 20s) [production]
11:02 <Urbanecm> [urbanecm@mwmaint2001 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hewiktionary wikilove # T262181 [production]
11:01 <marostegui> Reboot pc1007 for upgrade [production]
10:37 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:35 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:00 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:36 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
09:30 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
09:12 <dcausse@deploy1001> Finished deploy [wdqs/wdqs@c96b49e]: deploy wdqs-0.3.47 to wdqs1009 (test server) (duration: 00m 33s) [production]
09:11 <dcausse@deploy1001> Started deploy [wdqs/wdqs@c96b49e]: deploy wdqs-0.3.47 to wdqs1009 (test server) [production]
09:10 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
09:09 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:02 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
08:53 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
08:49 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
08:29 <jayme@deploy2001> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:19 <marostegui> Upgrade and restart pc1010 [production]
08:18 <jayme@deploy2001> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:10 <jayme@deploy2001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
08:03 <marostegui> Compress InnoDB on s8 eqiad master (db1109) - T232446 [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 after MCR schema change', diff saved to https://phabricator.wikimedia.org/P12501 and previous config saved to /var/cache/conftool/dbconfig/20200907-051157-marostegui.json [production]
04:56 <marostegui> Compress InnoDB on s1 eqiad master - this will generate a few day of lag on s1 and labsdb for enwiki T254462 [production]
04:53 <marostegui> Deploy schema change on db1109 (eqiad wikidata master) - T256685 [production]
2020-09-06 §
19:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Decrease db2127's weight a bit', diff saved to https://phabricator.wikimedia.org/P12496 and previous config saved to /var/cache/conftool/dbconfig/20200906-194512-marostegui.json [production]
08:20 <elukey> powercycle mw1360 (mgmt console available, network errors while running anything) [production]
08:04 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw1360.eqiad.wmnet [production]
08:01 <elukey> executed "sudo ipmitool -I lanplus -H mw1360.mgmt.eqiad.wmnet -U root mc reset cold" from cumin (mgmt not available for mw1360) [production]
2020-09-05 §
00:23 <foks> removing 2 files for legal compliance [production]
2020-09-04 §
22:15 <ryankemper> wdqs deploy complete, service is healthy [production]
21:54 <ryankemper> `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'` [production]
21:52 <ryankemper> `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
21:49 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@c7e6b35]: 0.3.47 (duration: 12m 55s) [production]
21:37 <ryankemper> Tests on canary `wdqs1003` passing, beginning full wdqs deploy [production]
21:36 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@c7e6b35]: 0.3.47 [production]
21:31 <ryankemper> `ryankemper@wdqs2002:~$ sudo systemctl restart wdqs-blazegraph` [production]
21:06 <mutante> apt1001 - removed all libnginx-mod* packages except libnginx-mod-http-echo ; sudo apt-get autoremove ; run puppet ; restarted nginx - apt.wikimedia.org switched to nginx-light (T261962) [production]