5251-5300 of 10000 results (80ms)
2019-11-12 §
06:57 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
06:44 <marostegui> Change triggers on s5 db2094 - T234704 [production]
06:40 <marostegui> Deploy schema change on s5 codfw with replication, this will generate lag on s5 codfw T233135 T234066 [production]
06:21 <marostegui> Compress db2087:3316, db2087:3317 T235599 [production]
06:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2087:3316, db2087:3317 for compression - T235599', diff saved to https://phabricator.wikimedia.org/P9588 and previous config saved to /var/cache/conftool/dbconfig/20191112-061959-marostegui.json [production]
03:41 <vgutierrez> restart wdqs-blazegraph on wdqs1004 [production]
2019-11-11 §
22:51 <ema@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3065.esams.wmnet [production]
22:49 <ema> power-cycle cp3065, currently down [production]
19:36 <XioNoX> disable ALGs on mr1-esams [production]
18:20 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@222b1c2]: New WDQS build - 0.3.6-SNAPSHOT (duration: 00m 57s) [production]
18:19 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@222b1c2]: New WDQS build - 0.3.6-SNAPSHOT [production]
18:16 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@222b1c2]: New WDQS build - 0.3.6-SNAPSHOT (duration: 15m 14s) [production]
18:01 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@222b1c2]: New WDQS build - 0.3.6-SNAPSHOT [production]
17:53 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) [production]
17:44 <jeh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:41 <jeh@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:44 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
15:42 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
15:30 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
14:26 <ema> pool cp3050 with ATS backend T227432 [production]
13:50 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:48 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:25 <ema> depool cp3050 and reimage as text_ats T227432 [production]
12:59 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
12:46 <effie> Upgrade to 7.2.24-1 mwdebug[2001-2002].codfw.wmnet,mwmaint2001.codfw.wmnet,deploy2001.codfw.wmnet - T237239 [production]
12:31 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@2cb2dde]: Deploy updates on wdqs1010 (duration: 00m 28s) [production]
12:30 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@2cb2dde]: Deploy updates on wdqs1010 [production]
12:28 <effie> Upgrade mw2* to 7.2.24-1 with elegance and restart php-fpm - T237239 [production]
12:21 <effie> Upgrade mw2* to 7.2.24-1 with elegance and restart php-fpm - T231881 [production]
11:55 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
10:52 <hoo> Updated the Wikidata property suggester with data from the 2019-11-04 JSON dump and applied the T132839 workarounds [production]
10:48 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
10:47 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
10:45 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
10:32 <vgutierrez> restarting ats-tls on cp1088 [production]
10:21 <jynus> upgrade mariadb on db2102 [production]
10:16 <ema> repool cp4027 after successful X-Wikimedia-Debug testing P9585 T237687 [production]
10:12 <jynus> manually run full backup of labtestpuppetmaster2001 T235819 [production]
09:41 <ema> test x-wikimedia-debug-routing.lua on cp4027 (depooled) T237687 [production]
09:09 <volker-e@deploy1001> Finished deploy [design/style-guide@0ea65f2]: Deploy design/style-guide: (duration: 00m 07s) [production]
09:09 <volker-e@deploy1001> Started deploy [design/style-guide@0ea65f2]: Deploy design/style-guide: [production]
08:28 <marostegui> Stop MySQL on db2048 before decommissioning - T237913 [production]
08:28 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2048 from config T237913 (duration: 00m 51s) [production]
08:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2048 from config T237913 (duration: 00m 54s) [production]
08:21 <marostegui> Remove db2048 from tendril and zarcillo T237913 [production]
06:56 <elukey> delete /etc/logrotate.d/wdqs-reload-categories from wdqs* as attempt to reduce cronspam [production]
06:44 <marostegui> Delete globalblocks table from napwikisource T230055 [production]
05:27 <vgutierrez> Switch from nginx to ats-tls on cp3058 - T231627 [production]
2019-11-09 §
20:25 <reedy@deploy1001> Synchronized langlist-labs: T237823 (duration: 00m 54s) [production]
02:39 <volker-e@deploy1001> Finished deploy [design/style-guide@d2bfc09]: Deploy design/style-guide: (duration: 00m 07s) [production]