601-650 of 10000 results (58ms)
2019-11-25 §
06:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P9729 and previous config saved to /var/cache/conftool/dbconfig/20191125-061542-marostegui.json [production]
06:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P9728 and previous config saved to /var/cache/conftool/dbconfig/20191125-060728-marostegui.json [production]
06:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P9727 and previous config saved to /var/cache/conftool/dbconfig/20191125-060011-marostegui.json [production]
05:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2125 - crashed T239042', diff saved to https://phabricator.wikimedia.org/P9726 and previous config saved to /var/cache/conftool/dbconfig/20191125-055813-marostegui.json [production]
05:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P9725 and previous config saved to /var/cache/conftool/dbconfig/20191125-055305-marostegui.json [production]
03:13 <vgutierrez> repooling cp3053 - T239041 [production]
03:00 <vgutierrez@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3053.esams.wmnet [production]
02:59 <vgutierrez> depooling & power-cycling cp3053 - T239041 [production]
00:10 <eileen> also speed the repair process-control config revision is c4ad2f5990 [production]
2019-11-24 §
20:54 <eileen> process-control config revision is 371782a667 [production]
15:41 <ariel@deploy1001> Finished deploy [dumps/dumps@bfdea34]: can skip locks for misc dumps (duration: 00m 03s) [production]
15:41 <ariel@deploy1001> Started deploy [dumps/dumps@bfdea34]: can skip locks for misc dumps [production]
15:01 <apergos> rebooting dumpsdata1002 to clear up the other half of the nfs issues [production]
14:24 <apergos> rebooting snapshot1008 to clear up some nfs + kernel issues [production]
2019-11-23 §
18:19 <gehel> repool wdqs1007, catched up on lag - T238229 [production]
14:23 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 55s) [production]
11:56 <_joe_> oblivian@cumin1001:~$ sudo cumin -b2 -s60 A:mw-eqiad 'restart-php7.2-fpm' [production]
11:47 <_joe_> restarting php7.2-fpm on mw1329 [production]
09:49 <XioNoX> downtime all ripe-atlas checks until Monday (most likely an upstream issue/maintenance) [production]
2019-11-22 §
21:55 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T238955 (duration: 00m 53s) [production]
18:02 <shdubsh> restore prometheus services default settings - T238807 [production]
17:52 <_joe_> repooling restbase2018 [production]
17:36 <bblack@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
17:34 <bblack@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:30 <shdubsh> clean tombstones on prometheus1004 - T238807 [production]
17:09 <shdubsh> restart prometheus on prometheus1004 - T238807 [production]
16:22 <shdubsh> clean tombstones on prometheus1003 - T238807 [production]
15:40 <XioNoX> renumber AS17639 sessions in eqsin [production]
15:16 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/Wikibase/repo/: Stop outputting anything in case of 304 responses in Special:EntityData (T238901) (duration: 00m 57s) [production]
14:49 <_joe_> disabling puppet on restbase2018, testing envoy upgrade T238050 [production]
14:48 <_joe_> uploaded envoyproxy 1.12.1 to {buster,stretch} T237235 [production]
13:11 <Amir1> start of foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T238119 T238524 T237375 T238120) [production]
13:06 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/SqlEntityInfoBuilder.php: T238473 (duration: 00m 52s) [production]
12:34 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T221774 - wgWikidataOrgQueryServiceMaxLagFactor 60 RESYNC (duration: 00m 51s) [production]
12:32 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T221774 - wgWikidataOrgQueryServiceMaxLagFactor 60 (duration: 00m 53s) [production]
11:59 <effie> reload php7 on canaries [production]
11:34 <effie> Roll out wikidiff2 1.10.0-1 to canaries - T236963 [production]
11:29 <effie> upload wikidiff2 1.10.0-1 - T236963 [production]
09:59 <ladsgroup@deploy1001> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 10s) [production]
09:56 <ladsgroup@deploy1001> Synchronized langlist: T238105 (duration: 00m 51s) [production]
09:47 <ladsgroup@deploy1001> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 20s) [production]
09:44 <ladsgroup@deploy1001> Synchronized langlist: T238104 T238104 (duration: 00m 52s) [production]
09:28 <ema> pool cp1081 with ATS backend T227432 [production]
09:27 <gehel> depool wdqs1007 to allow to catch up on lag - T238229 [production]
09:23 <reedy@deploy1001> Synchronized php-1.35.0-wmf.5/includes/specials/pagers/ContribsPager.php: Remove live hack of limit for T234450 (duration: 00m 54s) [production]
09:19 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: T234450 (duration: 00m 55s) [production]
09:07 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:05 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:04 <gehel> remove blazegraph 2.1.5-wmf.11 from archiva, broken upload [production]
08:54 <gehel> restarting blazegraph and updater on wdqs1007 [production]