1401-1450 of 10000 results (25ms)
2020-09-01 §
07:11 <moritzm> installing 4.9.228 kernel on stretch systems (only installing the deb, reboots separately) [production]
07:05 <moritzm> restarting jenkins on releases1002 to pick up Java security updates [production]
06:59 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:57 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:44 <elukey> reimage kafka-jumbo1002 to Buster [production]
06:20 <marostegui> Install query killers on db2137:3314 T243373 [production]
01:17 <chaomodus> updated the pynetbox package to 5.0.7 and uploaded to buster [production]
00:02 <mutante> wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while [production]
00:00 <mutante> restarting wikibugs [production]
2020-08-31 §
23:38 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 (final) (duration: 00m 17s) [production]
23:38 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 (final) [production]
23:37 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox2001 (duration: 01m 12s) [production]
23:36 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox2001 [production]
23:36 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox1001 (duration: 00m 58s) [production]
23:35 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Deploy of 2.8.9 to netbox1001 [production]
23:31 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next pt2 (duration: 00m 05s) [production]
23:31 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next pt2 [production]
23:31 <crusnov@deploy1001> Finished deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next (duration: 00m 57s) [production]
23:30 <crusnov@deploy1001> Started deploy [netbox/deploy@2fc439e]: Test deploy of 2.8.9 to netbox-next [production]
23:09 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Disable (future) mw-reverted tag for all wikis except testwiki (T254074) (duration: 00m 57s) [production]
21:06 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:00 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
20:20 <ryankemper> `sudo systemctl restart elasticsearch_6@production-search-psi-eqiad.service` on `elastic1052.eqiad.wmnet` [production]
18:38 <Urbanecm> Morning B&C done [production]
18:32 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 16197aabc88f098568a04984a20149de3b7fdeaf: Add two domains to wgCopyUploadsDomains for commonswiki (T261562; T261575) (duration: 00m 54s) [production]
18:27 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bb28e9da8057a4c92cd4d564ffd000f320338cda: itwiki: Assign patrol right to autopatrolled instead of autoconfirmed (T261587) (duration: 00m 53s) [production]
18:23 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: a1b0d6e4e7da9bf45ae7381d2c1d9814e6b36498: b609cd53273e922cd8af5507660b9d10c6da09b3: CommonSettings.php: limit new Echos `push-subscription-manager` group to Meta-Wiki (T261625) (duration: 00m 54s) [production]
18:11 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 846c5448f950b4d0d7eedce570e46d74ca62ca38: wgEventStreams: Stream for MEP-iOS pilot (T260382) (duration: 00m 55s) [production]
17:21 <volans> uploaded spicerack_0.0.42 to apt.wikimedia.org buster-wikimedia [production]
15:50 <rzl@cumin1001> conftool action : set/pooled=false; selector: dnsdisc=swift,name=eqiad [production]
15:49 <ejegg> updated payments-wiki from ef7ebd08cb to be81063168 [production]
15:34 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
15:33 <rzl@cumin1001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
15:33 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=99) [production]
15:32 <rzl@cumin1001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
14:58 <ema> Traffic: depool eqiad from user traffic T243316 [production]
14:38 <moritzm> installing rake security updates on stretch [production]
14:33 <rzl@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [production]
14:21 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
14:20 <rzl@cumin1001> Switching services apertium, termbox, search, api-gateway, ores, sessionstore, eventgate-main, graphoid, eventstreams, wikifeeds, wdqs, parsoid, eventgate-logging-external, wdqs-internal, echostore, mathoid, mobileapps, proton, restbase, kartotherian, recommendation-api, eventgate-analytics-external, restbase-async, citoid, schema, cxserver, eventgate-analytics, zotero: eqiad => codfw [production]
14:20 <rzl@cumin1001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
14:18 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
14:13 <rzl@cumin1001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
14:12 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=99) [production]
14:11 <rzl@cumin1001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
13:41 <andrewbogott> dropping many databases from m5, as per T261152 [production]
13:15 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:15 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:07 <marostegui> Failover m3 (phabricator) proxy from dbproxy1016 to dbproxy1020 - T261459 [production]
13:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]