3851-3900 of 10000 results (35ms)
2021-02-15 §
15:34 <jmm@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on bast3004.wikimedia.org with reason: REIMAGE [production]
15:33 <moritzm> installing linux-4.19 update for Stretch on servers which have it installed (no reboots, just updating the kernels) [production]
15:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host schema1003.eqiad.wmnet [production]
15:17 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host schema1003.eqiad.wmnet [production]
15:09 <moritzm> reimaging bast3004 to buster [production]
15:04 <godog> upgrade grafana to 7.4.1 on grafana1002 - T263747 [production]
14:17 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 00905c4a7e4bb69f39e52e1c4d4d6168006b0e7b: Add *.president.az to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T274789) (duration: 01m 09s) [production]
14:08 <godog> swift eqiad-prod: add weight back to sdg on ms-be1054 - T273582 [production]
13:57 <moritzm> installing libonig security update for stretch [production]
13:53 <gehel@cumin2001> START - Cookbook sre.wdqs.data-reload [production]
13:38 <moritzm> installing subversion security updates [production]
13:33 <marostegui> Stop MySQL on db1093 - T273955 [production]
13:19 <kharlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
13:06 <godog> swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 [production]
13:05 <Lucas_WMDE> notice: stashbot had issues between 8:19 and 12:50, see for https://wm-bot.wmflabs.org/browser/index.php?start=02%2F15%2F2021&end=02%2F15%2F2021&display=%23wikimedia-operations for missed !log messages [production]
13:03 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on bast4002.wikimedia.org with reason: REIMAGE [production]
13:02 <kharlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
13:02 <kharlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
13:01 <jmm@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on bast4002.wikimedia.org with reason: REIMAGE [production]
12:58 <kharlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . [production]
12:58 <kharlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . [production]
08:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 4%: Slowly pool db1162', diff saved to https://phabricator.wikimedia.org/P14343 and previous config saved to /var/cache/conftool/dbconfig/20210215-080435-root.json [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1162 (re)pooling @ 3%: Slowly pool db1162', diff saved to https://phabricator.wikimedia.org/P14342 and previous config saved to /var/cache/conftool/dbconfig/20210215-074932-root.json [production]
07:42 <elukey@cumin1001> START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes - elukey@cumin1001 [production]
07:33 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1010.eqiad.wmnet [production]
07:28 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1010.eqiad.wmnet [production]
07:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1009.eqiad.wmnet [production]
07:24 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1009.eqiad.wmnet [production]
07:22 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1008.eqiad.wmnet [production]
07:20 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1008.eqiad.wmnet [production]
07:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-tool1007.eqiad.wmnet [production]
07:14 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-tool1007.eqiad.wmnet [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14341 and previous config saved to /var/cache/conftool/dbconfig/20210215-070206-marostegui.json [production]
06:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1162 with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P14340 and previous config saved to /var/cache/conftool/dbconfig/20210215-064628-marostegui.json [production]
06:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1162 to dbctl - depooled T258361', diff saved to https://phabricator.wikimedia.org/P14339 and previous config saved to /var/cache/conftool/dbconfig/20210215-064001-marostegui.json [production]
2021-02-14 §
13:13 <akosiaris> sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service' [production]
13:10 <_joe_> restarted varnish-fe on cp5004 [production]
13:09 <akosiaris> restart varnish-fe on cp5001 [production]
09:27 <joal@deploy1001> Finished deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] (duration: 00m 06s) [production]
09:27 <joal@deploy1001> Started deploy [analytics/refinery@dd5f947] (thin): Hotfix analytics deployment - THIN [analytics/refinery@dd5f947] [production]
09:27 <joal@deploy1001> Finished deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] (duration: 12m 52s) [production]
09:14 <joal@deploy1001> Started deploy [analytics/refinery@dd5f947]: Hotfix analytics deployment [analytics/refinery@dd5f947] [production]
2021-02-13 §
03:23 <ryankemper> Depooled `wdqs1006` to catch up on lag [production]
03:23 <ryankemper> Restarted blazegraph on `wdqs1006` [production]
01:30 <crusnov@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host mwdebug1002.eqiad.wmnet [production]
01:00 <crusnov@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:55 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet [production]
00:49 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet [production]
00:38 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host mwdebug1002.eqiad.wmnet [production]
00:31 <legoktm@cumin1001> conftool action : set/pooled=yes; selector: name=mw1281.eqiad.wmnet [production]