5101-5150 of 10000 results (40ms)
2021-07-27 ยง
14:11 <moritzm> installing aspell security updates [production]
14:11 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' . [production]
14:07 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
14:07 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
14:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
14:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
14:00 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
13:59 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:59 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
13:54 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:54 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
13:52 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:42 <volans@deploy1002> Finished deploy [netbox/deploy@660ad14]: Deploy v2.10.4-wmf5 (duration: 02m 29s) [production]
13:40 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:39 <volans@deploy1002> Started deploy [netbox/deploy@660ad14]: Deploy v2.10.4-wmf5 [production]
13:36 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
13:34 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
13:30 <ottomata> deploying eventgate-analytics with native prometheus support. Doing this slowly on canary release first to ensure https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-07-14_eventgate-analytics_latency_spike_caused_MW_app_server_overload is fixed. [production]
13:29 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
12:56 <elukey> created component/iptables185 for buster-wikimedia + imported packages from buster-backports [production]
12:50 <dcausse@deploy1002> Finished deploy [wikimedia/discovery/analytics@346ac10]: (no justification provided) (duration: 06m 13s) [production]
12:43 <dcausse@deploy1002> Started deploy [wikimedia/discovery/analytics@346ac10]: (no justification provided) [production]
11:23 <Lucas_WMDE> EU backport+config window done [production]
11:20 <oblivian@deploy1002> Synchronized debug.json: Config: [[gerrit:708255|Add the experimental kubernetes backend to mwdebug (T283056)]] (duration: 00m 56s) [production]
11:10 <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704456|Add stream configuration for ContentTranslation events (T281982)]] (duration: 00m 58s) [production]
10:39 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1269.eqiad.wmnet [production]
10:16 <jelto> gitlab-ansible playbook on gitlab2001.wikimedia.org END (PASS) [production]
10:11 <mutante> replacing scap proxies: mw1269 with mw1420, mw1285 with mw1306 [production]
10:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 100%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16909 and previous config saved to /var/cache/conftool/dbconfig/20210727-101053-root.json [production]
10:10 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw1269.eqiad.wmnet [production]
10:06 <jelto> running gitlab-ansible playbook on gitlab2001.wikimedia.org [production]
10:05 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1269.eqiad.wmnet [production]
09:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 75%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16908 and previous config saved to /var/cache/conftool/dbconfig/20210727-095549-root.json [production]
09:52 <jynus> reverting query killer parameters on s3 codfw replicas [production]
09:41 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw1285.eqiad.wmnet [production]
09:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 50%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16906 and previous config saved to /var/cache/conftool/dbconfig/20210727-094046-root.json [production]
09:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 25%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16905 and previous config saved to /var/cache/conftool/dbconfig/20210727-092542-root.json [production]
09:13 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw1285.eqiad.wmnet [production]
09:12 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw1285.eqiad.wmnet [production]
09:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 15%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16904 and previous config saved to /var/cache/conftool/dbconfig/20210727-091038-root.json [production]
09:04 <_joe_> restarting pybal on lvs2009 to pick up the new api depool threshold [production]
08:57 <_joe_> repooling mw225[12] for apis [production]
08:56 <_joe_> restart pybal on lvs2010 to pick up the depool threshold change [production]
08:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 10%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16902 and previous config saved to /var/cache/conftool/dbconfig/20210727-085535-root.json [production]
08:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db2147 (re)pooling @ 5%: After mariadb restart and upgraed', diff saved to https://phabricator.wikimedia.org/P16901 and previous config saved to /var/cache/conftool/dbconfig/20210727-084031-root.json [production]
08:36 <jynus> reenabled puppet on mwmaint1002 [production]
08:29 <volans@deploy1002> Finished deploy [netbox/deploy@660ad14]: Test v2.10.4-wmf5 on netbox-next (duration: 01m 01s) [production]
08:28 <volans@deploy1002> Started deploy [netbox/deploy@660ad14]: Test v2.10.4-wmf5 on netbox-next [production]
08:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2147 to restart mysql', diff saved to https://phabricator.wikimedia.org/P16900 and previous config saved to /var/cache/conftool/dbconfig/20210727-082820-marostegui.json [production]
07:52 <jynus> disabling puppet on mwmaint1002 [production]