3951-4000 of 10000 results (41ms)
2021-09-14 §
08:47 <moritzm> installing testvm2002 [production]
08:42 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2002.codfw.wmnet [production]
08:28 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2002.codfw.wmnet [production]
08:27 <hashar@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.23 [production]
08:25 <godog> poweroff ms-be2045 and set it as failed in netbox - T290881 [production]
08:24 <hashar> train: applied security patches for 1.37.0-wmf.23 # T281164 [production]
08:05 <godog> wipe non-os partitions from ms-be2045 - T290881 [production]
07:50 <vgutierrez> update acme-chief to version 0.31 on acmechief hosts - T290249 [production]
04:47 <eileen> civicrm revision changed from 1f071f6c6c to e6bf81d99c, config revision is 23eda8ba3a [production]
02:41 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:07 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:07 <James_F> wmf/1.37.0-wmf.23 was branched at ea72c9b690c2159a12beec2f518b61cc499ed521 for T281164 [production]
02:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:04 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
2021-09-13 §
23:54 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:45 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T290759: Undeploy VipsScaler: III – Don't set wmgUseVips, now ignored (duration: 00m 58s) [production]
23:45 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
23:41 <jforrester@deploy1002> Synchronized wmf-config/CommonSettings.php: T290759: Undeploy VipsScaler: II – Don't load regardless of config (duration: 00m 58s) [production]
19:52 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T290759 Undeploy VipsScaler: I – Disable on all wikis (duration: 00m 57s) [production]
19:49 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:47 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:04 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:59 <urbanecm> [urbanecm@mwmaint2002 ~]$ mwscript resetAuthenticationThrottle.php --wiki={cswiki,cswikiversity} --signup --ip=185.47.223.49 # T290809 [production]
18:58 <urbanecm@deploy1002> Synchronized wmf-config/throttle.php: 9db1d1ac938ca053c82fed88c8b6e75f97a52416: Add throttle rule for Czech wiki course (T290809) (duration: 00m 58s) [production]
18:29 <ryankemper> [Cirrus] `eqiad` fully recovered (100% of shards), `codfw` at 99.816%. `codfw` is getting held up by recovery of `enwiki` shards which tend to be quite large [production]
18:25 <razzi> reenable replication on dbstore1007 for T290841 [production]
18:16 <cwhite> apply high log volume from ES mitigations to deprecated inputs [production]
18:13 <razzi> razzi@dbstore1007:~$ sudo systemctl restart mariadb@s3.service for T290841 [production]
18:05 <razzi> sudo systemctl restart mariadb@s2.service [production]
17:48 <ryankemper> [Cirrus] `eqiad` is at 99.13% shards recovered and `codfw` is at 98.83% [production]
17:20 <volans@cumin1001> END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host sretest1002.eqiad.wmnet [production]
17:17 <ryankemper> [Cirrus] `enwiki` searches appear to be working now. `production-search-eqiad` is at 93.5% recovered shards, `production-search-codfw` is at 95.3% recovered [production]
16:57 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1002.eqiad.wmnet [production]
16:18 <legoktm@cumin1001> conftool action : set/pooled=false; selector: name=codfw,dnsdisc=eventgate-main [production]
16:16 <volans@cumin1001> conftool action : set/pooled=yes; selector: name=mw1414.* [production]
16:08 <volans@cumin1001> conftool action : set/pooled=no; selector: name=mw1414.* [production]
16:06 <volans@cumin1001> END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host mw1414.eqiad.wmnet [production]
15:54 <moritzm> filtered mx2001 on the routers for reimage T286911 [production]
15:43 <vgutierrez> update acme-chief to version 0.31 on acmechief-test hosts - T290249 [production]
15:40 <vgutierrez> upload acme-chief 0.31 to apt.wm.o (buster) - T290249 [production]
15:32 <jelto> Traffic: depool codfw from user traffic [production]
15:26 <jelto@cumin2002> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
15:25 <jelto@cumin2002> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
15:25 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host mw1414.eqiad.wmnet [production]
15:20 <Emperor> rebooting ms-be2045 to see if that brings the disk back properly T290881 [production]