4001-4050 of 10000 results (49ms)
2021-05-11 §
05:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1121.eqiad.wmnet with reason: REIMAGE [production]
05:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1121.eqiad.wmnet with reason: REIMAGE [production]
05:11 <marostegui> Reimage db1121 to buster, this will generate lag on s4 (commonswiki) on wikireplicas T280492 [production]
05:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 - going to be reimaged to buster T280492', diff saved to https://phabricator.wikimedia.org/P15895 and previous config saved to /var/cache/conftool/dbconfig/20210511-051102-marostegui.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1182', diff saved to https://phabricator.wikimedia.org/P15894 and previous config saved to /var/cache/conftool/dbconfig/20210511-050816-marostegui.json [production]
00:01 <wm-bot> <bd808> Restart to deploy ba7998f (T243394) [tools.jouncebot]
2021-05-10 §
23:38 <urbanecm@deploy1002> Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s) [production]
23:32 <urbanecm@deploy1002> Synchronized wmf-config/extension-list: ba8b786c7f3a290f0747a6859fd07502eb83108f: NO-OP: Enable ChessBrowser on beta (T244075) (duration: 00m 57s) [production]
23:12 <urbanecm@deploy1002> Synchronized wmf-config/logos.php: dd6fa6504350a90c9f14c218bc972558791f0a6d: Use ptwiki 20th anniversary logos (T281925) (duration: 00m 59s) [production]
23:08 <wm-bot> <bd808> Restart to revert 560a22a (T243394) [tools.jouncebot]
23:08 <urbanecm@deploy1002> Synchronized static/images/project-logos/: f2a76b1a6eb55749395e67d74c74a7fc5df52f1b: Add ptwiki 20th anniversary logos (T281925) (duration: 00m 58s) [production]
22:58 <bstorm> cleared error state on a grid queue [tools]
22:58 <bstorm> setting `profile::wmcs::kubeadm::docker_vol: false` on ingress nodes [tools]
22:28 <eileen> civicrm revision changed from 2052d79248 to 38ac15233f, config revision is 47f21e4568 [production]
21:59 <dancy@deploy1002> Synchronized php-1.37.0-wmf.4/extensions/MediaSearch/MediaSearch.i18n.php: Backport: [[gerrit:688295|Manually include I18nUtils class (T282206)]] (duration: 00m 56s) [production]
21:45 <dancy@deploy1002> Synchronized php-1.37.0-wmf.3/extensions/MediaSearch/MediaSearch.i18n.php: Backport: [[gerrit:688294|Manually include I18nUtils class (T282206)]] (duration: 01m 01s) [production]
21:39 <legoktm> nvm, downgraded flufl.bounce on lists1001 [production]
21:26 <legoktm> upgraded flufl.bounce on lists1001 and restarted mailman3 T282348 [production]
20:44 <andrew@deploy1002> Finished deploy [horizon/deploy@2604d7b]: more deployment fixes (duration: 03m 44s) [production]
20:41 <andrew@deploy1002> Started deploy [horizon/deploy@2604d7b]: more deployment fixes [production]
20:40 <andrew@deploy1002> Finished deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 (duration: 02m 07s) [production]
20:37 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
20:35 <andrew@deploy1002> Finished deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 (duration: 01m 55s) [production]
20:33 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
20:31 <andrew@deploy1002> Finished deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 (duration: 01m 21s) [production]
20:29 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
20:29 <andrew@deploy1002> deploy aborted: update horizon to fix T282489 (duration: 00m 36s) [production]
20:29 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
20:29 <andrew@deploy1002> deploy aborted: update horizon to fix T282489 (duration: 00m 15s) [production]
20:28 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
20:25 <andrew@deploy1002> Finished deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 (duration: 04m 10s) [production]
20:21 <andrew@deploy1002> Started deploy [horizon/deploy@6dc83bd]: update horizon to fix T282489 [production]
19:42 <bstorm> setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes [toolsbeta]
18:34 <jforrester@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:632598|loginwiki: Allow users to mark Notifications as read (T264834)]] (duration: 00m 57s) [production]
18:25 <jforrester@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:677325|Disable LocalisationUpdate, part I (T158360)]] (duration: 00m 58s) [production]
18:24 <XioNoX> add cmooney to all network devices [production]
18:18 <jforrester@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:679940|[wikitech] Enable VE desktop section edit links (T280291)]] (duration: 00m 57s) [production]
18:13 <jforrester@deploy1002> Synchronized wmf-config: Config: [[gerrit:657697|wgAbuseFilterAflFilterMigrationStage: Stop setting, COMPAT_NEW is default (T269712)]] (duration: 00m 57s) [production]
18:10 <jforrester@deploy1002> Synchronized wmf-config/flaggedrevs.php: Config: [[gerrit:673306|FlaggedRevs: Stop setting wgFlaggedRevsWhitelist, now ignored]] (duration: 00m 57s) [production]
18:08 <legoktm> imported new mailman3, flufl.bounce packages to apt.wm.o [production]
17:59 <wm-bot> <lucaswerkmeister> deployed 3c78f9f4b5 (remove dead code) [tools.ranker]
17:43 <Majavah> testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta T264221 [toolsbeta]
16:38 <wm-bot> <lucaswerkmeister> deployed 248527544d (l10n updates) [tools.lexeme-forms]
16:27 <jbond42> rm -r /var/lib/routinator/repository and rebuilding repo [production]
16:23 <herron@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:688281|arclamp/xenon: point all hosts to eqiad (mwlog1002) (T224565)]] (duration: 00m 59s) [production]
15:29 <Majavah> restarting, unresponsive [tools.fourohfour]
15:22 <Majavah> change k8s.svc.tools.eqiad1.wikimedia.cloud. to point to the tools-k8s-haproxy-keepalived-vip address 172.16.6.113 (T252239) [tools]
15:20 <elukey> restart rsyslog on rpki1001 [production]
15:06 <Majavah> carefully rolling out keepalived to tools-k8s-haproxy-[3-4] while making sure [1-2] do not have changes [tools]
15:03 <Majavah> clear all error states caused by overloaded exec nodes [tools]