651-700 of 10000 results (21ms)
2021-06-23 §
05:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Start repooling db1100', diff saved to https://phabricator.wikimedia.org/P16700 and previous config saved to /var/cache/conftool/dbconfig/20210623-054252-marostegui.json [production]
04:52 <marostegui@cumin1001> dbctl commit (dc=all): 'db1100 (re)pooling @ 25%: Repool db1100 after upgrade', diff saved to https://phabricator.wikimedia.org/P16699 and previous config saved to /var/cache/conftool/dbconfig/20210623-045217-root.json [production]
01:04 <eileen> process-control config revision is 6a88618c3e [production]
00:50 <eileen> civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033 [production]
00:40 <legoktm> uploaded new versions of flufl.bounce_4.0-1_amd64.changes hyperkitty_1.3.4-2~bpo10+4_amd64.changes mailman3_3.3.3-1~bpo10+5_amd64.changes mailman-hyperkitty_1.1.0-10~bpo10+1_amd64.changes to apt1001 [production]
00:02 <Trey314159> reindexing Portuguese wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T284185) [production]
2021-06-22 §
23:23 <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Enable canary events for search event streams (duration: 01m 05s) [production]
23:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 7865f27430f8eea2975d7154f6009a9206fc75d6: Add unwatchedpages to rollbacker on frwiki (T285334) (duration: 01m 06s) [production]
23:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 9a594f0ce249e2b4752ea2b8d7c4258bf14ad86a: Enable Growth features in dark mode at nlwiki (T285254; 3/3) (duration: 01m 07s) [production]
23:05 <urbanecm@deploy1002> Synchronized wmf-config/config/nlwiki.yaml: 9a594f0ce249e2b4752ea2b8d7c4258bf14ad86a: Enable Growth features in dark mode at nlwiki (T285254; 2/3) (duration: 01m 05s) [production]
23:04 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 9a594f0ce249e2b4752ea2b8d7c4258bf14ad86a: Enable Growth features in dark mode at nlwiki (T285254; 1/3) (duration: 01m 37s) [production]
22:42 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript recountCategories.php --wiki=zhwiki --mode=subcats # T170737 [production]
22:41 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript recountCategories.php --wiki=zhwiki --mode=pages # T170737 [production]
22:38 <urbanecm> mwscript recountCategories.php --wiki=eowiktionary --mode={pages,subcats,files} (T170737) [production]
21:05 <eileen> civicrm revision changed from 629bd3b7b7 to c745d4f075, config revision is 4ab72c1033 [production]
21:05 <ejegg> updated payments-wiki from 7be0534b91 to 42cfbe832d [production]
20:46 <brennen> gitlab1001: running ansible to deploy [[gerrit:699819|CAS: stop marking users as external]] (T274461) [production]
20:23 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-web1001.eqiad.wmnet with reason: REIMAGE [production]
20:19 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-web1001.eqiad.wmnet with reason: REIMAGE [production]
20:12 <Trey314159> reindexing Portuguese wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T284185) [production]
20:12 <Trey314159> reindexing Dutch wikis on elastic@eqiad, elastic@codfw, and cloudelastic complete (T284185) [production]
19:58 <brennen> gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/699812 (T264231) [production]
19:26 <legoktm> set mediawiki-l message acceptance to discard non-member posts instead of reject [production]
19:09 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.11 [production]
19:06 <dduvall> preparing to promote wmf.11 group0 (T281152) cc'ing risking patch contacts Amir1, Krinkle, DannyS712 [production]
19:01 <dduvall@deploy1002> Pruned MediaWiki: 1.37.0-wmf.6 (duration: 03m 35s) [production]
18:46 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@75d35b4]: revert expect eventgate canary events in all dcs (duration: 04m 23s) [production]
18:42 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@75d35b4]: revert expect eventgate canary events in all dcs [production]
18:31 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thumbor1006.eqiad.wmnet with reason: REIMAGE [production]
18:30 <awight@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/VisualEditor: Backport: [[gerrit:700734|Revert "Fall back from explicit parameter order to TemplateData sort" ()]] (duration: 01m 09s) [production]
18:28 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thumbor1006.eqiad.wmnet with reason: REIMAGE [production]
18:28 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on thumbor1005.eqiad.wmnet with reason: REIMAGE [production]
18:27 <awight@deploy1002> sync-file aborted: Backport: [[gerrit:700734|Revert "Fall back from explicit parameter order to TemplateData sort" ()]] (duration: 00m 40s) [production]
18:26 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on thumbor1005.eqiad.wmnet with reason: REIMAGE [production]
18:19 <legoktm> pulled in updates for thirdparty/kubeadm-k8s-1-18 buster-wikimedia on apt1001 [production]
17:47 <brennen> gitlab1001: run ansible to deploy https://gerrit.wikimedia.org/r/700851 (T274463) [production]
17:42 <dduvall> testwikis to 1.37.0-wmf.11 (cc open blockers T285125 T285118 T271011) [production]
17:41 <dduvall@deploy1002> Finished scap: testwikis wikis to 1.37.0-wmf.11 (duration: 30m 59s) [production]
17:21 <moritzm> installing isc-dhcp security updates [production]
17:18 <mbsantos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
17:16 <mbsantos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
17:14 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
17:11 <moritzm> installing ruby-websocket-extensions security updates [production]
17:10 <dduvall@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.11 [production]
17:08 <mbsantos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
17:07 <moritzm> installing velocity security updates [production]
17:07 <mbsantos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
17:04 <dduvall> 1.37.0-wmf.11 was branched at c161d3bd063b06d09be4167b38a72087db3ba7d2 for T281152 [production]
17:04 <mbsantos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
16:41 <Trey314159> reindexing Dutch wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T284185) [production]