901-950 of 10000 results (29ms)
2021-06-08 ยง
19:08 <ryankemper> [WDQS] `ryankemper@wdqs1005:~$ sudo pool` (all caught up on lag) [production]
18:47 <bblack> dns4001: update gdnsd to 3.7.0-1~wmf1 [production]
18:43 <bblack> apt: update gdnsd package to gdnsd-3.7.0-1~wmf1 [production]
17:49 <jgiannelos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:36 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:25 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
17:10 <elukey> fix dbstore1007's ip address in analytics-in4 on cr{1,2}-eqiad [production]
17:06 <jhuneidi@deploy1002> Finished scap: testwikis wikis to 1.37.0-wmf.9 refs T281150 (duration: 34m 12s) [production]
16:32 <jhuneidi@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.9 refs T281150 [production]
16:27 <papaul> powerdown moss-fe2002 for relocation [production]
16:06 <papaul> powerdown ms-backup2002 for relocation [production]
16:02 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:40 <papaul> powerdown ms-be2061 for relocation [production]
15:40 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=cp203[34].codfw.wmnet [production]
15:33 <papaul> powerdown thanos-fe2003 for relocation [production]
15:23 <Krinkle> mwmaint1002: Running purge-parsercache-now.php on server 4/4 (pc1009) ref P16060, T280605, T282761. [production]
15:19 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on pc2009.codfw.wmnet,pc1009.eqiad.wmnet with reason: Purging parsercache pc3 T282761 [production]
15:19 <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on pc2009.codfw.wmnet,pc1009.eqiad.wmnet with reason: Purging parsercache pc3 T282761 [production]
15:13 <papaul> powerdown cp2034 for relocation [production]
15:04 <papaul> powerdown cp2033 for relocation [production]
14:59 <bblack@cumin1001> conftool action : set/pooled=no; selector: name=cp203[34].codfw.wmnet [production]
14:43 <moritzm> cleanup now unused nginx mods and former deps (various X11 libs and libxslt) on testreduce1001/scandium after switch towards nginx-light T164456 [production]
14:08 <marostegui> Restart sanitarium hosts (db2094, db2095, db1154, db1155) to pick up new filters T284106 [production]
14:05 <kormat@deploy1002> Synchronized wmf-config/db-eqiad.php: Set pc1010 as pc3 master T282761 (duration: 00m 57s) [production]
14:05 <kormat> setting pc1010 as pc3 primary T282761 [production]
13:51 <jbond@deploy1002> Finished deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next (duration: 00m 42s) [production]
13:51 <jbond@deploy1002> Started deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next [production]
13:48 <otto@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
13:41 <otto@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
13:40 <jbond@deploy1002> Finished deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next (duration: 00m 47s) [production]
13:39 <jbond@deploy1002> Started deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next [production]
13:36 <jbond@deploy1002> Finished deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next (duration: 01m 03s) [production]
13:35 <jbond@deploy1002> Started deploy [netbox/deploy@c70df91]: Force deploy of gerrit/672831 to netbox-next [production]
13:33 <otto@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons. - otto@cumin1001 [production]
13:22 <otto@cumin1001> START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons. - otto@cumin1001 [production]
12:15 <kormat@deploy1002> Synchronized wmf-config/db-eqiad.php: Repool pc1008 as pc2 master T282761 (duration: 00m 57s) [production]
12:14 <kormat> setting pc1008 back as pc2 primary T282761 [production]
11:54 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ef49422b162ab0161bc39da857b3230175ac4492: enwiki: Disable indexing on the Book namespace (T283522) (duration: 00m 56s) [production]
11:46 <urbanecm> Start server-side upload for 1 file (T283470) [production]
11:45 <moritzm> installing nginx security updates on buster [production]
11:43 <urbanecm> Start server-side upload for 2 files (T283645, T283583) [production]
11:39 <urbanecm> EU B&C deployment done [production]
11:38 <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16329 and previous config saved to /var/cache/conftool/dbconfig/20210608-113857-kormat.json [production]
11:38 <moritzm> installing ruby-nokogiri security updates [production]
11:37 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/WikimediaEvents/: b0b46530b731d2a5f17b0aa04a4cf99df175e23d: universalLanguageSelector: Add missing properties (T280770) (duration: 00m 56s) [production]
11:31 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.7/extensions/UniversalLanguageSelector/resources/js/ext.uls.launch.js: 5df13eeae3b52b98eaf3fdb99ddfa5a0f7b2b1e4: Pass context to compact_language_links.open hook (T280770) (duration: 00m 57s) [production]
11:23 <kormat@cumin1001> dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: reimaged to buster T283131', diff saved to https://phabricator.wikimedia.org/P16328 and previous config saved to /var/cache/conftool/dbconfig/20210608-112354-kormat.json [production]
11:15 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 73dc708efc25caa667be516c685885db3983be73: lvwiki: Enable Growth features in dark mode (T278191; 3/3) (duration: 00m 58s) [production]
11:13 <urbanecm@deploy1002> Synchronized wmf-config/config/lvwiki.yaml: 73dc708efc25caa667be516c685885db3983be73: lvwiki: Enable Growth features in dark mode (T278191; 2/3) (duration: 00m 56s) [production]
11:12 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 73dc708efc25caa667be516c685885db3983be73: lvwiki: Enable Growth features in dark mode (T278191; 1/3) (duration: 00m 57s) [production]