801-850 of 10000 results (24ms)
2020-10-07 §
00:36 <cdanis> T259621 cdanis@re1.cr2-esams> request system software add /var/tmp/junos-install-mx-x86-64-17.3R3-S8.1.tgz re0 no-validate [production]
00:26 <cdanis> T259621 cdanis@re0.cr2-esams> request chassis routing-engine master switch [production]
00:22 <cdanis> T259621 cdanis@re0.cr2-esams> request system reboot other-routing-engine [production]
00:15 <cdanis> T259621 cdanis@re0.cr2-esams> request system software add re1 no-validate /var/tmp/junos-install-mx-x86-64-17.3R3-S8.1.tgz [production]
00:01 <mutante> reinstalling testvm[345]001 to confirm OS installs work as normal after switching DHCP servers in POPs (T252526) [production]
2020-10-06 §
23:55 <mutante> 🖧 switched DHCP server for eqsin from install2003 to install5001 - homer deployed to cr*eqsin* (T252526) 🖧 [production]
23:53 <mutante> 🖧 switched DHCP server for ulsfo from install2003 to install4001 - homer deployed to cr*ulsfo* (T252526) 🖧 [production]
23:52 <mutante> 🖧 switched DHCP server for esams from install1003 to install3001 - homer deployed to cr*esams* (T252526) 🖧 [production]
23:43 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:11 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
23:07 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
22:32 <ryankemper> Restart of `wdqs-categories` done. WDQS deploy is complete [production]
21:57 <ryankemper> Restarting `wdqs-categories` across production instances one-at-a-time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'` [production]
21:57 <ryankemper> Restarting `wdqs-categories` across all test instances (not public facing): `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` [production]
21:56 <ryankemper> Restarting `wdqs-updater` across the fleet: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
21:55 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@e56a20e]: 0.3.51 (duration: 13m 09s) [production]
21:43 <ryankemper> All tests passing on canary `wdqs1003`, proceeding to rest of fleet [production]
21:42 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@e56a20e]: 0.3.51 [production]
21:14 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: gerrit:632535 (duration: 01m 00s) [production]
20:25 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:23 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:40 <Urbanecm> Morning B&C done [production]
18:40 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.11/skins/MinervaNeue/: 2118d265c0f5b6c914efeba86ba7eacd30c5ee0f: Hot fix: Use display for hiding/showing sidebar on OS 14_0 (T264376) (duration: 01m 00s) [production]
18:37 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.10/skins/MinervaNeue/: d428ccbdf3be9a45139f8b8c0874c113f1732198: Hot fix: Use display for hiding/showing sidebar on OS 14_0 (T264376) (duration: 01m 03s) [production]
18:24 <ppchelko@deploy1001> Synchronized wmf-config/Wikibase.php: Wikibase.php gerrit:631775 T263493 T259622 (duration: 00m 58s) [production]
18:23 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: IS.php gerrit:631775 T263493 T259622 (duration: 00m 59s) [production]
18:19 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:632516 T264043 (duration: 00m 59s) [production]
18:15 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:632323 T264637 (duration: 00m 58s) [production]
18:12 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: gerrit:632484 T264637 (duration: 00m 58s) [production]
15:41 <godog> centrallog* delete archived logs from old, single file, organization [production]
15:23 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
15:23 <jayme> updated envoyproxy to 1.15.1-2 on mw-canary and restbase-canary [production]
14:57 <sukhe> upload dnsdist_1.5.0-1wm1 to apt.wm.o (buster) - T263789 [production]
14:47 <kormat@cumin1001> dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12936 and previous config saved to /var/cache/conftool/dbconfig/20201006-144701-kormat.json [production]
14:45 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:45 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:45 <vgutierrez> Bump ECDHE-ECDSA-AES128-SHA pageview replacement to 5% - T262946 [production]
14:45 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:44 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:40 <jayme> updated envoyproxy to 1.15.1-2 on mw2295.codfw.wmnet,restbase2017.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase-backend,name=restbase2009.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase2009.codfw.wmnet [production]
14:38 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2009.codfw.wmnet [production]
14:36 <hnowlan> repooling restbase2009 [production]
14:31 <kormat@cumin1001> dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 75%: schema change T259831', diff saved to https://phabricator.wikimedia.org/P12935 and previous config saved to /var/cache/conftool/dbconfig/20201006-143157-kormat.json [production]
14:19 <volker-e@deploy1001> Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 05s) [production]
14:19 <volker-e@deploy1001> Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide: [production]
14:15 <jayme> installed envoyproxy 1.15.1-2 on mwdebug1001 [production]
14:08 <marostegui> Reboot db1076 for kernel upgrade T264755 [production]
14:04 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]