3451-3500 of 10000 results (43ms)
2021-09-15 §
06:57 <elukey> shutdown ms-be2045 (again) after seeing T290881 [production]
06:02 <elukey> powercycle ms-be2045 - no ssh, no remote tty available [production]
05:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Restore db1109 original load', diff saved to https://phabricator.wikimedia.org/P17274 and previous config saved to /var/cache/conftool/dbconfig/20210915-052802-marostegui.json [production]
04:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase db1109 load', diff saved to https://phabricator.wikimedia.org/P17273 and previous config saved to /var/cache/conftool/dbconfig/20210915-043053-marostegui.json [production]
2021-09-14 §
23:01 <legoktm@deploy1002> Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s) [production]
22:59 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Re-enable VipsScaler (1 of 2) (duration: 01m 05s) [production]
22:58 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:43 <legoktm> legoktm@cumin2001:~$ sudo systemctl reset-failed # clear httpbb_hourly_tests failure, moved to cumin1001 [production]
22:34 <legoktm@deploy1002> Finished scap: Rebuild i18n for redeployment of VipsScaler (T290759) (duration: 23m 49s) [production]
22:34 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:12 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:11 <legoktm@deploy1002> Started scap: Rebuild i18n for redeployment of VipsScaler (T290759) [production]
22:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
20:20 <dancy> testing upcoming Scap release on beta [production]
20:20 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:720387|Early adopt wgIncludejQueryMigrate=false on nlwiki (T280944)]] (duration: 01m 48s) [production]
20:06 <cdanis> T290425 ✔️ cdanis@alert1001.wikimedia.org ~ 🕓🍵 sudo /usr/bin/statograph -c /etc/statograph/config.yml erase_metric_data lyfcttm2lhw4 [production]
20:06 <cdanis> T290425 ✔️ cdanis@alert1001.wikimedia.org ~ 🕓🍵 sudo /usr/bin/statograph -c /etc/statograph/config.yml erase_metric_data h5mvbny28713 [production]
19:19 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
19:08 <hashar@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.23 [production]
18:48 <moritzm> removed filter for tcp/25 on mx2001, reimage is complete T286911 [production]
18:33 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:24 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:20 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 2982638039720107d0b6e3227f5dce5b34ce7533: Offer the DiscussionTools reply tool as opt-out setting at ptwikinews (T285162) (duration: 01m 06s) [production]
18:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 7f1de32f4b5788e92291a5448563bc61a9f561e2: Offer the DiscussionTools reply tool as opt-out setting at Wikimania wiki (T284339) (duration: 01m 05s) [production]
18:15 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:13 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: e36f4d3dcc368f0afbce3649ce72f2135ab1c76f: DiscussionTools: Make newtopictool available to everyone on arwiki and cswiki (T285724) (duration: 01m 04s) [production]
18:09 <urbanecm@deploy1002> Synchronized debug.json: Idef64e72 (duration: 01m 29s) [production]
18:06 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
18:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
17:54 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: reimage [production]
17:54 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on mx2001.wikimedia.org with reason: reimage [production]
17:45 <moritzm> reimaging mx2001 to bullseye T286911 [production]
16:57 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:55 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
16:32 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:28 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:28 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:24 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:20 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:16 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
15:53 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on maps1010.eqiad.wmnet with reason: Resyncing from master [production]
15:53 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on maps1010.eqiad.wmnet with reason: Resyncing from master [production]