1651-1700 of 10000 results (61ms)
2020-04-09 §
10:50 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:50 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:49 <jmm@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
10:49 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:43 <ema> repool cp3051 T249809 [production]
10:30 <ema> cp3051: re-enable transient storage limit, downgrade varnish to 5.1.3-1wm12 (no 0035-vbf_stp_condfetch_crash.patch) and restart varnish-fe T249809 [production]
09:46 <ema> cp3051: disable transient storage limit and restart varnish-fe T249809 [production]
09:31 <XioNoX> offload traffic from NTT eqiad - T249808 [production]
07:56 <mutante> contint2001 - a2dismod mpm_event - then run puppet to let it enable php_mod_7.3 (race condition like mentioned in https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206) (T224591) [production]
07:56 <mutante> contint2001 - a2dismod mpm_event - then run puppet to let it enable php_mod_7.3 (race condition like mentioned in https://gerrit.wikimedia.org/r/c/operations/puppet/+/451206) [production]
07:24 <moritzm> synched jenkins 222.1 to apt.wikimedia.org (buster-wikimedia, thirdparty/ci) T224591 [production]
07:12 <marostegui> Repool labsdb1011 [production]
07:10 <XioNoX> switch urpf from log to syslog in ulsfo [production]
07:04 <XioNoX> re-activate BGP to Zayo in eqiad [production]
06:59 <vgutierrez> upgrade ats to version 8.0.6-1wm7 in cp[4026,4032,5006,5012] [production]
06:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:43 <XioNoX> confirmed on one host that the change didn't break logstash. Re-enable Puppet on logstash hosts - T244147 [production]
06:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:36 <XioNoX> disabling puppet on logstash host for CR deploy - T244147 [production]
06:30 <XioNoX> push urpf log only to eqiad - T244147 [production]
06:25 <XioNoX> push urpf log only to eqsin - T244147 [production]
06:21 <XioNoX> push urpf log only to AMS - T244147 [production]
05:40 <vgutierrez> upgrade ats to version 8.0.6-1wm6 in cp[4025,4031,5005,5011] - T249335 [production]
05:37 <marostegui> Stop MySQL on pc2008 for upgrade to Buster and 10.4 [production]
05:36 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool pc2008 for upgrade (duration: 01m 08s) [production]
05:08 <marostegui> Deploy schema change on db1123 [production]
05:07 <vgutierrez> upload trafficserver 8.0.6-1wm6 to apt.wm.o (buster) - T249335 [production]
2020-04-08 §
21:20 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.27/extensions/TemplateData/includes/TemplateDataHooks.php: Restore call to OutputPage::setupOOUI() (duration: 01m 07s) [production]
21:19 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.26/extensions/TemplateData/includes/TemplateDataHooks.php: Restore call to OutputPage::setupOOUI() (duration: 01m 09s) [production]
20:09 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
20:09 <otto@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
20:06 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
20:06 <otto@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
20:04 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
20:04 <otto@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
19:51 <gehel> restart wdqs-updater after deployment [production]
19:49 <mstyles@deploy1001> Finished deploy [wdqs/wdqs@c2995eb]: WDQS version 0.3.21 (duration: 14m 37s) [production]
19:44 <dpifke@deploy1001> Finished deploy [performance/navtiming@4acb04d]: Deploy new navtiming with First Input Delay metric https://phabricator.wikimedia.org/T238091 (duration: 00m 05s) [production]
19:44 <dpifke@deploy1001> Started deploy [performance/navtiming@4acb04d]: Deploy new navtiming with First Input Delay metric https://phabricator.wikimedia.org/T238091 [production]
19:35 <mstyles@deploy1001> Started deploy [wdqs/wdqs@c2995eb]: WDQS version 0.3.21 [production]
19:08 <jhuneidi@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.27 refs T247774 (duration: 01m 06s) [production]
19:07 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.27 refs T247774 [production]
19:02 <longma> deploying 1.35.0-wmf.27 to group1 [production]
18:37 <jforrester@deploy1001> Synchronized php-1.35.0-wmf.27/skins/Vector: T248761: Revert moving indicators in DOM (duration: 01m 07s) [production]
18:17 <reedy@deploy1001> Synchronized php-1.35.0-wmf.27/extensions/TemplateData/includes/TemplateDataHooks.php: T236809 (duration: 01m 06s) [production]
18:16 <reedy@deploy1001> Synchronized php-1.35.0-wmf.26/extensions/TemplateData/includes/TemplateDataHooks.php: T236809 (duration: 01m 10s) [production]
17:31 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
17:23 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
17:13 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:13 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]