1001-1050 of 10000 results (24ms)
2020-07-28 §
07:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:31 <jayme@cumin1001> conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet [production]
07:27 <liw@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.2 [production]
07:03 <liw> 1.36.0-wmf.2 was branched at 04e863fdf3646ee6ed5c05b784f85c9f323e1f19 for T257970 [production]
05:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314 for MCR schema change', diff saved to https://phabricator.wikimedia.org/P12065 and previous config saved to /var/cache/conftool/dbconfig/20200728-051928-marostegui.json [production]
05:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1144:3314 and restore db1146:3314 original weight', diff saved to https://phabricator.wikimedia.org/P12064 and previous config saved to /var/cache/conftool/dbconfig/20200728-051813-marostegui.json [production]
02:17 <eileen> process-control config revision is 6811ca294a - just delayed silverpop_daily a bit as clashing with dedupe [production]
00:18 <andrew@cumin1001> conftool action : set/pooled=inactive; selector: name=cloudcephmon1003.eqiad.wmnet [production]
00:17 <andrew@cumin1001> conftool action : set/pooled=no; selector: name=cloudcephmon1003.eqiad.wmnet [production]
2020-07-27 §
23:49 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@ac8e5d0]: airflow: head queries report, managed variables, refinery-drop-hive-partitions support (duration: 00m 54s) [production]
23:48 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@ac8e5d0]: airflow: head queries report, managed variables, refinery-drop-hive-partitions support [production]
23:28 <mutante> otrs1001 - ran puppet (it was alerting in icinga that puppet failed, but it was neither disabled nor failing and changed nothing when it ran) [production]
21:31 <sbassett@deploy1001> Synchronized wmf-config/CommonSettings.php: Deployed CentralNotice CSP conifg change for T258459 (duration: 00m 57s) [production]
21:10 <sbassett> Deployed mitigations for T238075 [production]
20:41 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.1/extensions/InterwikiSorting/: c5f6c97856a5dbe673064afd2804bebb9b787580: Use LanguageLinksHook to sort interwiki links (T257625) (duration: 00m 59s) [production]
19:50 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:44 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . [production]
19:36 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
19:23 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'sessionstore' for release 'production' . [production]
19:19 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'sessionstore' for release 'production' . [production]
19:11 <jhuneidi@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'staging' . [production]
19:06 <jhuneidi@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'echostore' for release 'production' . [production]
19:00 <jhuneidi@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'echostore' for release 'production' . [production]
18:57 <urbanecm@deploy1001> sync-file aborted: 3833b135caf4171daa0814eba81393b6c44db619: Move footer logos to /static/images/footer (T257732) (duration: 00m 04s) [production]
18:50 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: c6a9674366d9c8d273ce0e74dfb6a04c91d64307: Move footer logos to wmg* variables (T257732) (duration: 00m 56s) [production]
18:50 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: (no justification provided) (duration: 00m 57s) [production]
18:49 <urbanecm@deploy1001> sync-file aborted: (no justification provided) (duration: 00m 01s) [production]
18:48 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: c6a9674366d9c8d273ce0e74dfb6a04c91d64307: Move footer logos to wmg* variables (T257732) (duration: 00m 57s) [production]
18:29 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable desktop web UI click tracking instrumentation on frwiki, hewiki, fawiki (T258058) (duration: 00m 56s) [production]
18:13 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Remove WPBSkinBlacklist (T254675) (duration: 00m 57s) [production]
17:42 <liw@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.1 [production]
17:30 <liw> promoting train to group2 [production]
17:14 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
17:14 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
17:14 <dpifke@deploy1001> Finished deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) (duration: 00m 05s) [production]
17:14 <dpifke@deploy1001> Started deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) [production]
16:59 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet [production]
16:58 <andrew@cumin1001> conftool action : set/pooled=no; selector: name=cloudcephmon1002.eqiad.wmnet [production]
16:57 <andrew@cumin1001> conftool action : set/pooled=inactive; selector: name=cloudcephmon1002.eqiad.wmnet [production]
16:50 <andrew@cumin1001> conftool action : set/pooled=inactive; selector: name=cloudcephosd1003.eqiad.wmnet [production]
16:50 <andrew@cumin1001> conftool action : set/pooled=inactive; selector: name=cloudcephosd1002.eqiad.wmnet [production]
16:50 <andrew@cumin1001> conftool action : set/pooled=inactive; selector: name=cloudcephosd1001.eqiad.wmnet [production]
16:50 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephmon1003.eqiad.wmnet [production]
16:50 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephmon1002.eqiad.wmnet [production]
16:49 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephmon1001.eqiad.wmnet [production]
16:48 <andrew@cumin1001> conftool action : set/pooled=no; selector: name=cloudcephosd1003.wikimedia.org [production]
16:48 <andrew@cumin1001> conftool action : set/pooled=no; selector: name=cloudcephosd1002.wikimedia.org [production]
16:48 <andrew@cumin1001> conftool action : set/pooled=no; selector: name=cloudcephosd1001.wikimedia.org [production]
16:48 <andrew@cumin1001> conftool action : set/pooled=yes; selector: name=cloudcephosd1001.eqiad.wmnet [production]