301-350 of 10000 results (43ms)
2022-03-31 ยง
15:06 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
15:06 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
15:05 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on durum[1001-1002].eqiad.wmnet with reason: reboot for update T304938 [production]
15:05 <elukey@deploy1002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
15:05 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on durum[1001-1002].eqiad.wmnet with reason: reboot for update T304938 [production]
15:05 <elukey@deploy1002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
14:57 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp6016.drmrs.wmnet with OS buster [production]
14:57 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on doh6002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:56 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on doh6002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:56 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on doh6001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:56 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on doh6001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:56 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
14:52 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on doh5002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:52 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on doh5002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:52 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on doh5001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:52 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on doh5001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:52 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
14:50 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
14:47 <mmandere> depool cp6016 for reimage - T290005 [production]
14:46 <elukey@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
14:44 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on doh4002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:44 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on doh4002.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:44 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on doh4001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:43 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on doh4001.wikimedia.org with reason: reboot for kernel update T304938 [production]
14:39 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5009.eqsin.wmnet with reason: host reimage [production]
14:36 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5009.eqsin.wmnet with reason: host reimage [production]
14:22 <duesen> (late) about 5 hours ago, I removed /var/run/php/use-config-schema from mw1415 to disable config schema loading (T304460) [production]
14:09 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp5009.eqsin.wmnet with OS buster [production]
14:05 <mmandere@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5009.eqsin.wmnet with OS buster [production]
14:03 <mmandere@cumin1001> START - Cookbook sre.hosts.reimage for host cp5009.eqsin.wmnet with OS buster [production]
14:03 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
14:02 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
14:02 <moritzm> installing vim security updates on buster [production]
14:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
14:00 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1002.wikimedia.org [production]
13:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:56 <Lucas_WMDE> UTC afternoon backport+config window done [production]
13:55 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.39.0-wmf.5/includes/changetags/ChangeTags.php: Backport: [[gerrit:775437|ChangeTags: Use localizer with correct page title to parse messages (T302754)]] (duration: 00m 51s) [production]
13:53 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:53 <mmandere> depool cp5009 for reimage - T290005 [production]
13:52 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:52 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netmon1002.wikimedia.org [production]
13:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:51 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2001.wikimedia.org [production]
13:51 <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.39.0-wmf.5/resources/src/mediawiki.special.createaccount/HtmlformChecker.js: Backport: [[gerrit:775432|Fix error/warning boxes on signup form (T305098)]] (duration: 00m 50s) [production]
13:41 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host netmon2001.wikimedia.org [production]
13:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]