|
2021-12-08
§
|
| 14:17 |
<moritzm> |
drain primary/secondary instance off ganeti2020 T296622 |
[production] |
| 14:01 |
<moritzm> |
installing nss regression updates for stretch |
[production] |
| 13:57 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage |
[production] |
| 13:57 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage |
[production] |
| 13:56 |
<moritzm> |
drain primary/secondary instance off ganeti2015 T296622 |
[production] |
| 13:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2014.codfw.wmnet |
[production] |
| 13:34 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2014.codfw.wmnet |
[production] |
| 13:04 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2013.codfw.wmnet |
[production] |
| 13:00 |
<ema> |
powercycle cp5006 T290005 |
[production] |
| 12:59 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2013.codfw.wmnet |
[production] |
| 10:43 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet |
[production] |
| 10:42 |
<ema> |
depool cp5006, the host is down T290005#7555417 |
[production] |
| 10:35 |
<ema> |
cp3051: repool w/ single backend experiment enabled T288106 |
[production] |
| 10:23 |
<ema> |
cp3051: stop ats-be and clear its cache T288106 |
[production] |
| 10:22 |
<ema> |
cp3051: depool to enable single backend experiment T288106 |
[production] |
| 10:16 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=testwiki --custom-groups=steward --force "Dom walden" |
[production] |
| 10:05 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2014.codfw.wmnet with OS buster |
[production] |
| 09:58 |
<majavah> |
remove all users from obsolete "shell" and "clouadmin" groups on labtestwiki (labtestwikitech.wikimedia.org) |
[production] |
| 09:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:29 |
<dcausse> |
restarting blazegraph on wdqs1006 (jvm stuck for 24h) |
[production] |
| 09:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:23 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2014.codfw.wmnet with OS buster |
[production] |
| 09:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:18 |
<taavi@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743659|Remove UserMerge rights from labswiki (wikitech)]] (duration: 01m 07s) |
[production] |
| 09:05 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2013.codfw.wmnet with OS buster |
[production] |
| 08:34 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2013.codfw.wmnet with OS buster |
[production] |
| 04:50 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 03:37 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 02:09 |
<legoktm> |
powercycle graphite1004 via mgmt |
[production] |
| 00:51 |
<ebernhardson@deploy1002> |
Synchronized php-1.38.0-wmf.12/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: backport window for 744896 (duration: 01m 05s) |
[production] |
| 00:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:09 |
<ebernhardson@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T296897 Move cirrus traffic back to eqiad (duration: 01m 08s) |
[production] |
|
2021-12-07
§
|
| 23:21 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 23:01 |
<jgleeson> |
updated payments-wiki from 4a4ef51d to 2e164062 |
[production] |
| 22:56 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:15 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.12 refs T293953 |
[production] |
| 22:12 |
<dancy@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.7 (duration: 04m 18s) |
[production] |
| 22:07 |
<dancy@deploy1002> |
Finished scap: testwikis wikis to 1.38.0-wmf.12 refs T293953 (duration: 44m 14s) |
[production] |
| 22:07 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 22:06 |
<andrew@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 21:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |