|
2021-12-08
§
|
| 09:29 |
<dcausse> |
restarting blazegraph on wdqs1006 (jvm stuck for 24h) |
[production] |
| 09:23 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:23 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2014.codfw.wmnet with OS buster |
[production] |
| 09:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 09:18 |
<taavi@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743659|Remove UserMerge rights from labswiki (wikitech)]] (duration: 01m 07s) |
[production] |
| 09:05 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2013.codfw.wmnet with OS buster |
[production] |
| 08:34 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host ganeti2013.codfw.wmnet with OS buster |
[production] |
| 04:50 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 03:37 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 02:09 |
<legoktm> |
powercycle graphite1004 via mgmt |
[production] |
| 00:51 |
<ebernhardson@deploy1002> |
Synchronized php-1.38.0-wmf.12/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: backport window for 744896 (duration: 01m 05s) |
[production] |
| 00:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:09 |
<ebernhardson@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: T296897 Move cirrus traffic back to eqiad (duration: 01m 08s) |
[production] |
|
2021-12-07
§
|
| 23:21 |
<andrew@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 23:01 |
<jgleeson> |
updated payments-wiki from 4a4ef51d to 2e164062 |
[production] |
| 22:56 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:15 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.12 refs T293953 |
[production] |
| 22:12 |
<dancy@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.7 (duration: 04m 18s) |
[production] |
| 22:07 |
<dancy@deploy1002> |
Finished scap: testwikis wikis to 1.38.0-wmf.12 refs T293953 (duration: 44m 14s) |
[production] |
| 22:07 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 22:06 |
<andrew@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 21:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:23 |
<dancy@deploy1002> |
Started scap: testwikis wikis to 1.38.0-wmf.12 refs T293953 |
[production] |
| 21:18 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 19:58 |
<ebernhardson@deploy1002> |
Finished deploy [wdqs/wdqs@c21117f] (wcqs): Deploy version 0.3.95 to wcqs (duration: 01m 48s) |
[production] |
| 19:56 |
<ebernhardson@deploy1002> |
Started deploy [wdqs/wdqs@c21117f] (wcqs): Deploy version 0.3.95 to wcqs |
[production] |
| 19:56 |
<cmooney@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 19:49 |
<eileen> |
revision civicrm 311382de -> c47cf762 |
[production] |
| 19:26 |
<legoktm> |
upgrading sacp to 4.1.0 everywhere (T296867) |
[production] |
| 19:26 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster |
[production] |
| 19:18 |
<herron> |
graphite1004.mgmt: racadm serveraction powercycle |
[production] |
| 19:13 |
<ebernhardson> |
start outage recovery for commonswiki against eqiad cirrus cluster after snapshot restore |
[production] |
| 18:47 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 18:46 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 18:45 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 18:38 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 18:33 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 18:27 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 17:54 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2002.codfw.wmnet |
[production] |
| 17:51 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki2002.codfw.wmnet |
[production] |
| 17:41 |
<herron> |
graphite1004.mgmt: racadm serveraction powercycle |
[production] |