|
2021-10-25
§
|
| 22:30 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS (duration: 03m 04s) |
[production] |
| 22:27 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 21:53 |
<mutante> |
new project language "pwn" added - Paiwan is a native language of Taiwan, spoken by the Paiwan, a Taiwanese indigenous people. T292415 |
[production] |
| 21:52 |
<mutante> |
new project language "ami" added - Sowal no 'Amis is the Formosan language of the 'Amis (or Ami), an indigenous people living along the east coast of Taiwan. - T292414 |
[production] |
| 21:50 |
<mutante> |
log authdns1001 (DNS) - sudo authdns-update, add new project language "ami" (Amis) for T292414 - edited langlist.tmpl which regenerates all project zones |
[production] |
| 21:41 |
<wm-bot> |
<jeanfred> Deploy 7e3343a (T278156) |
[tools.integraality] |
| 21:40 |
<mutante> |
authdns1001 (DNS) - sudo authdns-update, add new project language "pwn" (Paiwan) for T292415 |
[production] |
| 20:50 |
<James_F> |
Docker: Publishing php*-comile images without the PECL test so they work again. |
[releng] |
| 20:32 |
<James_F> |
Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 |
[releng] |
| 20:22 |
<wm-bot> |
<jeanfred> Deploy ef0f537 (T284684) |
[tools.integraality] |
| 20:15 |
<wm-bot> |
<jeanfred> Deploy 977236b (T284183) |
[tools.integraality] |
| 19:47 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<mutante> |
mw2255 - depooled=inactive (incl "dsh groups"), shut down physically for T283582 - can be worked on anytime |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:42 |
<mutante> |
icinga - ACKing all unhandled CRIT alerts on hosts with "dev" or "test" in their name, regardless of notifications being disabled or not. just so that we get more signal than noise in actual unhandled CRITs in web UI |
[production] |
| 19:40 |
<mutante> |
cumin2002 - sudo systemctl reset-failed to clear Icinga alert about failed but (now) non-existing service database-backups-snapshots.service, assuming it's a case of "only in active DC" |
[production] |
| 19:38 |
<wm-bot> |
<lucaswerkmeister> deployed 0f5b5de66a (bump startupProbe failureThreshold 3→10) |
[tools.lexeme-forms] |
| 19:34 |
<wm-bot> |
<lucaswerkmeister> deployment was successful after all 🤷 |
[tools.lexeme-forms] |
| 19:31 |
<wm-bot> |
<lucaswerkmeister> belay that, the new pod hasn’t actually started properly. investigating |
[tools.lexeme-forms] |
| 19:29 |
<wm-bot> |
<lucaswerkmeister> deployed 754342b9a3 (language name for bn-x-Q6747180) |
[tools.lexeme-forms] |
| 19:21 |
<James_F> |
Docker: Publishing new php74 and cascaded images with PHP 7.4 from Wikimedia package T293851 |
[releng] |
| 19:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:12 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:07 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Temporarily move mw groups to db1123 T294295', diff saved to https://phabricator.wikimedia.org/P17597 and previous config saved to /var/cache/conftool/dbconfig/20211025-190717-kormat.json |
[production] |
| 19:06 |
<mutante> |
db1112 - powercycling |
[production] |
| 19:04 |
<legoktm@cumin1001> |
dbctl commit (dc=all): 'Depool db1112 (T294295)', diff saved to https://phabricator.wikimedia.org/P17596 and previous config saved to /var/cache/conftool/dbconfig/20211025-190436-legoktm.json |
[production] |
| 18:47 |
<wm-bot> |
<jeanfred> Deploy 5dfef39, e940ae7, 765b314, 3e4c2de, c859c45, 5526cb3 |
[tools.integraality] |
| 18:45 |
<Reedy> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734377 T294238 |
[releng] |
| 18:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<jforrester@deploy1002> |
Synchronized php-1.38.0-wmf.5/extensions/timeline/includes/Timeline.php: Backport: [[gerrit:734312|Input may be null when rendering a self-closing tag `<timeline />` (T294020)]] (duration: 00m 55s) |
[production] |
| 18:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732971|Fix some easy codestyle issues]] (duration: 00m 55s) |
[production] |
| 18:22 |
<jforrester@deploy1002> |
Synchronized w/static.php: Config: [[gerrit:732971|Fix some easy codestyle issues]] (duration: 00m 54s) |
[production] |
| 18:19 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732840|Fix array declaration of NS_USER_TALK abbreviation on ruwikiquote (T197058)]] (duration: 00m 55s) |
[production] |
| 18:16 |
<Reedy> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734370 https://phabricator.wikimedia.org/T294238 |
[releng] |
| 18:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:15 |
<jforrester@deploy1002> |
Synchronized wmf-config/flaggedrevs.php: Config: [[gerrit:732836|flaggedrevs: Drop legacy wgFlaggedRevsStatsAge config, no longer read]] (duration: 00m 55s) |
[production] |
| 18:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:11 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732254|Make reply tool available as opt-out on frwiki (T293687)]] (duration: 00m 56s) |
[production] |
| 17:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2253.codfw.wmnet |
[production] |
| 17:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2253.codfw.wmnet |
[production] |
| 17:39 |
<mutante> |
mw2253 - scap pull after hw maintenance is over |
[production] |
| 17:32 |
<bd808@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:26 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:24 |
<mmandere@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:23 |
<bd808@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |