|
2021-10-25
ยง
|
| 23:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:57 |
<ryankemper> |
[wcqs] Downtimed `wcqs*` until roughly a week from now (while we setup oauth) |
[production] |
| 22:53 |
<legoktm> |
uploaded PHP 7.4.25 to apt.wm.o (DSA-4992-1) |
[production] |
| 22:44 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@e908052] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 22:30 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS (duration: 03m 04s) |
[production] |
| 22:27 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 21:53 |
<mutante> |
new project language "pwn" added - Paiwan is a native language of Taiwan, spoken by the Paiwan, a Taiwanese indigenous people. T292415 |
[production] |
| 21:52 |
<mutante> |
new project language "ami" added - Sowal no 'Amis is the Formosan language of the 'Amis (or Ami), an indigenous people living along the east coast of Taiwan. - T292414 |
[production] |
| 21:50 |
<mutante> |
log authdns1001 (DNS) - sudo authdns-update, add new project language "ami" (Amis) for T292414 - edited langlist.tmpl which regenerates all project zones |
[production] |
| 21:40 |
<mutante> |
authdns1001 (DNS) - sudo authdns-update, add new project language "pwn" (Paiwan) for T292415 |
[production] |
| 19:47 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<mutante> |
mw2255 - depooled=inactive (incl "dsh groups"), shut down physically for T283582 - can be worked on anytime |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:42 |
<mutante> |
icinga - ACKing all unhandled CRIT alerts on hosts with "dev" or "test" in their name, regardless of notifications being disabled or not. just so that we get more signal than noise in actual unhandled CRITs in web UI |
[production] |
| 19:40 |
<mutante> |
cumin2002 - sudo systemctl reset-failed to clear Icinga alert about failed but (now) non-existing service database-backups-snapshots.service, assuming it's a case of "only in active DC" |
[production] |
| 19:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:12 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:07 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Temporarily move mw groups to db1123 T294295', diff saved to https://phabricator.wikimedia.org/P17597 and previous config saved to /var/cache/conftool/dbconfig/20211025-190717-kormat.json |
[production] |
| 19:06 |
<mutante> |
db1112 - powercycling |
[production] |
| 19:04 |
<legoktm@cumin1001> |
dbctl commit (dc=all): 'Depool db1112 (T294295)', diff saved to https://phabricator.wikimedia.org/P17596 and previous config saved to /var/cache/conftool/dbconfig/20211025-190436-legoktm.json |
[production] |
| 18:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<jforrester@deploy1002> |
Synchronized php-1.38.0-wmf.5/extensions/timeline/includes/Timeline.php: Backport: [[gerrit:734312|Input may be null when rendering a self-closing tag `<timeline />` (T294020)]] (duration: 00m 55s) |
[production] |
| 18:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732971|Fix some easy codestyle issues]] (duration: 00m 55s) |
[production] |
| 18:22 |
<jforrester@deploy1002> |
Synchronized w/static.php: Config: [[gerrit:732971|Fix some easy codestyle issues]] (duration: 00m 54s) |
[production] |
| 18:19 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732840|Fix array declaration of NS_USER_TALK abbreviation on ruwikiquote (T197058)]] (duration: 00m 55s) |
[production] |
| 18:16 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:15 |
<jforrester@deploy1002> |
Synchronized wmf-config/flaggedrevs.php: Config: [[gerrit:732836|flaggedrevs: Drop legacy wgFlaggedRevsStatsAge config, no longer read]] (duration: 00m 55s) |
[production] |
| 18:13 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:11 |
<jforrester@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:732254|Make reply tool available as opt-out on frwiki (T293687)]] (duration: 00m 56s) |
[production] |
| 17:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2253.codfw.wmnet |
[production] |
| 17:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2253.codfw.wmnet |
[production] |
| 17:39 |
<mutante> |
mw2253 - scap pull after hw maintenance is over |
[production] |
| 17:32 |
<bd808@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:26 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:24 |
<mmandere@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:23 |
<bd808@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:22 |
<XioNoX> |
update core routers ACLs |
[production] |
| 17:20 |
<mmandere@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:49 |
<XioNoX> |
update management routers ACLs |
[production] |
| 16:36 |
<XioNoX> |
DNS - Add eqsin-ulsfo transport v6 prefix - T273308 |
[production] |
| 16:31 |
<mmandere@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:28 |
<mmandere@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:25 |
<accraze@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:25 |
<mmandere@cumin2002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |