|
2021-10-26
§
|
| 07:05 |
<effie> |
pool wtp1026.eqiad.wmnet |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17606 and previous config saved to /var/cache/conftool/dbconfig/20211026-063647-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17605 and previous config saved to /var/cache/conftool/dbconfig/20211026-062144-root.json |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17604 and previous config saved to /var/cache/conftool/dbconfig/20211026-060640-root.json |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17603 and previous config saved to /var/cache/conftool/dbconfig/20211026-055136-root.json |
[production] |
| 05:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17602 and previous config saved to /var/cache/conftool/dbconfig/20211026-053633-root.json |
[production] |
| 05:35 |
<Spookyville> |
moodle instance on Mars VM enabled (T289309) |
[wikisp] |
| 05:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17601 and previous config saved to /var/cache/conftool/dbconfig/20211026-052129-root.json |
[production] |
| 04:59 |
<James_F> |
Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline T293924 |
[releng] |
| 02:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:06 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 01:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 01:24 |
<krinkle@deploy1002> |
Synchronized wmf-config/logging.php: I0211e1c77 (duration: 00m 55s) |
[production] |
| 01:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
|
2021-10-25
§
|
| 23:27 |
<dduvall> |
fully provisioned runner-{1008,1011,1012,1013,1014,1015,1016,1017,1018,1019} instances for use as new gitlab runners and removed old instances (T293835) |
[releng] |
| 23:12 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s) |
[production] |
| 23:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:57 |
<ryankemper> |
[wcqs] Downtimed `wcqs*` until roughly a week from now (while we setup oauth) |
[production] |
| 22:53 |
<legoktm> |
uploaded PHP 7.4.25 to apt.wm.o (DSA-4992-1) |
[production] |
| 22:44 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@e908052] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 22:30 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS (duration: 03m 04s) |
[production] |
| 22:27 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 21:53 |
<mutante> |
new project language "pwn" added - Paiwan is a native language of Taiwan, spoken by the Paiwan, a Taiwanese indigenous people. T292415 |
[production] |
| 21:52 |
<mutante> |
new project language "ami" added - Sowal no 'Amis is the Formosan language of the 'Amis (or Ami), an indigenous people living along the east coast of Taiwan. - T292414 |
[production] |
| 21:50 |
<mutante> |
log authdns1001 (DNS) - sudo authdns-update, add new project language "ami" (Amis) for T292414 - edited langlist.tmpl which regenerates all project zones |
[production] |
| 21:41 |
<wm-bot> |
<jeanfred> Deploy 7e3343a (T278156) |
[tools.integraality] |
| 21:40 |
<mutante> |
authdns1001 (DNS) - sudo authdns-update, add new project language "pwn" (Paiwan) for T292415 |
[production] |
| 20:50 |
<James_F> |
Docker: Publishing php*-comile images without the PECL test so they work again. |
[releng] |
| 20:32 |
<James_F> |
Zuul: Run set_mw_dependencies() for all mwext-/mwskin- jobs, not just php72 |
[releng] |
| 20:22 |
<wm-bot> |
<jeanfred> Deploy ef0f537 (T284684) |
[tools.integraality] |
| 20:15 |
<wm-bot> |
<jeanfred> Deploy 977236b (T284183) |
[tools.integraality] |
| 19:47 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<mutante> |
mw2255 - depooled=inactive (incl "dsh groups"), shut down physically for T283582 - can be worked on anytime |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:42 |
<mutante> |
icinga - ACKing all unhandled CRIT alerts on hosts with "dev" or "test" in their name, regardless of notifications being disabled or not. just so that we get more signal than noise in actual unhandled CRITs in web UI |
[production] |
| 19:40 |
<mutante> |
cumin2002 - sudo systemctl reset-failed to clear Icinga alert about failed but (now) non-existing service database-backups-snapshots.service, assuming it's a case of "only in active DC" |
[production] |
| 19:38 |
<wm-bot> |
<lucaswerkmeister> deployed 0f5b5de66a (bump startupProbe failureThreshold 3→10) |
[tools.lexeme-forms] |
| 19:34 |
<wm-bot> |
<lucaswerkmeister> deployment was successful after all 🤷 |
[tools.lexeme-forms] |
| 19:31 |
<wm-bot> |
<lucaswerkmeister> belay that, the new pod hasn’t actually started properly. investigating |
[tools.lexeme-forms] |
| 19:29 |
<wm-bot> |
<lucaswerkmeister> deployed 754342b9a3 (language name for bn-x-Q6747180) |
[tools.lexeme-forms] |
| 19:21 |
<James_F> |
Docker: Publishing new php74 and cascaded images with PHP 7.4 from Wikimedia package T293851 |
[releng] |
| 19:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:12 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:07 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Temporarily move mw groups to db1123 T294295', diff saved to https://phabricator.wikimedia.org/P17597 and previous config saved to /var/cache/conftool/dbconfig/20211025-190717-kormat.json |
[production] |
| 19:06 |
<mutante> |
db1112 - powercycling |
[production] |