|
2021-10-26
§
|
| 09:40 |
<godog> |
flip back write traffic to graphite1004 (all but mediawiki) - T247963 |
[production] |
| 09:27 |
<godog> |
move read traffic back to graphite1004 - T247963 |
[production] |
| 08:37 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:33 |
<ema> |
upload varnish_6.0.8-1wm2 to component/varnish6 on apt.wm.org T293879 |
[production] |
| 08:31 |
<urbanecm@deploy1002> |
Synchronized php-1.38.0-wmf.5/extensions/GrowthExperiments/maintenance: 91316ed5714c4426a29fefded5c4db08dbba48bb: Add purgeExpiredMentorStatus.php (T280307) (duration: 00m 56s) |
[production] |
| 08:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 07:21 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 07:07 |
<effie> |
pool mw1319 and mw1312 |
[production] |
| 07:05 |
<effie> |
pool wtp1026.eqiad.wmnet |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17606 and previous config saved to /var/cache/conftool/dbconfig/20211026-063647-root.json |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17605 and previous config saved to /var/cache/conftool/dbconfig/20211026-062144-root.json |
[production] |
| 06:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17604 and previous config saved to /var/cache/conftool/dbconfig/20211026-060640-root.json |
[production] |
| 05:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17603 and previous config saved to /var/cache/conftool/dbconfig/20211026-055136-root.json |
[production] |
| 05:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17602 and previous config saved to /var/cache/conftool/dbconfig/20211026-053633-root.json |
[production] |
| 05:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1109 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17601 and previous config saved to /var/cache/conftool/dbconfig/20211026-052129-root.json |
[production] |
| 02:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:06 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 01:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 01:24 |
<krinkle@deploy1002> |
Synchronized wmf-config/logging.php: I0211e1c77 (duration: 00m 55s) |
[production] |
| 01:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
|
2021-10-25
§
|
| 23:12 |
<catrope@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s) |
[production] |
| 23:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 22:57 |
<ryankemper> |
[wcqs] Downtimed `wcqs*` until roughly a week from now (while we setup oauth) |
[production] |
| 22:53 |
<legoktm> |
uploaded PHP 7.4.25 to apt.wm.o (DSA-4992-1) |
[production] |
| 22:44 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@e908052] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 22:30 |
<ryankemper@deploy1002> |
Finished deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS (duration: 03m 04s) |
[production] |
| 22:27 |
<ryankemper@deploy1002> |
Started deploy [wdqs/wdqs@13448f1] (wcqs): Deploy 0.3.90 to WCQS |
[production] |
| 21:53 |
<mutante> |
new project language "pwn" added - Paiwan is a native language of Taiwan, spoken by the Paiwan, a Taiwanese indigenous people. T292415 |
[production] |
| 21:52 |
<mutante> |
new project language "ami" added - Sowal no 'Amis is the Formosan language of the 'Amis (or Ami), an indigenous people living along the east coast of Taiwan. - T292414 |
[production] |
| 21:50 |
<mutante> |
log authdns1001 (DNS) - sudo authdns-update, add new project language "ami" (Amis) for T292414 - edited langlist.tmpl which regenerates all project zones |
[production] |
| 21:40 |
<mutante> |
authdns1001 (DNS) - sudo authdns-update, add new project language "pwn" (Paiwan) for T292415 |
[production] |
| 19:47 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2255.codfw.wmnet with reason: DRAC upgrade |
[production] |
| 19:47 |
<mutante> |
mw2255 - depooled=inactive (incl "dsh groups"), shut down physically for T283582 - can be worked on anytime |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2255.codfw.wmnet |
[production] |
| 19:42 |
<mutante> |
icinga - ACKing all unhandled CRIT alerts on hosts with "dev" or "test" in their name, regardless of notifications being disabled or not. just so that we get more signal than noise in actual unhandled CRITs in web UI |
[production] |
| 19:40 |
<mutante> |
cumin2002 - sudo systemctl reset-failed to clear Icinga alert about failed but (now) non-existing service database-backups-snapshots.service, assuming it's a case of "only in active DC" |
[production] |
| 19:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:12 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db1112.eqiad.wmnet with reason: hardware fail |
[production] |
| 19:07 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Temporarily move mw groups to db1123 T294295', diff saved to https://phabricator.wikimedia.org/P17597 and previous config saved to /var/cache/conftool/dbconfig/20211025-190717-kormat.json |
[production] |
| 19:06 |
<mutante> |
db1112 - powercycling |
[production] |
| 19:04 |
<legoktm@cumin1001> |
dbctl commit (dc=all): 'Depool db1112 (T294295)', diff saved to https://phabricator.wikimedia.org/P17596 and previous config saved to /var/cache/conftool/dbconfig/20211025-190436-legoktm.json |
[production] |
| 18:41 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<jforrester@deploy1002> |
Synchronized php-1.38.0-wmf.5/extensions/timeline/includes/Timeline.php: Backport: [[gerrit:734312|Input may be null when rendering a self-closing tag `<timeline />` (T294020)]] (duration: 00m 55s) |
[production] |