|
2021-11-03
ยง
|
| 19:35 |
<mutante> |
depooled wcqs2003 (pooled=inactive) because Icinga alerts that servers are down but pooled. not in production yet but issues (T294961) |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=wcqs2003.codfw.wmnet |
[production] |
| 19:33 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=wcqs2003.codfw.wmnet |
[production] |
| 19:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:26 |
<mmandere> |
pool cp4035.ulsfo.wmnet - T290694 |
[production] |
| 19:19 |
<dduvall> |
1.38.0-wmf.7 now on group0. no new errors. leaving ~ 30 minutes before promoting group1 (T293948) |
[production] |
| 19:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:15 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 19:15 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:10 |
<tgr> |
UTC evening deploys done |
[production] |
| 19:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:01 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:59 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 18:55 |
<razzi@cumin1001> |
START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 |
[production] |
| 18:51 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster |
[production] |
| 18:51 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:40 |
<legoktm> |
re-enabling puppet on lists1001 |
[production] |
| 18:38 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:34 |
<urbanecm> |
Purge https://en.wikipedia.org/.well-known/assetlinks.json, https://www.wikipedia.org/.well-known/assetlinks.json and https://wikipedia.org/.well-known/assetlinks.json (T294776) |
[production] |
| 18:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:24 |
<volans> |
rebooting ganeti-test2002 with fixed /etc/network/interfaces |
[production] |
| 18:22 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'recommendation-api' for release 'production' . |
[production] |
| 18:22 |
<urbanecm@deploy1002> |
Synchronized docroot/wikipedia.org/: 2331d061b95ba3fc4de8844008fac93ce18f9063: Add Android site association file (T294776) (duration: 01m 02s) |
[production] |
| 18:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:18 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' . |
[production] |
| 18:17 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 18:15 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 18:15 |
<ppchelko@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:710126|Clean up temporary variable wgMathUseRestBase (T274436)]] (duration: 01m 02s) |
[production] |
| 18:15 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 18:15 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 18:13 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 18:12 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'similar-users' for release 'main' . |
[production] |
| 18:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:09 |
<ppchelko@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:710126|Clean up temporary variable wgMathUseRestBase (T274436)]] (duration: 01m 03s) |
[production] |
| 18:09 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . |
[production] |
| 18:08 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . |
[production] |
| 18:08 |
<Amir1> |
ran set session sql_log_bin=0; RENAME TABLE wb_changes_dispatch TO T294121_DROP_wb_changes_dispatch; on db1111 (T294121) |
[production] |
| 18:07 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox-media' for release 'main' . |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:06 |
<ppchelko@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:736032|Remove hook set for incident reponse in 2020]] (duration: 01m 03s) |
[production] |
| 18:04 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' . |
[production] |
| 18:03 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' . |
[production] |
| 18:02 |
<jelto@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 17:50 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp4035.ulsfo.wmnet with OS buster |
[production] |
| 17:49 |
<vgutierrez> |
update codfw cp instances to ATS 8.0.8-1wm5 - T294897 |
[production] |
| 17:48 |
<mmandere> |
depool cp4035.ulsfo.wmnet - T290694 |
[production] |
| 17:47 |
<topranks> |
adding BGP peering session to "Liquid Telecommunications" AS30844 on cr2-esams (AMS-IX) |
[production] |