|
2021-04-08
§
|
| 08:03 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be2028.codfw.wmnet |
[production] |
| 07:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: Repool es1023', diff saved to https://phabricator.wikimedia.org/P15237 and previous config saved to /var/cache/conftool/dbconfig/20210408-075457-root.json |
[production] |
| 07:50 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1020.eqiad.wmnet |
[production] |
| 07:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove weight from es5 master', diff saved to https://phabricator.wikimedia.org/P15236 and previous config saved to /var/cache/conftool/dbconfig/20210408-074911-marostegui.json |
[production] |
| 07:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1175 for schema change', diff saved to https://phabricator.wikimedia.org/P15235 and previous config saved to /var/cache/conftool/dbconfig/20210408-074524-marostegui.json |
[production] |
| 07:42 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1020.eqiad.wmnet |
[production] |
| 07:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: Repool es1023', diff saved to https://phabricator.wikimedia.org/P15234 and previous config saved to /var/cache/conftool/dbconfig/20210408-073953-root.json |
[production] |
| 07:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: Repool es1023', diff saved to https://phabricator.wikimedia.org/P15233 and previous config saved to /var/cache/conftool/dbconfig/20210408-072450-root.json |
[production] |
| 07:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: Repool es1023', diff saved to https://phabricator.wikimedia.org/P15232 and previous config saved to /var/cache/conftool/dbconfig/20210408-070946-root.json |
[production] |
| 06:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1023 to upgrade kernel and mysql, remove weight from es1021, to leave it as it was yesterday T279281', diff saved to https://phabricator.wikimedia.org/P15231 and previous config saved to /var/cache/conftool/dbconfig/20210408-065627-marostegui.json |
[production] |
| 06:44 |
<elukey@deploy1002> |
Finished deploy [analytics/refinery@1dbbd3d] (hadoop-test): (no justification provided) (duration: 02m 20s) |
[production] |
| 06:41 |
<elukey@deploy1002> |
Started deploy [analytics/refinery@1dbbd3d] (hadoop-test): (no justification provided) |
[production] |
| 06:33 |
<marostegui> |
Stop MySQL on db1111 to clone db1177 T275633 |
[production] |
| 06:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111 to clone db1177 T275633', diff saved to https://phabricator.wikimedia.org/P15229 and previous config saved to /var/cache/conftool/dbconfig/20210408-063331-marostegui.json |
[production] |
| 06:01 |
<kart_> |
Updated cxserver to 2021-04-07-062518-production (T278141, T263139, T271711, T201491, T240525, T207662) |
[production] |
| 05:58 |
<kartik@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:54 |
<kartik@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . |
[production] |
| 05:43 |
<kartik@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . |
[production] |
| 02:50 |
<AaronSchulz> |
Restarted importMissingLocalNames.php (mwmaint 1002, wiki=metawiki,batch-size=1000) |
[production] |
|
2021-04-07
§
|
| 23:38 |
<ejegg> |
updated payments-wiki from b06009c099 to 70f5163816, |
[production] |
| 23:35 |
<cstone> |
civicrm revision changed from eb9379daa3 to fdb4f90c74 |
[production] |
| 23:10 |
<urbanecm@deploy1002> |
Synchronized wmf-config/Wikibase.php: 321bf91da7e823f026c2c2bdcc57d8cf60a52ba5: Wikibase: sample function call counters at 1:100 (T277817) (duration: 01m 08s) |
[production] |
| 22:49 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@1dbbd3d] (hadoop-test): Regular analytics weekly train TEST retry1 [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] (duration: 01m 47s) |
[production] |
| 22:48 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@1dbbd3d] (hadoop-test): Regular analytics weekly train TEST retry1 [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] |
[production] |
| 22:33 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@1dbbd3d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] (duration: 04m 15s) |
[production] |
| 22:29 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@1dbbd3d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] |
[production] |
| 22:29 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@1dbbd3d] (thin): Regular analytics weekly train THIN [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] (duration: 00m 07s) |
[production] |
| 22:29 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@1dbbd3d] (thin): Regular analytics weekly train THIN [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] |
[production] |
| 22:28 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@1dbbd3d]: Regular analytics weekly train [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] (duration: 42m 54s) |
[production] |
| 22:03 |
<Amir1> |
clearing watchlist of bots in wikidatawiki (https://www.wikidata.org/w/index.php?title=Wikidata:Project_chat&oldid=1397670734#Clean_up_watchlist_of_bots) |
[production] |
| 22:01 |
<legoktm> |
deployed patch for T279451 (part 2) |
[production] |
| 21:45 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@1dbbd3d]: Regular analytics weekly train [analytics/refinery@1dbbd3dfa996d2e970eb1cbc0a63d53040d4e3a3] |
[production] |
| 21:31 |
<legoktm> |
deployed patch for T279451 |
[production] |
| 21:22 |
<mutante> |
mw2397 through mw2402 - pooled as new API appservers after scap pull and all monitoring green (T278396) |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw240[0-2].codfw.wmnet |
[production] |
| 21:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw239[7-9].codfw.wmnet |
[production] |
| 21:05 |
<mutante> |
mw2397 - mw2402 - scap pull |
[production] |
| 21:04 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw240[0-2].codfw.wmnet |
[production] |
| 21:04 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw239[7-9].codfw.wmnet |
[production] |
| 21:04 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw240[0-2].codfw.wmnet |
[production] |
| 21:03 |
<Amir1> |
clearing watchlist of bots in enwiki (https://en.wikipedia.org/w/index.php?title=Wikipedia:Bots/Noticeboard&oldid=1016563560#Clearing_bot_watchlists) |
[production] |
| 21:02 |
<dzahn@cumin1001> |
conftool action : set/weight=30; selector: name=mw239[7-9].codfw.wmnet |
[production] |
| 20:58 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw[2400-2401].codfw.wmnet with reason: new_install |
[production] |
| 20:56 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw[2400-2401].codfw.wmnet with reason: new_install |
[production] |
| 20:56 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw[2397-2399].codfw.wmnet with reason: new_install |
[production] |
| 20:55 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw[2397-2399].codfw.wmnet with reason: new_install |
[production] |
| 20:54 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:54 |
<mutante> |
mw2397 - mw2402 - rebooting |
[production] |
| 20:51 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |