|
2020-11-05
§
|
| 09:21 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1031 (re)pooling @ 50%: Slowly pool es1031 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13219 and previous config saved to /var/cache/conftool/dbconfig/20201105-091350-root.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 50%: Slowly pool es1030 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13218 and previous config saved to /var/cache/conftool/dbconfig/20201105-091341-root.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1029 (re)pooling @ 50%: Slowly pool es1029 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13217 and previous config saved to /var/cache/conftool/dbconfig/20201105-091329-root.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1031 (re)pooling @ 25%: Slowly pool es1031 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13216 and previous config saved to /var/cache/conftool/dbconfig/20201105-085846-root.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 25%: Slowly pool es1030 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13215 and previous config saved to /var/cache/conftool/dbconfig/20201105-085838-root.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1029 (re)pooling @ 25%: Slowly pool es1029 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13214 and previous config saved to /var/cache/conftool/dbconfig/20201105-085826-root.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1031 (re)pooling @ 10%: Slowly pool es1031 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13213 and previous config saved to /var/cache/conftool/dbconfig/20201105-084343-root.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1030 (re)pooling @ 10%: Slowly pool es1030 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13212 and previous config saved to /var/cache/conftool/dbconfig/20201105-084334-root.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1029 (re)pooling @ 10%: Slowly pool es1029 after being recloned T261717', diff saved to https://phabricator.wikimedia.org/P13211 and previous config saved to /var/cache/conftool/dbconfig/20201105-084323-root.json |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1090:3312', diff saved to https://phabricator.wikimedia.org/P13210 and previous config saved to /var/cache/conftool/dbconfig/20201105-084250-marostegui.json |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1090:3312', diff saved to https://phabricator.wikimedia.org/P13209 and previous config saved to /var/cache/conftool/dbconfig/20201105-083304-marostegui.json |
[production] |
| 08:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: After regenerating table stats', diff saved to https://phabricator.wikimedia.org/P13208 and previous config saved to /var/cache/conftool/dbconfig/20201105-083142-root.json |
[production] |
| 08:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: After regenerating table stats', diff saved to https://phabricator.wikimedia.org/P13207 and previous config saved to /var/cache/conftool/dbconfig/20201105-081638-root.json |
[production] |
| 08:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: After regenerating table stats', diff saved to https://phabricator.wikimedia.org/P13206 and previous config saved to /var/cache/conftool/dbconfig/20201105-080135-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es1031 on es3 with minimium weight after being cloned from es1017 T261717', diff saved to https://phabricator.wikimedia.org/P13205 and previous config saved to /var/cache/conftool/dbconfig/20201105-075625-marostegui.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es1030 on es2 with minimium weight after being cloned from es1013 T261717', diff saved to https://phabricator.wikimedia.org/P13204 and previous config saved to /var/cache/conftool/dbconfig/20201105-075507-marostegui.json |
[production] |
| 07:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Pool es1029 on es1 with minimium weight after being cloned from es1016 T261717', diff saved to https://phabricator.wikimedia.org/P13203 and previous config saved to /var/cache/conftool/dbconfig/20201105-075358-marostegui.json |
[production] |
| 07:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: After regenerating table stats', diff saved to https://phabricator.wikimedia.org/P13202 and previous config saved to /var/cache/conftool/dbconfig/20201105-074631-root.json |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1129 T267216', diff saved to https://phabricator.wikimedia.org/P13201 and previous config saved to /var/cache/conftool/dbconfig/20201105-072352-marostegui.json |
[production] |
| 07:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1016 (re)pooling @ 100%: After cloning es1029 T261717', diff saved to https://phabricator.wikimedia.org/P13200 and previous config saved to /var/cache/conftool/dbconfig/20201105-071017-root.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1013 (re)pooling @ 100%: After cloning es1030 T261717', diff saved to https://phabricator.wikimedia.org/P13199 and previous config saved to /var/cache/conftool/dbconfig/20201105-070616-root.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1017 (re)pooling @ 100%: After cloning es1031 T261717', diff saved to https://phabricator.wikimedia.org/P13198 and previous config saved to /var/cache/conftool/dbconfig/20201105-070610-root.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1016 (re)pooling @ 75%: After cloning es1029 T261717', diff saved to https://phabricator.wikimedia.org/P13197 and previous config saved to /var/cache/conftool/dbconfig/20201105-065514-root.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1013 (re)pooling @ 75%: After cloning es1030 T261717', diff saved to https://phabricator.wikimedia.org/P13196 and previous config saved to /var/cache/conftool/dbconfig/20201105-065113-root.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1017 (re)pooling @ 75%: After cloning es1031 T261717', diff saved to https://phabricator.wikimedia.org/P13195 and previous config saved to /var/cache/conftool/dbconfig/20201105-065107-root.json |
[production] |
| 06:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1016 (re)pooling @ 50%: After cloning es1029 T261717', diff saved to https://phabricator.wikimedia.org/P13193 and previous config saved to /var/cache/conftool/dbconfig/20201105-064010-root.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1013 (re)pooling @ 50%: After cloning es1030 T261717', diff saved to https://phabricator.wikimedia.org/P13192 and previous config saved to /var/cache/conftool/dbconfig/20201105-063610-root.json |
[production] |
| 06:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1017 (re)pooling @ 50%: After cloning es1031 T261717', diff saved to https://phabricator.wikimedia.org/P13191 and previous config saved to /var/cache/conftool/dbconfig/20201105-063603-root.json |
[production] |
| 06:34 |
<elukey> |
truncate application_1601916545561_129457's taskmanager.log (~600G) on an-worker1113 due to partition 'e' full |
[production] |
| 06:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1016 (re)pooling @ 25%: After cloning es1029 T261717', diff saved to https://phabricator.wikimedia.org/P13190 and previous config saved to /var/cache/conftool/dbconfig/20201105-062507-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1013 (re)pooling @ 25%: After cloning es1027 T261717', diff saved to https://phabricator.wikimedia.org/P13189 and previous config saved to /var/cache/conftool/dbconfig/20201105-062454-root.json |
[production] |
| 06:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1017 (re)pooling @ 25%: After cloning es1028 T261717', diff saved to https://phabricator.wikimedia.org/P13188 and previous config saved to /var/cache/conftool/dbconfig/20201105-062446-root.json |
[production] |
| 01:57 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@6913407] (thin): Regular analytics weekly train THIN [analytics/refinery@6913407] (duration: 00m 08s) |
[production] |
| 01:56 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@6913407] (thin): Regular analytics weekly train THIN [analytics/refinery@6913407] |
[production] |
| 01:56 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407] (duration: 08m 34s) |
[production] |
| 01:47 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407] |
[production] |
|
2020-11-04
§
|
| 20:36 |
<Urbanecm> |
Late B&C Morning window completed, deployment host is clear |
[production] |
| 20:36 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: ee0ba541fa55f6707276fdc5bd3f032cb9be3e60: Disable the search in header A/B test (T265333) (duration: 01m 06s) |
[production] |
| 20:33 |
<ejegg> |
updated payments-wiki from 1ad4ba9639 to 388490e86d |
[production] |
| 20:31 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Migrate NewcomerTask event stream to EventGate on testwiki - T259163 (duration: 01m 07s) |
[production] |
| 20:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 82579bf9d71bd3c9d97da0132ce8d92a8863da5b: Enable wgImagePreconnect on remaining wikis (T123582) (duration: 01m 06s) |
[production] |
| 20:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: d2a57725f8f6fdaa3f40c834e84b43a0260077f2: Enable DiscussionTools as a beta feature on almost all Wikipedias (T266303) (duration: 01m 07s) |
[production] |
| 20:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: fb5c03262c20b5e99b3c2f6e91abb024f12da1f5: Enable wgCheckUserLogLogins at all wikis but loginwiki (T253802) (duration: 01m 08s) |
[production] |
| 19:59 |
<brennen@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.16 (duration: 62m 44s) |
[production] |
| 18:57 |
<brennen@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.16 |
[production] |
| 18:52 |
<brennen@deploy1001> |
Pruned MediaWiki: 1.36.0-wmf.10 (duration: 27m 38s) |
[production] |
| 18:51 |
<Urbanecm> |
Strip 2FA for Mark83 at SUL (T267257) |
[production] |
| 18:20 |
<elukey> |
restart memcached on mc1036 to pick up new settings (see https://gerrit.wikimedia.org/r/639099) |
[production] |
| 18:15 |
<hknust> |
holger@mwmaint1002 END - Run updateRestrictions.php |
[production] |