|
2019-12-11
§
|
| 08:10 |
<Urbanecm> |
Clear signup throttle for IP 195.113.183.5 |
[production] |
| 08:10 |
<urbanecm@deploy1001> |
Synchronized wmf-config/throttle.php: f62edfe: Add throttle rule for Czech student workshop (duration: 01m 02s) |
[production] |
| 08:04 |
<elukey> |
powercycle cp3055 - down since hours ago, no ssh, no mgmt serial console usable |
[production] |
| 08:02 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp3055.esams.wmnet |
[production] |
| 07:54 |
<marostegui> |
Compress cx_corpora on db1140:3320 T240325 |
[production] |
| 07:51 |
<marostegui> |
Upgrade db2096 (x1 codfw master) |
[production] |
| 06:59 |
<marostegui> |
Compress cx_corpora on db2096 T240325 |
[production] |
| 06:57 |
<marostegui> |
Upgrade x1 codfw |
[production] |
| 06:55 |
<eileen> |
process-control config revision is f34450e3ba - turn off dedupe to do Benevity import |
[production] |
| 06:46 |
<effie> |
restart graphoid on scb1001 |
[production] |
| 06:44 |
<marostegui> |
Stop mysql on db1124 for upgrade |
[production] |
| 06:28 |
<marostegui> |
Stop MySQL on db2070 - T239684 |
[production] |
| 06:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db2070 from config as it will be decommissioned T239684', diff saved to https://phabricator.wikimedia.org/P9848 and previous config saved to /var/cache/conftool/dbconfig/20191211-062700-marostegui.json |
[production] |
| 06:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2070 from config T239684 (duration: 01m 08s) |
[production] |
| 06:24 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2070 from config T239684 (duration: 01m 18s) |
[production] |
| 06:22 |
<marostegui> |
Remove db2070 from tendril and zarcillo T239684 |
[production] |
| 06:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:07 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:00 |
<marostegui> |
Compress cx_corpora on db2131 T240325 |
[production] |
| 05:45 |
<marostegui> |
Deploy schema change on dbstore1004:3314 |
[production] |
| 00:54 |
<eileen> |
rocess-control config revision is 3f60e8fe9e |
[production] |
| 00:46 |
<eileen> |
civicrm revision changed from b519d4fb73 to 7b971ac58c, config revision is 9fb34fd93a |
[production] |
| 00:39 |
<tgr@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:546894|Add growthexperiments dblist, for puppet usage (T208369)]] (duration: 01m 00s) |
[production] |
| 00:37 |
<tgr@deploy1001> |
Synchronized wmf-config/config: SWAT: [[gerrit:546894|Add growthexperiments dblist, for puppet usage (T208369)]] (duration: 01m 01s) |
[production] |
| 00:35 |
<tgr@deploy1001> |
Synchronized dblists/growthexperiments.dblist: SWAT: [[gerrit:546894|Add growthexperiments dblist, for puppet usage (T208369)]] (duration: 01m 02s) |
[production] |
|
2019-12-10
§
|
| 22:33 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@7c8cb9d]: Update mobileapps to 3b1ba07 (duration: 05m 58s) |
[production] |
| 22:27 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@7c8cb9d]: Update mobileapps to 3b1ba07 |
[production] |
| 21:25 |
<marxarelli> |
promoted group0 to 1.35.0-wmf.10 cc: T233858 |
[production] |
| 21:23 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.10 |
[production] |
| 21:16 |
<dduvall@deploy1001> |
Finished scap: testwiki to php-1.35.0-wmf.10 and rebuild l10n cache (duration: 37m 20s) |
[production] |
| 20:39 |
<dduvall@deploy1001> |
Started scap: testwiki to php-1.35.0-wmf.10 and rebuild l10n cache |
[production] |
| 20:38 |
<dduvall@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.5 (duration: 01m 36s) |
[production] |
| 20:37 |
<cdanis> |
✔️ cdanis@mw1323.eqiad.wmnet ~ 🕞🍵 sudo renice -n -19 `pidof mcrouter` |
[production] |
| 20:36 |
<dduvall@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.3 (duration: 01m 52s) |
[production] |
| 20:33 |
<dduvall@deploy1001> |
Pruned MediaWiki: 1.35.0-wmf.4 (duration: 06m 40s) |
[production] |
| 20:31 |
<cdanis@cumin2001> |
conftool action : set/weight=20; selector: cluster=appserver,dc=eqiad,service=nginx,name=mw132[34].* |
[production] |
| 20:31 |
<cdanis@cumin2001> |
conftool action : set/weight=20; selector: cluster=appserver,dc=eqiad,service=apache2,name=mw132[34].* |
[production] |
| 19:45 |
<_joe_> |
restarting php-fpm on mw1332,1319 (high latency) |
[production] |
| 19:01 |
<marxarelli> |
cutting branch for 1.35.0-wmf.10 cc: T233858 |
[production] |
| 18:22 |
<rlazarus> |
restarted php7.2-fpm on mw1328 |
[production] |
| 18:19 |
<bblack> |
cp2007: restart traffic-manager.service, seems to have been left in a bad state? |
[production] |
| 18:09 |
<jeh> |
imported ceph nautilus debian packages into buster-wikimedia/thirdparty/ceph-nautilus-buster T239917 |
[production] |
| 18:08 |
<rlazarus> |
restarting php7.2-fpm on all remaining slow hosts except 1328, held back for investigation: mw[1333,1331,1322,1327,1325] |
[production] |
| 17:54 |
<_joe_> |
repooled mw1322, just depooling solved the issue |
[production] |
| 17:48 |
<_joe_> |
depool mw1322 for debugging |
[production] |
| 17:44 |
<rlazarus> |
mw1322$ php7adm /apcu-free |
[production] |
| 17:22 |
<andrew-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Cite: SWAT: [[gerrit:556218|Catch one last undefined index (T240248)]] (duration: 01m 02s) |
[production] |
| 17:05 |
<bblack> |
lvs100{14,16} - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 17:00 |
<bblack> |
lvs200[25] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 16:50 |
<bblack> |
lvs500[23] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |