2019-01-17
§
|
11:16 |
<onimisionipe> |
shutdown elastic103[0-5] to prepare for T213859 |
[production] |
11:09 |
<elukey> |
stop eventlogging on eventlog1002 and eventlogging replication on db1108 as prep step for db1107 maintenance |
[production] |
10:55 |
<marostegui> |
Lag will be generated on labs due to maintenance on sanitarium db masters |
[production] |
10:54 |
<marostegui> |
Stop MySQL on db1082 db1081 db1080 db1079 db1075 db1074 es1012 es1011 - T213748 |
[production] |
10:54 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool DBs on A2 rack T213748 (duration: 00m 54s) |
[production] |
10:39 |
<moritzm> |
installing libcaca security updates |
[production] |
10:30 |
<arturo> |
T213859 icinga downtime cloudservices1004 for 1 day |
[production] |
10:29 |
<moritzm> |
installing ruby-loofah security updates |
[production] |
10:09 |
<marostegui> |
Stop MySQL on db1103:3312 and db1103:3314, also poweroff the server - T213859 |
[production] |
10:08 |
<moritzm> |
installing krb5 security updates on trusty |
[production] |
10:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1103 - T213859 (duration: 00m 53s) |
[production] |
09:59 |
<marostegui> |
Poweroff dbproxy1001 dbproxy1002 dbproxy1003 for a3 maintenance - T213859 |
[production] |
09:25 |
<marostegui> |
Poweroff dbstore1003 for hw maintenance T213859 |
[production] |
09:24 |
<moritzm> |
power off graphite1003 for later hw maintenance (T213859) |
[production] |
09:18 |
<marostegui> |
Deploy schema change on db1095:3313 - T85757 |
[production] |
09:02 |
<vgutierrez> |
rolling NIC firmware upgrade cp[1081-1090] - T203194 |
[production] |
08:42 |
<jijiki> |
Enabling puppet on rdb1005 and switch redis::misc::master to rdb1006 - T213859 |
[production] |
08:37 |
<moritzm> |
installing remaining systemd security updates on stretch |
[production] |
08:32 |
<jijiki> |
Restarting nutcracker on scb100* for 484572 - T213859 |
[production] |
08:32 |
<jynus> |
stop, upgrade and restart db1075 |
[production] |
08:31 |
<marostegui> |
Deploy schema change on s3 codfw, lag will be generated - T85757 |
[production] |
08:28 |
<marostegui> |
Drop table tag_summary from enwiki - T212255 |
[production] |
08:24 |
<jijiki> |
Disabling puppet on rdb1005 and switch redis::misc::master to rdb1006 - T213859 |
[production] |
07:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Increase weight for db1123 (duration: 00m 53s) |
[production] |
07:20 |
<marostegui> |
Change thread_pool_stall_limit on db1075 and db1078 - T213858 |
[production] |
07:18 |
<marostegui> |
Enable GTID on db1075 - T213858 |
[production] |
07:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove s3 ready only T213858 (duration: 00m 30s) |
[production] |
07:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover s3master eqiad from db1075 to db1078 T213858 (duration: 00m 30s) |
[production] |
07:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set s3 on read-only T213858 (duration: 00m 31s) |
[production] |
07:00 |
<marostegui> |
Start s3 failover T213858 |
[production] |
06:30 |
<marostegui> |
Disable puppet on db1075 and db1078 - T213858 |
[production] |
06:26 |
<marostegui> |
Enable GTID back on all hosts but db1075 db1078 - T213858 |
[production] |
06:19 |
<marostegui> |
Change s3 topology to get ready for s3 failover - T213858 |
[production] |
06:14 |
<marostegui> |
Disable gtid on s3 hosts - T213858 |
[production] |
06:10 |
<marostegui> |
Downtime s3 hosts for 2 hours - T213858 |
[production] |
04:12 |
<ppchelko@deploy1001> |
Finished deploy [mobileapps/deploy@89c4d8d]: revert new summary (duration: 01m 55s) |
[production] |
04:10 |
<ppchelko@deploy1001> |
Started deploy [mobileapps/deploy@89c4d8d]: revert new summary |
[production] |
04:02 |
<cdanis@deploy1001> |
Started restart [parsoid/deploy@4b82683]: (no justification provided) |
[production] |
2019-01-16
§
|
23:25 |
<ppchelko@deploy1001> |
Finished deploy [recommendation-api/deploy@0ff39e2]: Deployment attempt with decreased worker count (duration: 04m 08s) |
[production] |
23:21 |
<ppchelko@deploy1001> |
Started deploy [recommendation-api/deploy@0ff39e2]: Deployment attempt with decreased worker count |
[production] |
23:10 |
<Krinkle> |
krinkle@tungsten:/srv/: rm -rf xhprof; for T196406 |
[production] |
21:35 |
<ppchelko@deploy1001> |
Finished deploy [recommendation-api/deploy@c1b6b32]: Rollback update to 1a1f824 (duration: 01m 59s) |
[production] |
21:33 |
<ppchelko@deploy1001> |
Started deploy [recommendation-api/deploy@c1b6b32]: Rollback update to 1a1f824 |
[production] |
21:29 |
<ppchelko@deploy1001> |
deploy aborted: log (duration: 00m 02s) |
[production] |
21:29 |
<ppchelko@deploy1001> |
Started deploy [recommendation-api/deploy@da83637]: log |
[production] |
21:28 |
<bmansurov@deploy1001> |
Finished deploy [recommendation-api/deploy@da83637]: Update to 1a1f824 (duration: 06m 14s) |
[production] |
21:22 |
<bmansurov@deploy1001> |
Started deploy [recommendation-api/deploy@da83637]: Update to 1a1f824 |
[production] |
21:17 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@6b344ca]: Update mobileapps to 258d76b page summary changes (duration: 06m 31s) |
[production] |
21:10 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@6b344ca]: Update mobileapps to 258d76b page summary changes |
[production] |
20:20 |
<dduvall@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.13 (duration: 00m 51s) |
[production] |