|
2017-01-26
ยง
|
| 20:30 |
<andrewbogott> |
refreshing logins on wikitech |
[production] |
| 19:13 |
<elukey> |
restore analytics1001 as RM and HDFS masters |
[production] |
| 18:56 |
<otto@tin> |
Finished deploy [eventstreams/deploy@f1a1866]: (no message) (duration: 03m 16s) |
[production] |
| 18:52 |
<otto@tin> |
Starting deploy [eventstreams/deploy@f1a1866]: (no message) |
[production] |
| 18:36 |
<elukey> |
restarting Yarn node managers on an102[89] and an103[01], impacted by the switch restart |
[production] |
| 18:32 |
<paravoid> |
starting pybal on lvs1001/lvs1002/lvs1003 |
[production] |
| 18:15 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1055, 56, 57, 59 (duration: 00m 54s) |
[production] |
| 18:14 |
<paravoid> |
rebooting newly provisioned asw-c2-eqiad to enable mixed mode |
[production] |
| 17:57 |
<elukey> |
boostrapping aqs1007-a cassandra instance |
[production] |
| 17:51 |
<paravoid> |
replacing asw-c2-eqiad |
[production] |
| 17:46 |
<paravoid> |
stopping pybal on lvs1001/lvs1002/lvs1003 |
[production] |
| 17:34 |
<elukey@tin> |
Finished deploy [analytics/aqs/deploy@5917fd4]: (no message) (duration: 02m 25s) |
[production] |
| 17:31 |
<elukey@tin> |
Starting deploy [analytics/aqs/deploy@5917fd4]: (no message) |
[production] |
| 15:43 |
<bblack> |
cache_misc puppet re-enabled and up to date |
[production] |
| 15:37 |
<godog> |
bounce uwsgi on graphite1003 with less workers - T155872 |
[production] |
| 15:35 |
<moritzm> |
installing gnupg2 updates from jessie point update |
[production] |
| 15:15 |
<akosiaris> |
T156242 add /dev/sdb partitions to mdadm devices |
[production] |
| 15:15 |
<bblack> |
puppet disabled on cache_misc for merging complicated stuff |
[production] |
| 15:06 |
<moritzm> |
installing gnupg updates from jessie point update |
[production] |
| 14:52 |
<jynus> |
stopping mysql on db1048 T156373 |
[production] |
| 14:29 |
<zeljkof> |
finished with eu swat |
[production] |
| 14:28 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:334134|IP Cap Lift for Edit-a-Thon (T156258)]] [[gerrit:334156|[throttle] Her Girl Friday + Lenny Unconference / Editathon in NYC, 2017-01-28 (T156278)]] (duration: 00m 41s) |
[production] |
| 13:54 |
<godog> |
delete labs 'instances' graphite three for data >30d, graphite low on disk space |
[production] |
| 13:53 |
<elukey> |
restarting cassandra on aqs100[56] to complete the openjdk update |
[production] |
| 13:32 |
<moritzm> |
rolling restart of maps cluster in eqiad |
[production] |
| 13:08 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1054 - T156225 (duration: 00m 40s) |
[production] |
| 13:05 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.29.0-wmf.9 T156310 |
[production] |
| 13:02 |
<godog> |
reboot ms-be1013 |
[production] |
| 12:54 |
<elukey> |
restarting the aqs1004-b casandra instance to pick up the new openjdk (last test before complete rollout) |
[production] |
| 12:28 |
<elukey> |
restarting the aqs1004-a casandra instance to pick up the new openjdk |
[production] |
| 12:28 |
<moritzm> |
upgrading java on maps cluster, rolling restart of maps cluster in codfw |
[production] |
| 12:17 |
<hashar@tin> |
Synchronized php-1.29.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevision.php: Fix fatal in prod caused by deprecated function removal T156310 (duration: 00m 41s) |
[production] |
| 12:04 |
<moritzm> |
installing java security updates on aqs cluster |
[production] |
| 11:12 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: FlaggedRevs is broken in wmf.9 causing blank pages. T156356 T156310 |
[production] |
| 09:54 |
<marostegui> |
Disable semi-sync on db1057 old s1 master - https://phabricator.wikimedia.org/T156008 |
[production] |
| 09:39 |
<marostegui> |
Enable semi-sync replication on db1052 (s1 master) - T156008 |
[production] |
| 09:04 |
<marostegui> |
Change dbstore1002 to replicate from the new s1 master db1052 - T156008 |
[production] |
| 08:57 |
<marostegui> |
Change db1047 to replicate from the new s1 master db1052 - T156008 |
[production] |
| 08:48 |
<marostegui> |
Change db1069 to replicate from the new s1 master db1052 - T156008 |
[production] |
| 08:48 |
<jynus> |
deploying dns CNAME updates due to master swithover |
[production] |
| 07:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Change s1 master to db1057 - T156008 (duration: 00m 20s) |
[production] |
| 07:18 |
<jynus> |
last message was master of db1057 |
[production] |
| 07:18 |
<jynus> |
change master of db1052 from db2016 to db1052 |
[production] |
| 06:49 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1052 - T156008 (duration: 00m 31s) |
[production] |
| 03:50 |
<mutante> |
rsyncing apt.wikimedia.org data from carbon to install2001 (T84380) |
[production] |
| 02:07 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jan 26 02:07:05 UTC 2017 (duration 4m 42s) |
[production] |
| 02:02 |
<l10nupdate@tin> |
LocalisationUpdate failed (1.29.0-wmf.9) at 2017-01-26 02:02:23+00:00 |
[production] |
| 02:02 |
<l10nupdate@tin> |
LocalisationUpdate failed (1.29.0-wmf.8) at 2017-01-26 02:02:23+00:00 |
[production] |
| 02:02 |
<twentyafterfour> |
phabricator update complete |
[production] |
| 01:47 |
<twentyafterfour> |
upgrading phabricator, downtime should be minimal but expect the service to be offline for up to a few minutes |
[production] |