|
2017-02-01
ยง
|
| 19:10 |
<ostriches> |
phabricator: now in read-only mode |
[production] |
| 19:08 |
<jynus> |
scheduling 10 minutes of emergency downtime on phabricator |
[production] |
| 19:06 |
<mobrovac> |
restbase deploy end of 96a641aa |
[production] |
| 18:49 |
<joal@tin> |
Finished deploy [analytics/refinery@2b9a70a]: (no justification provided) (duration: 02m 33s) |
[production] |
| 18:46 |
<joal@tin> |
Started deploy [analytics/refinery@2b9a70a]: (no justification provided) |
[production] |
| 18:34 |
<mobrovac> |
restbase deploy start of 96a641aa |
[production] |
| 16:54 |
<marostegui> |
Optimize table phabricator_search.search_documentfield on db2012 - T156905 |
[production] |
| 16:41 |
<jynus> |
mariadb rolling restart of db2037, db2044, db2051, db2058, db2065 |
[production] |
| 16:20 |
<elukey> |
restarting Yarn Node Manager daemons on all the Hadoop nodes to bandaid a memory leak causing OOMs |
[production] |
| 16:18 |
<marostegui> |
Optimizing table search_documentfield on db1048 - T156905 |
[production] |
| 15:50 |
<akosiaris> |
stop ircecho for a while to weather out most of the puppet alert storm |
[production] |
| 15:46 |
<akosiaris> |
restart puppetdb on nihal (openjdk upgrade) |
[production] |
| 15:43 |
<akosiaris> |
restart puppetdb on nitrogen |
[production] |
| 15:40 |
<jynus> |
preparing db1067 for reimage to jessie |
[production] |
| 15:37 |
<moritzm> |
upgrading canary app servers to new HHVM package (initially mwdebug and mw1261) |
[production] |
| 15:17 |
<Dereckson> |
`mwscript populateCategory.php plwikisource --force` to refresh categories stats (T156670) |
[production] |
| 15:17 |
<dereckson@tin> |
Finished scap: Full scap to propagate a core namespace l10n change (duration: 40m 10s) |
[production] |
| 14:41 |
<godog> |
upgrade thumbor to 0.1.34 |
[production] |
| 14:37 |
<dereckson@tin> |
Started scap: Full scap to propagate a core namespace l10n change |
[production] |
| 14:25 |
<jynus> |
dropping and replacing events on db1057 - db1052 T156008 |
[production] |
| 14:24 |
<dereckson@tin> |
Synchronized php-1.29.0-wmf.9/languages/messages/MessagesJv.php: Update namespace localisation in Javanese (T155957) (duration: 00m 40s) |
[production] |
| 14:21 |
<dereckson@tin> |
Synchronized php-1.29.0-wmf.10/languages/messages/MessagesJv.php: Update namespace localisation in Javanese (T155957) (duration: 00m 45s) |
[production] |
| 14:12 |
<moritzm> |
uploaded hhvm 3.12.12 to carbon |
[production] |
| 14:10 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable ElectronPdfService on meta (T150943) (duration: 00m 48s) |
[production] |
| 13:39 |
<marostegui> |
Deploy alter table dbstore1002 metawiki.pagelinks - T153300 |
[production] |
| 13:38 |
<akosiaris> |
issue sudo hdparm -Y /dev/sdb on bast3001 to force a problematic drive to sleep |
[production] |
| 13:21 |
<marostegui> |
Clean up db1043 replication thread (it was replicating from db1048 which looks like an old thing) - T156905 |
[production] |
| 12:09 |
<elukey@tin> |
Finished deploy [analytics/refinery@e6254a4]: (no justification provided) (duration: 04m 41s) |
[production] |
| 12:04 |
<elukey@tin> |
Started deploy [analytics/refinery@e6254a4]: (no justification provided) |
[production] |
| 11:52 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2061 - T153300 (duration: 00m 40s) |
[production] |
| 11:25 |
<moritzm> |
removing ntfs-3g from various trusty servers |
[production] |
| 11:14 |
<godog> |
bounce leaking thumbor@8813 on thumbor1001 |
[production] |
| 11:08 |
<kartik@tin> |
Finished deploy [cxserver/deploy@0e4ae4f]: (no justification provided) (duration: 02m 04s) |
[production] |
| 11:06 |
<kartik@tin> |
Started deploy [cxserver/deploy@0e4ae4f]: (no justification provided) |
[production] |
| 07:53 |
<marostegui> |
Deploy alter table metawiki.pagelinks db2061 - T153300 |
[production] |
| 07:52 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2061 - T153300 (duration: 00m 53s) |
[production] |
| 07:43 |
<moritzm> |
rolling restart of cassandra in eqiad to pick up openjdk and NSS security updates |
[production] |
| 07:41 |
<elukey> |
bootstrapping aqs1008-a on aqs1008 (new AQS cassandra node) |
[production] |
| 07:31 |
<marostegui> |
Force WB policy on the raid controller db1072 - T156226 |
[production] |
| 07:13 |
<akosiaris> |
restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002 |
[production] |
| 07:13 |
<akosiaris> |
restart thumbor process on thumbor1001, thumbor1002, apply a different LimitNOFILE on thumbo1002 |
[production] |
| 04:17 |
<mutante> |
carbon - rsyncing entire /srv over to install2002 (T156440) |
[production] |
| 03:00 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Feb 1 03:00:32 UTC 2017 (duration 5m 35s) |
[production] |
| 02:54 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.10) (duration: 03m 42s) |
[production] |
| 02:48 |
<mutante> |
install1002, install2002 - install jessie, sign puppet certs, initial puppet run (T132757, T156440) |
[production] |
| 02:34 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.9) (duration: 11m 52s) |
[production] |
| 02:20 |
<demon@tin> |
Synchronized scap/plugins/clean.py: no-op (duration: 00m 39s) |
[production] |
| 01:19 |
<mutante> |
ganeti: create instance install2002 with 80G disk, 2G RAM (T156440) |
[production] |
| 01:15 |
<mutante> |
ganeti: install1001 - remove virtual disk 1 from instance | create instance install1002 instead (T132757) |
[production] |
| 00:57 |
<mutante> |
Ganglia is now deprecated in favor of Grafana (https://phabricator.wikimedia.org/T145659#2925104) |
[production] |