|
2017-05-04
§
|
| 07:59 |
<marostegui@naos> |
Synchronized wmf-config/db-eqiad.php: Remove tempdb2001 from config files as it will be decommissioned - T161712 (duration: 01m 07s) |
[production] |
| 07:58 |
<marostegui@naos> |
Synchronized wmf-config/db-codfw.php: Remove tempdb2001 from config files as it will be decommissioned - T161712 (duration: 01m 25s) |
[production] |
| 07:25 |
<_joe_> |
restarted cp3043 backend varnish at 7:13 UTC while trying to debug issues |
[production] |
| 06:58 |
<moritzm> |
installing freetype security updates |
[production] |
| 06:26 |
<marostegui@naos> |
Synchronized wmf-config/db-codfw.php: Depool tempdb2001, no longer needed - T161712 (duration: 01m 08s) |
[production] |
| 06:17 |
<marostegui> |
Stop MySQL on tempdb2001 to take a backup and prepare to decomission - T161712 |
[production] |
| 06:10 |
<marostegui> |
Deploy alter table on wikidatawiki.wb_terms - db2066 - T162539 T163548 |
[production] |
| 06:10 |
<marostegui@naos> |
Synchronized wmf-config/db-codfw.php: Depool db2066 - T162539 T163548 (duration: 01m 25s) |
[production] |
| 06:09 |
<Dereckson> |
CentralAuth: Removed MediaWiki 2FA for Alexsh (T164265) |
[production] |
| 06:03 |
<marostegui> |
Deploy alter table on wikidatawiki.wb_terms - dbstore2002 - T162539 T163548 |
[production] |
| 02:31 |
<l10nupdate@naos> |
ResourceLoader cache refresh completed at Thu May 4 02:31:22 UTC 2017 (duration 5m 21s) |
[production] |
| 02:26 |
<l10nupdate@naos> |
scap sync-l10n completed (1.29.0-wmf.21) (duration: 08m 02s) |
[production] |
| 02:08 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3a (duration: 08m 37s) |
[production] |
| 02:00 |
<urandom> |
T160759: lowering tombstone threshold to 1000 on all eqiad nodes |
[production] |
| 01:59 |
<mobrovac@naos> |
Started deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3a |
[production] |
| 01:58 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3 (duration: 03m 29s) |
[production] |
| 01:54 |
<mobrovac@naos> |
Started deploy [restbase/deploy@4d04dfd]: blacklist dewiki page, take 3 |
[production] |
| 01:51 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki (duration: 03m 28s) |
[production] |
| 01:47 |
<mobrovac@naos> |
Started deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki |
[production] |
| 01:47 |
<mobrovac@naos> |
Finished deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki (duration: 04m 12s) |
[production] |
| 01:42 |
<mobrovac@naos> |
Started deploy [restbase/deploy@4d04dfd]: Blacklist a page on dewiki |
[production] |
| 01:22 |
<urandom_> |
T160759: lowering tombstone_threshold on restbase1013 & restbase1014 |
[production] |
| 01:09 |
<urandom_> |
T160759: starting restbase1012-a |
[production] |
|
2017-05-03
§
|
| 22:59 |
<RainbowSprinkles> |
gerrit: Quick restart to pick up logging config change |
[production] |
| 22:47 |
<ejegg> |
updated fundraising tools from 20afe9d20a44a0dbee58f4f82e65ba5689c60de1 to f2522cdabf1741a60b7b60ac8f7ead7afd50b054 |
[production] |
| 22:23 |
<ejegg> |
updated fundraising tools from a1e9342e093a85032255fc1d9904db7df13680b7 to 20afe9d20a44a0dbee58f4f82e65ba5689c60de1 |
[production] |
| 21:06 |
<demon@naos> |
Synchronized README: No-op, forcing co-master sync (duration: 02m 28s) |
[production] |
| 20:35 |
<mutante> |
mw1167 - same as mw1166 (jobrunners) - there was a hhvm[12547]: Fatal error: unknown exception followed by mysql slow query, SELECT MASTER_TID_WAIT... | systemctl restart hhvm recovers it |
[production] |
| 20:30 |
<mutante> |
mw1166 - restart hhvm service (Fatal error: request has exceeded memory limit) |
[production] |
| 20:13 |
<urandom> |
T160759: restoring default tombstone thresholds, restbase10{3,4,6} |
[production] |
| 19:57 |
<mutante> |
mw1287 - also restarting hhvm (with systemctl restart) |
[production] |
| 19:56 |
<mutante> |
mw1287 - restarted crashed apache (proxy_fcgi:error) |
[production] |
| 19:48 |
<demon@naos> |
Finished scap: Cleaning up some unused branches, no-op (duration: 15m 13s) |
[production] |
| 19:33 |
<demon@naos> |
Started scap: Cleaning up some unused branches, no-op |
[production] |
| 19:32 |
<demon@naos> |
Pruned MediaWiki: 1.29.0-wmf.18 (duration: 00m 19s) |
[production] |
| 19:30 |
<demon@naos> |
Pruned MediaWiki: 1.29.0-wmf.20 [keeping static files] (duration: 00m 44s) |
[production] |
| 19:27 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 attempt #2 - checks timeout (duration: 01m 39s) |
[production] |
| 19:26 |
<ppchelko@naos> |
Started deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 attempt #2 - checks timeout |
[production] |
| 19:25 |
<ppchelko@naos> |
Finished deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 (duration: 07m 39s) |
[production] |
| 19:18 |
<ppchelko@naos> |
Started deploy [restbase/deploy@76d909f]: Blacklist a title to fix cassandra OOMs T160759 |
[production] |
| 18:48 |
<papaul> |
db2084 - signing puppet certs, salt-key, initial run |
[production] |
| 18:48 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1014 |
[production] |
| 18:46 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1016 |
[production] |
| 18:39 |
<urandom> |
T160759: reducing tombstone threshold to 1000, restbase1013 |
[production] |
| 18:35 |
<urandom> |
restarting restbase1016-c |
[production] |
| 18:34 |
<urandom> |
restarting restbase1013-b |
[production] |
| 18:00 |
<bblack> |
restart cp2005 backend (lag) |
[production] |
| 17:33 |
<moritzm> |
uploaded openjdk-8 u131 to apt.wikimedia.org |
[production] |
| 17:14 |
<jynus@naos> |
Synchronized wmf-config/InitialiseSettings.php: Disable cognate- it is causing an outage on x1 (duration: 01m 06s) |
[production] |
| 16:30 |
<jynus@naos> |
Synchronized wmf-config/db-eqiad.php: Fine-tune per-server load to reduce db connection errors (duration: 01m 27s) |
[production] |