|
2017-11-20
§
|
| 23:46 |
<mutante> |
phab2001 - reboot for kernel upgrade |
[production] |
| 23:35 |
<legoktm@tin> |
Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s) |
[production] |
| 23:35 |
<awight> |
purge cache keys for ORES thresholds on frwiki and ruwiki |
[production] |
| 23:25 |
<awight@tin> |
Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 |
[production] |
| 23:19 |
<awight> |
aborted ORES rollback |
[production] |
| 23:18 |
<awight@tin> |
Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006 |
[production] |
| 23:14 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s) |
[production] |
| 23:11 |
<awight> |
purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006 |
[production] |
| 23:06 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug |
[production] |
| 22:55 |
<awight@tin> |
Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s) |
[production] |
| 22:55 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life |
[production] |
| 22:55 |
<awight> |
rolling back ORES to fix T181006 |
[production] |
| 22:54 |
<eileen> |
update civicrm from 272580a4b0511b7eb5392dc087a2d21e56474ab9 to a16e566940c43e0155b9dbbdaba3399b0d2e01bd |
[production] |
| 22:54 |
<awight@tin> |
Started deploy [ores/deploy@5084251]: Rollback ORES; T179711 |
[production] |
| 22:50 |
<Krinkle> |
MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006 |
[production] |
| 22:49 |
<Krinkle> |
Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago) |
[production] |
| 22:29 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s) |
[production] |
| 22:27 |
<awight@tin> |
Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s) |
[production] |
| 22:26 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s) |
[production] |
| 22:11 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49: namespace WAN cache variant keys (duration: 00m 48s) |
[production] |
| 22:04 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8 |
[production] |
| 21:52 |
<bblack> |
ns2 back on eeden |
[production] |
| 21:52 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a: cache key name fix (duration: 00m 51s) |
[production] |
| 21:44 |
<bblack> |
rebooting eeden |
[production] |
| 21:37 |
<awight@tin> |
Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 |
[production] |
| 21:33 |
<bblack> |
routing ns2.wikimedia.org to radon for eeden reboot |
[production] |
| 21:19 |
<bblack> |
routing ns0.wikimedia.org back to radon post-reboot |
[production] |
| 21:13 |
<bblack> |
rebooting radon |
[production] |
| 21:07 |
<bblack> |
re-routing ns0.wikimedia.org traffic to baham for radon reboot |
[production] |
| 20:17 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 19m 41s) |
[production] |
| 20:16 |
<bblack> |
ns1 dns traffic back to normal on baham |
[production] |
| 20:16 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Translate extension in amwikimedia (T180879)" (duration: 00m 48s) |
[production] |
| 20:11 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable Translate extension in amwikimedia (T180879) (duration: 00m 49s) |
[production] |
| 20:11 |
<hashar> |
CI docker jobs were all broken due to a mistake. Should be back now. T177684 |
[production] |
| 20:07 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: RCFilters: Only apply excluded label to namespace items (T180863) (duration: 00m 49s) |
[production] |
| 20:06 |
<bblack> |
rebooting baham |
[production] |
| 19:58 |
<bblack> |
re-routing ns1.wikimedia.org traffic to radon for baham reboot |
[production] |
| 19:52 |
<chasemp> |
updating phab mail handler |
[production] |
| 19:51 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove overlapping userrights (T101983) (duration: 00m 49s) |
[production] |
| 19:47 |
<ottomata> |
restarted coal with fixes for eventcapsule changes in T179625 |
[production] |
| 19:42 |
<ladsgroup@tin> |
Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 49s) |
[production] |
| 19:40 |
<ladsgroup@tin> |
Synchronized wmf-config/throttle.php: Adjust throttle.php for dewiki workshop (T180046) (duration: 00m 48s) |
[production] |
| 19:36 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Allow admins to remove users from MP3 uploaders user group (T180002) (duration: 00m 49s) |
[production] |
| 19:31 |
<ladsgroup@tin> |
Synchronized portals: SWAT: [[gerrit:392432|Bumping portals to master (T128546)]] (duration: 00m 51s) |
[production] |
| 19:30 |
<ladsgroup@tin> |
Synchronized portals/prod/wikipedia.org/assets: SWAT: [[gerrit:392432|Bumping portals to master (T128546)]] (duration: 00m 49s) |
[production] |
| 19:22 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch submit button from 'save' to 'publish' on dewiki (duration: 00m 50s) |
[production] |
| 18:51 |
<chasemp> |
disable puppet across cloud things to rollout https://gerrit.wikimedia.org/r/#/c/392168/ slowly |
[production] |