|
2017-11-21
§
|
| 10:10 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@e35aa05]: Set consumer_batch_size to 10 T181007 |
[production] |
| 09:39 |
<elukey> |
upload prometheus-druid-exporter 0.4 to jessie/stretch-wikimedia |
[production] |
| 09:28 |
<marostegui> |
Shutdown db2068 for maintenance - T180927 |
[production] |
| 09:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s) |
[production] |
| 09:02 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase weight for db1096 (duration: 00m 49s) |
[production] |
| 08:39 |
<marostegui> |
Drop index on db1089 enwiki.ores_classification - T180045 |
[production] |
| 08:35 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1096 with low weight - T178359 (duration: 00m 48s) |
[production] |
| 08:27 |
<marostegui> |
Reboot db1096 for kernel and MariaDB upgrade |
[production] |
| 08:19 |
<marostegui> |
Compress s5 on db1101 - T178359 |
[production] |
| 08:11 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully pool db1109 and db1110 - T180700 (duration: 00m 48s) |
[production] |
| 07:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase weight for db1109 and db1110 - T180700 (duration: 00m 49s) |
[production] |
| 07:03 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1109 and db1110 in s5 with small weight - T180700 (duration: 00m 48s) |
[production] |
| 06:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1087 - T179106 (duration: 00m 48s) |
[production] |
| 06:47 |
<marostegui> |
Remove index wb_terms_language from db1087 - https://phabricator.wikimedia.org/T179106 |
[production] |
| 06:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1087 - T179106 (duration: 00m 48s) |
[production] |
| 06:27 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1096 - T178359 (duration: 00m 48s) |
[production] |
| 06:27 |
<marostegui> |
Stop MySQL on db1096 to clone db1101.s5 - T178359 |
[production] |
| 06:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T177208 (duration: 00m 49s) |
[production] |
| 02:58 |
<mutante> |
phabricator back up |
[production] |
| 02:55 |
<mutante> |
phab1001 (phabricator prod) reboot for kernel upgrade |
[production] |
| 02:48 |
<krinkle@tin> |
Synchronized docroot/noc: clean up (duration: 00m 49s) |
[production] |
| 02:23 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.7) (duration: 05m 42s) |
[production] |
| 02:03 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@7d951d2]: Restore categories vocabulary to V003 |
[production] |
| 00:19 |
<catrope@tin> |
Synchronized php-1.31.0-wmf.8/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.ItemMenuOptionWidget.js: T180863 (duration: 00m 49s) |
[production] |
|
2017-11-20
§
|
| 23:46 |
<mutante> |
phab2001 - reboot for kernel upgrade |
[production] |
| 23:35 |
<legoktm@tin> |
Synchronized wmf-config/InitialiseSettings.php: emergency disable ORES on frwp/ruwp T181006 (duration: 00m 49s) |
[production] |
| 23:35 |
<awight> |
purge cache keys for ORES thresholds on frwiki and ruwiki |
[production] |
| 23:25 |
<awight@tin> |
Started deploy [ores/deploy@82a13ae]: Rollback ORES (take 3); 181006 |
[production] |
| 23:19 |
<awight> |
aborted ORES rollback |
[production] |
| 23:18 |
<awight@tin> |
Started deploy [ores/deploy@95cd523]: Rollback ORES (take 2); 181006 |
[production] |
| 23:14 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug (duration: 08m 11s) |
[production] |
| 23:11 |
<awight> |
purged memcache key 'ruwiki:ORES:threshold_statistics:goodfaith:1’, T181006 |
[production] |
| 23:06 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@7d951d2]: Rollback categories vocabulary version due to a bug |
[production] |
| 22:55 |
<awight@tin> |
Finished deploy [ores/deploy@5084251]: Rollback ORES; T179711 (duration: 01m 05s) |
[production] |
| 22:55 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: no wmf.8 for group2. i hate my life |
[production] |
| 22:55 |
<awight> |
rolling back ORES to fix T181006 |
[production] |
| 22:54 |
<eileen> |
update civicrm from 272580a4b0511b7eb5392dc087a2d21e56474ab9 to a16e566940c43e0155b9dbbdaba3399b0d2e01bd |
[production] |
| 22:54 |
<awight@tin> |
Started deploy [ores/deploy@5084251]: Rollback ORES; T179711 |
[production] |
| 22:50 |
<Krinkle> |
MW HTTP 500 spike tracked as https://phabricator.wikimedia.org/T181006 |
[production] |
| 22:49 |
<Krinkle> |
Sharp rise in HTTP 500 errors as of 22:05 (45 minutes ago) |
[production] |
| 22:29 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.7/includes/libs/objectcache/WANObjectCache.php: fix cache key namespace (duration: 00m 49s) |
[production] |
| 22:27 |
<awight@tin> |
Finished deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 (duration: 49m 54s) |
[production] |
| 22:26 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.7/extensions/Collection: cache key name fix (duration: 00m 49s) |
[production] |
| 22:11 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.8/includes/libs/objectcache/WANObjectCache.php: 7e74b49: namespace WAN cache variant keys (duration: 00m 48s) |
[production] |
| 22:04 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.8 |
[production] |
| 21:52 |
<bblack> |
ns2 back on eeden |
[production] |
| 21:52 |
<aaron@tin> |
Synchronized php-1.31.0-wmf.8/extensions/Collection: 3baebf4a: cache key name fix (duration: 00m 51s) |
[production] |
| 21:44 |
<bblack> |
rebooting eeden |
[production] |
| 21:37 |
<awight@tin> |
Started deploy [ores/deploy@5084251]: Updating ORES to revscoring 2.0.10, T179711 |
[production] |
| 21:33 |
<bblack> |
routing ns2.wikimedia.org to radon for eeden reboot |
[production] |