|
2015-08-05
§
|
| 19:55 |
<gwicke> |
re-enabled puppet on restbase staging cluster in preparation for deploy |
[production] |
| 19:52 |
<gwicke> |
disabled puppet on restbase hosts in preparation for the deploy |
[production] |
| 19:36 |
<dcausse> |
es1.7.1: resume writes to indices |
[production] |
| 19:31 |
<dcausse> |
es1.7.1: restart elastic1013 |
[production] |
| 19:19 |
<bblack> |
all caches depooled for thermal stuff repooled |
[production] |
| 18:54 |
<bblack> |
depooled cp1060, cp1064 ( thermal batch 3: https://phabricator.wikimedia.org/T103226 ) |
[production] |
| 18:37 |
<dcausse> |
es1.7.1: restart elastic1012 |
[production] |
| 18:33 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf17 |
[production] |
| 18:07 |
<bblack> |
depooled cp1059, cp1062, cp1067 ( thermal batch 2: https://phabricator.wikimedia.org/T103226 ) |
[production] |
| 18:02 |
<moritzm> |
restarted HHVM on appservers (mw1136-mw1158) for tidy/pcre security updates |
[production] |
| 17:56 |
<dcausse> |
es1.7.1: restart elastic1011 |
[production] |
| 17:48 |
<dcausse> |
es1.7.1: freeze indices (take 2) |
[production] |
| 17:36 |
<bblack> |
Synchronized wmf-config/squid-labs.php: (no message) (duration: 00m 12s) |
[production] |
| 17:15 |
<moritzm> |
restarted HHVM on appservers (mw1149-mw1151, mw1161-1188, mw1209-1220) for tidy/pcre security updates |
[production] |
| 17:09 |
<hoo> |
Finished scap: Rebuild l10n cache for wmf17, got forgotten during the train (duration: 26m 02s) |
[production] |
| 17:07 |
<bblack> |
really depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 ) |
[production] |
| 17:02 |
<bblack> |
depooled cp1046, cp1061, cp1066 ( thermal batch 1: https://phabricator.wikimedia.org/T103226 ) |
[production] |
| 16:43 |
<hoo> |
Started scap: Rebuild l10n cache for wmf17, got forgotten during the train |
[production] |
| 16:28 |
<bblack> |
cache puppets disabled for a little while, to make sure do_esi doesn't melt things |
[production] |
| 15:11 |
<thcipriani> |
Synchronized php-1.26wmf17/extensions/ContentTranslation/modules/tools/ext.cx.tools.mt.js: SWAT: FIX: Not able to set cursor in previous sections [[gerrit:229328]] (duration: 00m 12s) |
[production] |
| 15:02 |
<andrewbogott> |
rebooting labvirt1009 |
[production] |
| 14:51 |
<gwicke> |
stopped restbase on restbase1009 |
[production] |
| 14:44 |
<moritzm> |
restarted HHVM on appservers (mw1026-mw1113) for tidy/pcre security updates |
[production] |
| 14:42 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool db1056 (duration: 00m 12s) |
[production] |
| 14:29 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool db1059 (duration: 00m 13s) |
[production] |
| 13:16 |
<hoo> |
Removed Wikidata JSON dumps from Monday and Tuesday as they were incomplete/ had the wrong serialization format |
[production] |
| 12:41 |
<moritzm> |
restarted HHVM on canary appservers for tidy/pcre security updates, remaining app servers following soon |
[production] |
| 12:32 |
<paravoid> |
upgrading asw-c-codfw and asw-d-codfw to newer junos |
[production] |
| 11:17 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool db1056, depool db1059 (duration: 00m 12s) |
[production] |
| 11:01 |
<godog> |
depool restbase1009, investigating healthcheck returning 500s |
[production] |
| 10:52 |
<godog> |
pool restbase100[789] in pybal |
[production] |
| 10:43 |
<paravoid> |
upgrading asw-b-codfw to newer junos |
[production] |
| 10:36 |
<jynus> |
applying schema change for s4 on codfw, some lag expected |
[production] |
| 09:08 |
<dcausse> |
es1.7.1: upgrade elastic1010 |
[production] |
| 07:46 |
<dcausse> |
es1.7.1: upgrade elastic1009 |
[production] |
| 07:12 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool db1056 for maintenance, db1064 set to 100% (duration: 00m 12s) |
[production] |
| 06:29 |
<springle> |
finish OSC gerrit 228756 s5 wb_items_per_site.ips_site_page |
[production] |
| 06:27 |
<@tin> |
ResourceLoader cache refresh completed at Wed Aug 5 06:27:08 UTC 2015 (duration 27m 7s) |
[production] |
| 06:26 |
<dcausse> |
es1.7.1: upgrade elastic1008 |
[production] |
| 04:56 |
<ebernhardson> |
restarted elasticsearch on elastic1007 for 1.7.1 upgrade |
[production] |
| 03:34 |
<legoktm> |
Synchronized wmf-config/InitialiseSettings.php: Disable two more wikis due to namespace conflicts - https://gerrit.wikimedia.org/r/229292 (duration: 00m 12s) |
[production] |
| 03:09 |
<ebernhardson> |
restarted elasticsearch on elastic1006 for 1.7.1 upgrade |
[production] |
| 03:04 |
<@tin> |
LocalisationUpdate completed (1.26wmf17) at 2015-08-05 03:04:08+00:00 |
[production] |
| 02:57 |
<l10nupdate> |
Synchronized php-1.26wmf17/cache/l10n: (no message) (duration: 10m 30s) |
[production] |
| 02:31 |
<@tin> |
LocalisationUpdate completed (1.26wmf16) at 2015-08-05 02:31:44+00:00 |
[production] |
| 02:28 |
<l10nupdate> |
Synchronized php-1.26wmf16/cache/l10n: (no message) (duration: 06m 56s) |
[production] |
| 01:44 |
<ebernhardson> |
restarting elasticsearch of es1005 |
[production] |