|
2016-06-16
§
|
| 14:31 |
<twentyafterfour> |
re-enabled and ran puppet agent --test on iridium. Everything appears to be normal. |
[production] |
| 13:04 |
<mobrovac> |
scb1001 enabled puppet back |
[production] |
| 12:57 |
<gehel> |
rebalancing shards on elasticsearch equiad cluster |
[production] |
| 12:33 |
<Amir1> |
manually restarted celery-ores-worker in scb1001 |
[production] |
| 12:32 |
<moritzm> |
installing apache2 trusty update on graphite1001 |
[production] |
| 12:32 |
<Amir1> |
manually restarted celery-ores-worker in scb1002 |
[production] |
| 12:10 |
<moritzm> |
restarted hhvm on mw1137, got stuck |
[production] |
| 10:44 |
<moritzm> |
depooling mw1154 for kernel update/reboot |
[production] |
| 10:14 |
<mobrovac> |
scb1001 disabling puppet for a while to manually test changeprop with transclusion rules |
[production] |
| 09:59 |
<mobrovac> |
restbase deploy end of ebeaa46 |
[production] |
| 09:56 |
<_joe_> |
powercycling mw1143, unresponsive on ssh, console |
[production] |
| 09:48 |
<mobrovac> |
restbase deploy start of ebeaa46 |
[production] |
| 09:18 |
<hashar@tin> |
Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend: MobileFrontend RL registration issue preventing Special:Nearby from working properly T137919 (duration: 00m 36s) |
[production] |
| 08:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1085, increase weight of all new db servers (duration: 00m 29s) |
[production] |
| 08:15 |
<jynus> |
rebooting db1085 before putting it back into production |
[production] |
| 02:34 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.5) (duration: 15m 49s) |
[production] |
| 00:57 |
<twentyafterfour> |
puppet disabled on iridium because https://gerrit.wikimedia.org/r/#/c/294653/ needs to merge (hotfix in preamble.php which puppet will undo if it's allowed to run) |
[production] |
| 00:43 |
<twentyafterfour> |
phabricator upgrade/maintenance complete. Everything appears to be back up and running normally. |
[production] |
| 00:41 |
<twentyafterfour> |
taking phabricator offline momentarily for scheduled maintenance. |
[production] |
| 00:24 |
<robh> |
mw1147 rebooted and manually running scap pull |
[production] |
| 00:21 |
<robh> |
mw1147 seems to have died during scap, unresponsive from serial console, powercycled |
[production] |
| 00:16 |
<mattflaschen@tin> |
Synchronized php-1.28.0-wmf.6/extensions/Kartographer: Search for maplinks inside and outside of content. (duration: 01m 08s) |
[production] |
|
2016-06-15
§
|
| 23:37 |
<mattflaschen@tin> |
Synchronized php-1.28.0-wmf.6/extensions/Echo: Sync Echo fix for cross-wiki notifications: 62324e3 (duration: 00m 33s) |
[production] |
| 21:32 |
<aaron@tin> |
Synchronized wmf-config/filebackend-production.php: Set "sync" filebackend replication to measure latency effect (duration: 00m 25s) |
[production] |
| 21:27 |
<aaron@tin> |
Synchronized php-1.28.0-wmf.6/includes/libs/objectcache/WANObjectCache.php: faff8f1ef1bfefd1804a3f46e58566711faa3224 (duration: 00m 27s) |
[production] |
| 21:15 |
<dapatrick> |
Deployed patch for T137264 to wmf.5 and wmf.6 |
[production] |
| 20:17 |
<hashar@tin> |
Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki T137917 (duration: 00m 30s) |
[production] |
| 20:09 |
<subbu> |
finished deploying parsoid sha 3445eceb |
[production] |
| 20:05 |
<bblack> |
cache frontend restarts complete |
[production] |
| 20:04 |
<subbu> |
synced new code; restarted parsoid on wtp1001 as a canary |
[production] |
| 20:02 |
<subbu> |
starting parsoid deploy |
[production] |
| 19:25 |
<bblack> |
rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~35 mins to completion - T107236 (...._ |
[production] |
| 19:15 |
<bblack> |
varnish frontend restart halted - v4 compat issue to address :P |
[production] |
| 19:11 |
<bblack> |
rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~30 mins to completion - T107236 |
[production] |
| 19:05 |
<hashar@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: (no message) |
[production] |
| 18:54 |
<ori> |
Started MySQL on es2019 (T130702) |
[production] |
| 16:32 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight (duration: 00m 25s) |
[production] |
| 16:07 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294514|Fix autopatrolled group for ko.wikipedia]] (duration: 00m 31s) |
[production] |
| 16:00 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.special/mediawiki.special.search.styles.css: SWAT: [[gerrit:294511|Explicitly specify the width of the search input on Special:Search]] (duration: 00m 25s) |
[production] |
| 15:52 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294254|Add autopatrolled group in kowiki]] (duration: 00m 24s) |
[production] |
| 15:33 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294488|Deploy ORES beta feature in wikidatawiki]] (duration: 00m 24s) |
[production] |
| 15:23 |
<elukey@palladium> |
conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet |
[production] |
| 15:23 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.5/extensions/ORES: SWAT: [[gerrit:294491|Skip when an edit is errored in PopulateDatabase.php]] (duration: 00m 27s) |
[production] |
| 15:17 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294462|Send authentication events to logstash]] (duration: 00m 28s) |
[production] |
| 15:15 |
<elukey@palladium> |
conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet |
[production] |
| 15:11 |
<thcipriani@tin> |
Synchronized wmf-config/logging.php: SWAT: [[gerrit:293701|Fix logging config for authmanager metrics channel rename]] (duration: 00m 24s) |
[production] |
| 15:10 |
<elukey@palladium> |
conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet |
[production] |
| 15:06 |
<thcipriani@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:293970|Remove old throttle rules]] (duration: 00m 30s) |
[production] |
| 15:00 |
<elukey@palladium> |
conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet |
[production] |
| 15:00 |
<mobrovac> |
scb disabled puppet for stopped change-prop during kafka nodes upgrade |
[production] |