|
2015-04-23
§
|
| 23:41 |
<andrewbogott> |
reverted labvirt1001 to 3.13.0-49-generic because 3.16 wouldn’t mount the fs |
[production] |
| 23:31 |
<rmoen> |
Synchronized php-1.26wmf2/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 20s) |
[production] |
| 23:30 |
<rmoen> |
Synchronized php-1.26wmf3/extensions/MobileFrontend/: Update MobileFrontend to cherry picks (duration: 00m 38s) |
[production] |
| 22:48 |
<andrewbogott> |
upgrading labvirt1001 to linux-image-3.16.0-34-generic, dist-upgrading, and rebooting |
[production] |
| 22:10 |
<bd808> |
Synchronized wmf-config/logging.php: logstash: Fix log level detection (c09014d) (duration: 00m 17s) |
[production] |
| 21:56 |
<ori> |
Additional (planned) outcome of Ie22658727 and Ice65e7e70: xff log flowing to fluorine, causing bytes-in to climb from ~1.2M/s to ~2.1M/s |
[production] |
| 21:54 |
<ori> |
Syncing Ie22658727 and Ice65e7e70 (which introduce new InitialiseSettings vars) in one go caused a small burst of 500s (peaking at 500/sec and lasting a few seconds) on four app servers. |
[production] |
| 21:42 |
<ori> |
Synchronized wmf-config: Ie22658727 and Ice65e7e70: use Monolog to configure logging (duration: 00m 15s) |
[production] |
| 21:04 |
<awight> |
update payments from 88b9f621bfee1de14a8cdef556a90e5567721754 to 83d09e09178c634ad35dbb684d1c3aebbb709969 |
[production] |
| 19:31 |
<mutante> |
restarting icinga-wm for config change |
[production] |
| 18:05 |
<andrewbogott> |
rebooting labvirt1006 |
[production] |
| 17:51 |
<kartik> |
Synchronized php-1.26wmf2/extensions/ContentTranslation: (no message) (duration: 00m 15s) |
[production] |
| 17:29 |
<kartik> |
Finished scap: Update ContentTranslation (duration: 26m 11s) |
[production] |
| 17:28 |
<ori> |
scap stuck on snapshot1004; not accepting mwdeploy key |
[production] |
| 17:03 |
<kartik> |
Started scap: Update ContentTranslation |
[production] |
| 16:53 |
<aaron> |
Synchronized php-1.26wmf2/includes/jobqueue/JobRunner.php: d23777e6832f660984ce4445ab04f98b7ff0d25f (duration: 00m 12s) |
[production] |
| 16:33 |
<andrewbogott> |
rebooting labvirt1005 |
[production] |
| 15:03 |
<manybubbles> |
Synchronized wmf-config/CommonSettings.php: swat: Re-enable Special:SupportedLanguages (duration: 00m 11s) |
[production] |
| 12:29 |
<godog> |
investigating icinga UNKNOWN for hhvm queue/threads |
[production] |
| 09:15 |
<godog> |
restart carbon on graphite1001, replace with carbon-c-relay |
[production] |
| 08:31 |
<godog> |
restart carbon on labmon1001, replace with carbon-c-relay |
[production] |
| 05:22 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Apr 23 05:21:17 UTC 2015 (duration 21m 16s) |
[production] |
| 02:49 |
<LocalisationUpdate> |
completed (1.26wmf3) at 2015-04-23 02:48:40+00:00 |
[production] |
| 02:46 |
<l10nupdate> |
Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 03m 46s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.26wmf2) at 2015-04-23 02:27:39+00:00 |
[production] |
| 02:24 |
<l10nupdate> |
Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 05m 46s) |
[production] |
| 00:15 |
<kaldari> |
Synchronized wmf-config/InitialiseSettings.php: Turning on WikiGrok on English Wikipedia (for 2 week test) (duration: 00m 11s) |
[production] |
| 00:07 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/206024 (duration: 00m 14s) |
[production] |
| 00:05 |
<krenair> |
Synchronized php-1.26wmf2/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/206023/ (duration: 00m 13s) |
[production] |
|
2015-04-22
§
|
| 23:47 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/206015 (duration: 00m 12s) |
[production] |
| 23:44 |
<krenair> |
Synchronized php-1.26wmf3/extensions/ZeroBanner/includes/ZeroSpecialPage.php: https://gerrit.wikimedia.org/r/#/c/206017/ (duration: 00m 13s) |
[production] |
| 23:26 |
<krenair> |
Synchronized php-1.26wmf3/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/206008/ (duration: 00m 13s) |
[production] |
| 23:17 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/205889/ (duration: 00m 12s) |
[production] |
| 23:16 |
<krenair> |
Synchronized php-1.26wmf2/extensions/OpenStackManager/nova/OpenStackNovaUser.php: https://gerrit.wikimedia.org/r/#/c/205887/ (duration: 00m 12s) |
[production] |
| 22:55 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf3 |
[production] |
| 22:52 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf2 |
[production] |
| 22:47 |
<twentyafterfour> |
Finished scap: testwiki to php-1.26wmf3 and rebuild l10n cache (duration: 37m 11s) |
[production] |
| 22:44 |
<hoo> |
Killed demon's "sudo -u www-data php /srv/mediawiki-staging/multiversion/MWScript.php refreshLinks.php --wiki=ptwiki" on terbium, sending the box into swap |
[production] |
| 22:10 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf3 and rebuild l10n cache |
[production] |
| 21:31 |
<Coren> |
reboot round of deployment-prep done |
[production] |
| 21:05 |
<Coren> |
Starting deployment-prep rolling reboots |
[production] |
| 20:10 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf3 and rebuild l10n cache |
[production] |
| 20:08 |
<subbu> |
deployed parsoid version 3311936a |
[production] |
| 19:51 |
<hashar> |
Zuul / Jenkins back up and processing the 1+ hour backlog of changes. Will take a while. Multiple causes: Zuul gearmand being stalled on a socket that has no more data to emit and Jenkins being deadlocked due to an IRC plugin |
[production] |
| 19:44 |
<hashar> |
Killing Jenkins cause .... we know |
[production] |
| 19:27 |
<hashar> |
zuul gearman server is stalled |
[production] |
| 15:30 |
<gwicke> |
stopped restbase on restbase1002 in preparation for cmjohnson1 checking the hardware |
[production] |
| 15:30 |
<demon> |
Finished scap: 1.26wmf2 was tracking master. should be fixed, being paranoid and doing full sync + i18n rebuild (duration: 08m 11s) |
[production] |
| 15:21 |
<demon> |
Started scap: 1.26wmf2 was tracking master. should be fixed, being paranoid and doing full sync + i18n rebuild |
[production] |
| 15:19 |
<demon> |
Synchronized php-1.26wmf2/extensions/VisualEditor/: (no message) (duration: 00m 12s) |
[production] |