|
2015-04-02
§
|
| 12:51 |
<andrewbogott> |
restarted opendj, pdns on neptunium, nembus, virt1000, labcontrol2001 |
[production] |
| 12:48 |
<paravoid> |
repooling esams |
[production] |
| 12:42 |
<paravoid> |
upgrading junos on mr1-esams |
[production] |
| 12:15 |
<mark> |
Shutting down cp3014 for 10G upgrade |
[production] |
| 11:02 |
<mark> |
Shutting down cp3012 for 10G upgrade |
[production] |
| 10:38 |
<_joe_> |
stopping pybal on lvs2003, running manually to help debugging |
[production] |
| 09:43 |
<paravoid> |
asw-d-eqiad: routing-engine backup switch FPC 8 -> FPC 4 |
[production] |
| 09:42 |
<paravoid> |
asw-d-eqiad: routing-engine backup switch FPC 7 -> FPC 5, master switchover FPC 8 -> FPC 5 |
[production] |
| 08:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Apr 2 08:30:42 UTC 2015 (duration 30m 41s) |
[production] |
| 07:48 |
<springle> |
on sync-file from tin: mw2213.codfw.wmnet returned [255]: Host key verification failed |
[production] |
| 07:46 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1027, warm up (duration: 00m 11s) |
[production] |
| 06:41 |
<bblack> |
re-depooled esams ... |
[production] |
| 06:34 |
<Jamesofur> |
manually merged Ximilian global account per request and account confirmation |
[production] |
| 06:13 |
<bblack> |
re-pooling esams (GTT event never happened AFAICS) |
[production] |
| 05:13 |
<springle> |
xtrabackup clone db1027 to db1035 |
[production] |
| 03:45 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1027 (duration: 00m 11s) |
[production] |
| 03:32 |
<springle> |
reinstall db1035, reclone data |
[production] |
| 03:18 |
<LocalisationUpdate> |
completed (1.25wmf24) at 2015-04-02 03:08:52+00:00 |
[production] |
| 03:03 |
<l10nupdate> |
Synchronized php-1.25wmf24/cache/l10n: (no message) (duration: 08m 55s) |
[production] |
| 02:41 |
<l10nupdate> |
Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 56s) |
[production] |
| 02:06 |
<legoktm> |
set email for TheFons@global and attached nlwiki |
[production] |
| 00:03 |
<krenair> |
Synchronized php-1.25wmf24/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201386 (duration: 00m 12s) |
[production] |
| 00:01 |
<krenair> |
Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/201385 (duration: 00m 13s) |
[production] |
|
2015-04-01
§
|
| 23:59 |
<bblack> |
depooling esams ahead of 2h planned GTT link outage coming up in 1h |
[production] |
| 23:41 |
<krenair> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/200098 (duration: 00m 12s) |
[production] |
| 23:37 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/201374/ (duration: 00m 12s) |
[production] |
| 23:35 |
<krenair> |
Synchronized php-1.25wmf24/extensions/Flow: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 13s) |
[production] |
| 23:33 |
<krenair> |
Synchronized php-1.25wmf24/extensions/Flow/modules/editor/editors/visualeditor/ext.flow.editors.visualeditor.js: https://gerrit.wikimedia.org/r/#/c/201360/ (duration: 00m 11s) |
[production] |
| 23:29 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/201093/ (duration: 00m 12s) |
[production] |
| 23:23 |
<krenair> |
Synchronized php-1.25wmf23/extensions/OpenStackManager/nova/OpenStackNovaHost.php: https://gerrit.wikimedia.org/r/#/c/201367/ (duration: 00m 15s) |
[production] |
| 23:02 |
<krenair> |
Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/201197/ (duration: 00m 11s) |
[production] |
| 22:59 |
<awight> |
update payments from f617326761887ed9a9100b472ea3b5736e2c10e6 to d37687239fa79842c0d6ea65e9230a3f14cda867 |
[production] |
| 22:57 |
<twentyafterfour> |
Purged l10n cache for 1.25wmf22 |
[production] |
| 22:57 |
<kaldari> |
Synchronized wmf-config/InitialiseSettings.php: enabling Gather on enwiki (duration: 00m 13s) |
[production] |
| 22:48 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf24 |
[production] |
| 22:31 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf23 |
[production] |
| 22:14 |
<twentyafterfour> |
Finished scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache (duration: 54m 14s) |
[production] |
| 21:56 |
<bblack> |
repooled cp107[1234] in pybal (eqiad upload, row D) |
[production] |
| 21:20 |
<twentyafterfour> |
Started scap: once again: testwiki to 1.25wmf24 and rebuild l10n cache |
[production] |
| 21:18 |
<twentyafterfour> |
Started scap: testwiki to 1.25wmf24 and rebuild l10n cache |
[production] |
| 20:32 |
<gwicke> |
finished RESTBase 0.5.0 deployment |
[production] |
| 20:28 |
<gwicke> |
deploying RESTBase 0.5.0 |
[production] |
| 20:19 |
<twentyafterfour> |
Started scap: retrying: testwiki to php-1.25wmf24 and rebuild l10n cache |
[production] |
| 20:08 |
<YuviPanda> |
run chmod -R g+w . on tin with CWD /srv/deployment/scap/scap/.git |
[production] |
| 19:48 |
<twentyafterfour> |
scap failed: CalledProcessError Command 'cp '/srv/mediawiki-staging/php-1.25wmf24/cache/l10n/'*.cdb '/tmp/scap_l10n_1816369030'' returned non-zero exit status 1 (duration: 14m 20s) |
[production] |
| 19:33 |
<twentyafterfour> |
Started scap: testwiki to php-1.25wmf24 and rebuild l10n cache |
[production] |
| 19:18 |
<paravoid> |
upgrading cr1/2-eqiad<->asw-d-eqiad capacity (T92914) |
[production] |
| 17:55 |
<mutante> |
added jmm to ops and wmf LDAP groups |
[production] |
| 17:20 |
<_joe_> |
restarted hhvm on mw1194, stuck in HPHP::StatCache::refresh |
[production] |
| 17:12 |
<ottomata> |
initiated kafka replica election |
[production] |