|
2015-04-01
§
|
| 19:18 |
<paravoid> |
upgrading cr1/2-eqiad<->asw-d-eqiad capacity (T92914) |
[production] |
| 17:55 |
<mutante> |
added jmm to ops and wmf LDAP groups |
[production] |
| 17:20 |
<_joe_> |
restarted hhvm on mw1194, stuck in HPHP::StatCache::refresh |
[production] |
| 17:12 |
<ottomata> |
initiated kafka replica election |
[production] |
| 17:04 |
<paravoid> |
repooling esams |
[production] |
| 16:48 |
<bd808> |
Updated Wikimania Scholarships to bde1a27 (Improve performance of phase2 report query) |
[production] |
| 16:09 |
<godog> |
bounce cassandra on test cluster |
[production] |
| 16:03 |
<thcipriani> |
Synchronized php-1.25wmf23/extensions/UniversalLanguageSelector: swat [[gerrit:201122]] (duration: 01m 07s) |
[production] |
| 15:58 |
<paravoid> |
upgrading junos on cr1-esams (esams is depooled, ignore alerts) |
[production] |
| 15:48 |
<phuedx> |
Synchronized php-1.25wmf22/extensions/Gather/: Updating the Gather extension for 1.25wmf22 (duration: 01m 06s) |
[production] |
| 15:47 |
<bd808> |
ssh: connect to host mw2213.codfw.wmnet port 22: Connection timed out during sync-dir initiated from tin |
[production] |
| 15:45 |
<phuedx> |
Synchronized php-1.25wmf23/extensions/Gather: Updating the Gather extension for 1.25wmf23 (duration: 01m 07s) |
[production] |
| 15:33 |
<godog> |
disable puppet on xenon, praseodymium, cerium, restbase* to test https://gerrit.wikimedia.org/r/197840 |
[production] |
| 15:31 |
<^d> |
ran sync-common on mw1017 for testwiki fun |
[production] |
| 14:35 |
<paravoid> |
upgrading junos on cr2-knams (esams is depooled, ignore alerts) |
[production] |
| 14:32 |
<mark> |
mark@csw2-esams> request system power-off member 1 |
[production] |
| 14:28 |
<mark> |
asw-esams: mark@asw-esams> request system power-off member 3 |
[production] |
| 14:20 |
<mark> |
Shutting down lvs3003 and lvs3004 |
[production] |
| 12:08 |
<paravoid> |
draining esams |
[production] |
| 07:28 |
<godog> |
bounce apertium-apy on sca1001/sca1002 |
[production] |
| 06:43 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Apr 1 06:42:24 UTC 2015 (duration 42m 23s) |
[production] |
| 06:10 |
<_joe_> |
manually rotated and compressed syslog and apache logs on uranium, still being spammed by kafka brokers |
[production] |
| 03:05 |
<LocalisationUpdate> |
completed (1.25wmf23) at 2015-04-01 03:04:00+00:00 |
[production] |
| 02:58 |
<l10nupdate> |
Synchronized php-1.25wmf23/cache/l10n: (no message) (duration: 08m 41s) |
[production] |
| 02:35 |
<LocalisationUpdate> |
completed (1.25wmf22) at 2015-04-01 02:34:41+00:00 |
[production] |
| 02:29 |
<l10nupdate> |
Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 08m 46s) |
[production] |
| 01:35 |
<legoktm> |
started zuul on gallium |
[production] |
| 00:40 |
<urandom> |
restarting cassandra on restbase1006 |
[production] |
| 00:40 |
<urandom> |
restarting cassandra on restbase1005 |
[production] |
| 00:35 |
<urandom> |
restarting cassandra on restbase1004 |
[production] |
| 00:33 |
<urandom> |
restarting cassandra on restbase1003 |
[production] |
| 00:30 |
<urandom> |
restarting cassandra on restbase1002 |
[production] |
| 00:12 |
<catrope> |
Synchronized php-1.25wmf23/extensions/Gather: SWAT (duration: 01m 07s) |
[production] |
| 00:11 |
<RoanKattouw> |
ssh: connect to host mw2213.codfw.wmnet port 22: Connection timed out |
[production] |
| 00:11 |
<catrope> |
Synchronized php-1.25wmf23/extensions/ImageMetrics: SWAT (duration: 01m 07s) |
[production] |
| 00:10 |
<catrope> |
Synchronized php-1.25wmf23/extensions/VisualEditor: SWAT (duration: 01m 07s) |
[production] |
| 00:05 |
<catrope> |
Synchronized php-1.25wmf22/extensions/Gather: SWAT (duration: 01m 06s) |
[production] |
|
2015-03-31
§
|
| 23:15 |
<urandom> |
restarting cassandra on restbase1001 |
[production] |
| 18:39 |
<awight> |
rollback crm from b4268a60225ae11f2c2b58d3b1f1c44e282f9ec6 to 59f03df6b689ef443cc7b7e31e6f5b2986bc8bc9 |
[production] |
| 18:09 |
<twentyafterfour> |
mw2213.codfw.wmnet still timing out |
[production] |
| 18:07 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to $VERSION, mw2213.codfe.wmnet failed, trying one more time |
[production] |
| 18:06 |
<twentyafterfour> |
sync_wikiversions failed for host mw2213.codfw.wmnet port 22: Connection timed out |
[production] |
| 18:04 |
<twentyafterfour> |
group1 to VERSION=1.25wmf23 |
[production] |
| 18:03 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 to VERSION |
[production] |
| 17:54 |
<krenair> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/200901/ (duration: 01m 06s) |
[production] |
| 17:20 |
<awight> |
update crm from 0e4a6ed961ca5a5882d42949c12f74c1d246b55e to b4268a60225ae11f2c2b58d3b1f1c44e282f9ec6 |
[production] |
| 16:59 |
<mutante> |
installing many package upgrades on wikitech-static |
[production] |
| 16:58 |
<awight> |
updated crm from 4c459f3dbf3c3466cdc26a351ba589f4f1aef587 to 0e4a6ed961ca5a5882d42949c12f74c1d246b55e |
[production] |
| 16:04 |
<krenair> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/196885/ (duration: 01m 06s) |
[production] |
| 15:55 |
<robh> |
ignore any alerts for wtp2001-2020, reinstalling |
[production] |