|
2015-10-09
§
|
| 15:06 |
<urandom> |
starting nodetool cleanup on restbase-test2002 |
[production] |
| 14:37 |
<jynus> |
more configuration testing (with puppet disabled) and several mysql restarts on db1022 |
[production] |
| 13:53 |
<hashar> |
Restarted Zuul, had a deadlocked job |
[production] |
| 11:50 |
<godog> |
bounce mathoid on sca100[12], stray instance found not running firejail |
[production] |
| 11:22 |
<mobrovac> |
restbase deploying aaee7c31 |
[production] |
| 11:16 |
<godog> |
force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/ |
[production] |
| 11:00 |
<jynus> |
restarting db1022's mysql (depooled) for configuration testing |
[production] |
| 09:17 |
<jynus> |
deployed visual glitch fix to dbtree |
[production] |
| 09:11 |
<akosiaris> |
poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. |
[production] |
| 09:11 |
<akosiaris> |
poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. |
[production] |
| 08:20 |
<ori> |
Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/ |
[production] |
| 05:13 |
<mutante> |
@RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464 |
[production] |
| 02:32 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s) |
[production] |
| 01:24 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s) |
[production] |
| 01:18 |
<RoanKattouw> |
Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh |
[production] |
| 01:17 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s) |
[production] |
|
2015-10-08
§
|
| 23:56 |
<catrope@tin> |
Finished scap: SWAT (duration: 29m 10s) |
[production] |
| 23:51 |
<yurik> |
reverted Kartotherian to HEAD^^ - the service wouldn't start |
[production] |
| 23:27 |
<catrope@tin> |
Started scap: SWAT |
[production] |
| 23:06 |
<yurik> |
deployed kartotherian |
[production] |
| 18:20 |
<twentyafterfour@tin> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2 |
[production] |
| 16:45 |
<mobrovac> |
mathoid deploying 110abaf |
[production] |
| 15:12 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) [[gerrit:244332]] (duration: 00m 18s) |
[production] |
| 15:05 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki [[gerrit:244142]] (duration: 00m 17s) |
[production] |
| 14:53 |
<paravoid> |
replacing cr1-codfw<->asw-a-codfw QSFPs |
[production] |
| 14:48 |
<reedy@tin> |
Synchronized docroot and w: (no message) (duration: 00m 17s) |
[production] |
| 14:13 |
<jynus> |
performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002) |
[production] |
| 13:59 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s) |
[production] |
| 13:11 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s) |
[production] |
| 12:09 |
<godog> |
update facts on puppet compiler from palladium |
[production] |
| 11:38 |
<_joe_> |
rebooting mc2002 |
[production] |
| 10:39 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s) |
[production] |
| 10:37 |
<_joe_> |
rebooting mw1008, in oom spiral |
[production] |
| 10:15 |
<paravoid> |
salt rm /home/*/.ssh/authorized_keys |
[production] |
| 09:21 |
<jynus> |
downtime for db1055 for maintenance (kernel update, mysql update, config update) |
[production] |
| 08:45 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 17s) |
[production] |
| 07:44 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Increase load of db1051 and db1055 also for regular traffic (duration: 00m 17s) |
[production] |
| 07:06 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Matching db1051 and db1055 weight for load balancing (duration: 00m 16s) |
[production] |
| 06:15 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Oct 8 06:15:16 UTC 2015 (duration 15m 15s) |
[production] |
| 05:56 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 18s) |
[production] |
| 03:04 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-08 03:04:52+00:00 |
[production] |
| 03:02 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 33s) |
[production] |
| 02:46 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.1) at 2015-10-08 02:46:57+00:00 |
[production] |
| 02:40 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 30s) |
[production] |
| 01:17 |
<twentyafterfour> |
phabricator is now up and running again |
[production] |
| 01:09 |
<twentyafterfour> |
buggy update, stopping apache2 on iridium. |
[production] |
| 01:02 |
<twentyafterfour> |
finished phabricator update |
[production] |
| 00:51 |
<mutante> |
mw1160 - was oom-killer (convert, mw job 31497) |
[production] |
| 00:50 |
<twentyafterfour> |
phabricator maintenance/upgrade. Expect 10 minutes downtime |
[production] |
| 00:44 |
<mutante> |
powercycled mw1160 - looks like broken hardware or cable (ata1: lost interrupt) |
[production] |