|
2015-07-08
§
|
| 11:08 |
<godog> |
bounce cassandra on restbase1004 and restbase1005 'cannot achieve consistency level quorum' |
[production] |
| 10:50 |
<godog> |
bounce cassandra on restbase1004, death by compaction |
[production] |
| 09:43 |
<ori> |
_joe_: starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:42 |
<ori> |
Nuked /var/lib/carbon/whisper/ResourceLoader on graphite[12]001. Data prior to rollout of I55f0c44cd considered bogus. |
[production] |
| 09:42 |
<ori> |
morebots, are you OK? |
[production] |
| 09:41 |
<godog> |
bounce nutcracker on silver |
[production] |
| 09:33 |
<_joe_> |
starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:26 |
<hashar> |
upgraded plugins on jenkins and restarting it |
[production] |
| 09:06 |
<hashar> |
Jenkins registering jobs with Zuul |
[production] |
| 08:41 |
<hashar> |
Jenkins is migrating old build histories. Lot of disk IO happening |
[production] |
| 08:12 |
<hashar> |
shutdowning Jenkins for upgrade. |
[production] |
| 05:57 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 8 05:57:10 UTC 2015 (duration 57m 9s) |
[production] |
| 05:46 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1041, warm up (duration: 00m 13s) |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-08 02:31:24+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-08 02:16:50+00:00 |
[production] |
| 02:16 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 48s) |
[production] |
|
2015-07-07
§
|
| 23:54 |
<jgage> |
kafka brokers 1018 & 1021 were demoted; i have triggered a leader election and they are leaders again |
[production] |
| 23:05 |
<catrope> |
Synchronized visualeditor-default.dblist: Enable VE by default on labswiki (duration: 00m 12s) |
[production] |
| 21:56 |
<hoo> |
Restarted hhvm on mw1003 "Fatal error: Function already defined: wmfLoadInitialiseSettings in /srv/mediawiki/wmf-config/CommonSettings.php on line 187" |
[production] |
| 21:16 |
<krinkle> |
Synchronized php-1.26wmf13/includes/resourceloader/ResourceLoader.php: T104769 (duration: 00m 13s) |
[production] |
| 20:53 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf13 |
[production] |
| 20:00 |
<twentyafterfour> |
Finished scap: testwiki to php-1.26wmf13 and rebuild l10n cache (duration: 39m 41s) |
[production] |
| 19:47 |
<gwicke> |
restarted cassandra on restbase1005 |
[production] |
| 19:20 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf13 and rebuild l10n cache |
[production] |
| 19:15 |
<moritzm> |
installed PHP security updates on all trusty hosts |
[production] |
| 18:58 |
<ejegg> |
updated payments from a17ee221db0dbde70c92e24fc188379b6dbad613 to ec34ebf61e5962f66b807abdcb519ff323d41e8e |
[production] |
| 18:09 |
<twentyafterfour> |
restarted apache2 on iridium (phab hotfix) |
[production] |
| 17:10 |
<robh> |
OTRS update appears to be functioning normally. As such, ending maintenance window. |
[production] |
| 17:06 |
<robh> |
otrs is now using the new sha256 cert |
[production] |
| 17:00 |
<robh> |
starting otrs maint window |
[production] |
| 16:58 |
<_joe_> |
restarted HHVM on mw1026, near to OOM |
[production] |
| 16:47 |
<twentyafterfour> |
applied hotfix for phabricator bug: https://secure.phabricator.com/D13544 |
[production] |
| 16:36 |
<mutante> |
protactinium - manual iptables rules replaced by puppet/ferm rules |
[production] |
| 16:11 |
<thcipriani> |
Synchronized php-1.26wmf12/extensions/ContentTranslation/extension.json: Remove default value for ContentTranslationCampaigns (duration: 00m 12s) |
[production] |
| 15:33 |
<jynus> |
manually editing table mediawiki.ipblocks to fully solve a former software bug |
[production] |
| 15:12 |
<Jeff_Green> |
ptr records for frack/codfw and authdns-update |
[production] |
| 15:10 |
<thcipriani> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: CX: Enable ContentTranslation in enwiki [[gerrit:222991]] (duration: 00m 13s) |
[production] |
| 14:21 |
<jynus> |
dropping optin_survey_old table from enwiki |
[production] |
| 13:23 |
<akosiaris> |
restarting gitblit on antimony |
[production] |
| 11:31 |
<mobrovac> |
restbase restarted cassandra on rb1005 |
[production] |
| 11:26 |
<godog> |
restart cassandra on restbase1004, heap exhausted |
[production] |
| 10:49 |
<godog> |
restarted cassandra on restbase1005, mutations through the roof |
[production] |
| 08:27 |
<godog> |
set operations/puppet/cassandra git submodule repo as hidden |
[production] |
| 06:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Jul 7 06:11:46 UTC 2015 (duration 11m 45s) |
[production] |
| 05:51 |
<krinkle> |
Synchronized php-1.26wmf12/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.js: I3e965dda1c4 (duration: 00m 12s) |
[production] |
| 02:27 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-07 02:27:55+00:00 |
[production] |
| 02:24 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 06m 09s) |
[production] |
| 01:12 |
<ori> |
Re-pooled mw1152 at 20:46 UTC, did not log it then. |
[production] |
| 00:41 |
<springle> |
upgrade db1041 trusty |
[production] |
| 00:37 |
<krenair> |
Synchronized php-1.26wmf12/extensions/CentralAuth/includes/CreateLocalAccountJob.php: https://gerrit.wikimedia.org/r/#/c/223211/ (duration: 00m 13s) |
[production] |