|
2015-07-08
§
|
| 21:08 |
<bblack> |
graphite: wiped /var/log/upstart/statsite* logs, restarted statsite processes |
[production] |
| 20:56 |
<csteipp> |
deployed patches for T103022 & T103023 |
[production] |
| 20:53 |
<csteipp> |
deployed patch for T94116 for wmf12/wmf13 |
[production] |
| 20:30 |
<gwicke> |
added explicit exit 1 in /etc/init.d/cassandra on restbase1008 to prevent cassandra from starting up there; is puppet restarting it? |
[production] |
| 20:29 |
<subbu> |
deployed parsoid sha c4cfc527 |
[production] |
| 20:15 |
<gwicke> |
bounced cassandra on restbase1001 |
[production] |
| 20:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 8 20:05:09 UTC 2015 (duration 5m 8s) |
[production] |
| 19:32 |
<gwicke> |
stopped cassandra on restbase1008 |
[production] |
| 19:27 |
<twentyafterfour> |
Synchronized php-1.26wmf13: deploying UniversalLanguageSelector commit 2e0990ac9879 (duration: 01m 58s) |
[production] |
| 19:26 |
<urandom> |
restbase rolling restart |
[production] |
| 18:21 |
<jgage> |
ran 'kafka preferred-replica-election' to promote analytics1021 back to Leader |
[production] |
| 18:05 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf13 |
[production] |
| 17:16 |
<moritzm> |
installed libwmf security updates on various systems |
[production] |
| 17:09 |
<gwicke> |
bounced cassandra on restbase1004 |
[production] |
| 15:25 |
<mutante> |
handing over adminship of the "test" mailman list to John F. Lewis (was: Thehelpfulone) due to inactivity |
[production] |
| 13:36 |
<springle> |
Synchronized wmf-config/db-eqiad.php: raise db1041 load (duration: 00m 13s) |
[production] |
| 12:58 |
<paravoid> |
manually dpkg -P ferm on potassium |
[production] |
| 12:52 |
<paravoid> |
rmmod all iptables/netfilter-related modules from potassium |
[production] |
| 11:23 |
<godog> |
bounce cassandra on restbase1004, heap space |
[production] |
| 11:12 |
<_joe_> |
mw1153 passed the smoke tests, repooling |
[production] |
| 11:08 |
<godog> |
bounce cassandra on restbase1004 and restbase1005 'cannot achieve consistency level quorum' |
[production] |
| 10:50 |
<godog> |
bounce cassandra on restbase1004, death by compaction |
[production] |
| 09:43 |
<ori> |
_joe_: starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:42 |
<ori> |
Nuked /var/lib/carbon/whisper/ResourceLoader on graphite[12]001. Data prior to rollout of I55f0c44cd considered bogus. |
[production] |
| 09:42 |
<ori> |
morebots, are you OK? |
[production] |
| 09:41 |
<godog> |
bounce nutcracker on silver |
[production] |
| 09:33 |
<_joe_> |
starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:26 |
<hashar> |
upgraded plugins on jenkins and restarting it |
[production] |
| 09:06 |
<hashar> |
Jenkins registering jobs with Zuul |
[production] |
| 08:41 |
<hashar> |
Jenkins is migrating old build histories. Lot of disk IO happening |
[production] |
| 08:12 |
<hashar> |
shutdowning Jenkins for upgrade. |
[production] |
| 05:57 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 8 05:57:10 UTC 2015 (duration 57m 9s) |
[production] |
| 05:46 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1041, warm up (duration: 00m 13s) |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-08 02:31:24+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-08 02:16:50+00:00 |
[production] |
| 02:16 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 48s) |
[production] |
|
2015-07-07
§
|
| 23:54 |
<jgage> |
kafka brokers 1018 & 1021 were demoted; i have triggered a leader election and they are leaders again |
[production] |
| 23:05 |
<catrope> |
Synchronized visualeditor-default.dblist: Enable VE by default on labswiki (duration: 00m 12s) |
[production] |
| 21:56 |
<hoo> |
Restarted hhvm on mw1003 "Fatal error: Function already defined: wmfLoadInitialiseSettings in /srv/mediawiki/wmf-config/CommonSettings.php on line 187" |
[production] |
| 21:16 |
<krinkle> |
Synchronized php-1.26wmf13/includes/resourceloader/ResourceLoader.php: T104769 (duration: 00m 13s) |
[production] |
| 20:53 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf13 |
[production] |
| 20:00 |
<twentyafterfour> |
Finished scap: testwiki to php-1.26wmf13 and rebuild l10n cache (duration: 39m 41s) |
[production] |
| 19:47 |
<gwicke> |
restarted cassandra on restbase1005 |
[production] |
| 19:20 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf13 and rebuild l10n cache |
[production] |
| 19:15 |
<moritzm> |
installed PHP security updates on all trusty hosts |
[production] |
| 18:58 |
<ejegg> |
updated payments from a17ee221db0dbde70c92e24fc188379b6dbad613 to ec34ebf61e5962f66b807abdcb519ff323d41e8e |
[production] |
| 18:09 |
<twentyafterfour> |
restarted apache2 on iridium (phab hotfix) |
[production] |
| 17:10 |
<robh> |
OTRS update appears to be functioning normally. As such, ending maintenance window. |
[production] |
| 17:06 |
<robh> |
otrs is now using the new sha256 cert |
[production] |
| 17:00 |
<robh> |
starting otrs maint window |
[production] |