|
2015-07-08
§
|
| 22:56 |
<gwicke> |
finished rolling restart of cassandra cluster to apply https://gerrit.wikimedia.org/r/#/c/223495/ |
[production] |
| 22:45 |
<mutante> |
zirconium - stop puppet for role switch |
[production] |
| 22:33 |
<legoktm> |
Synchronized php-1.26wmf13/includes/changes/EnhancedChangesList.php: Unbreak missing flags in enhanced RC (duration: 00m 12s) |
[production] |
| 22:08 |
<hoo> |
Synchronized php-1.26wmf13/extensions/Wikidata/: Update Wikibase: Fix JavaScript ULS usage (duration: 00m 20s) |
[production] |
| 21:51 |
<manybubbles> |
Synchronized php-1.26wmf12/extensions/CirrusSearch/: Stop some fatals in cirrus (duration: 00m 13s) |
[production] |
| 21:41 |
<bd808> |
Synchronized php-1.26wmf13/includes/api/ApiMain.php: Revert Count API module instantiations and Hook runs (2/2) (duration: 00m 12s) |
[production] |
| 21:40 |
<bd808> |
Synchronized php-1.26wmf13/includes/Hooks.php: Revert Count API module instantiations and Hook runs (1/2) (duration: 00m 12s) |
[production] |
| 21:39 |
<bd808> |
Synchronized php-1.26wmf13/extensions/CirrusSearch/includes/CirrusSearch.php: Suppress interwiki results when they would break (duration: 00m 12s) |
[production] |
| 21:08 |
<bblack> |
graphite: wiped /var/log/upstart/statsite* logs, restarted statsite processes |
[production] |
| 20:56 |
<csteipp> |
deployed patches for T103022 & T103023 |
[production] |
| 20:53 |
<csteipp> |
deployed patch for T94116 for wmf12/wmf13 |
[production] |
| 20:30 |
<gwicke> |
added explicit exit 1 in /etc/init.d/cassandra on restbase1008 to prevent cassandra from starting up there; is puppet restarting it? |
[production] |
| 20:29 |
<subbu> |
deployed parsoid sha c4cfc527 |
[production] |
| 20:15 |
<gwicke> |
bounced cassandra on restbase1001 |
[production] |
| 20:05 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 8 20:05:09 UTC 2015 (duration 5m 8s) |
[production] |
| 19:32 |
<gwicke> |
stopped cassandra on restbase1008 |
[production] |
| 19:27 |
<twentyafterfour> |
Synchronized php-1.26wmf13: deploying UniversalLanguageSelector commit 2e0990ac9879 (duration: 01m 58s) |
[production] |
| 19:26 |
<urandom> |
restbase rolling restart |
[production] |
| 18:21 |
<jgage> |
ran 'kafka preferred-replica-election' to promote analytics1021 back to Leader |
[production] |
| 18:05 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf13 |
[production] |
| 17:16 |
<moritzm> |
installed libwmf security updates on various systems |
[production] |
| 17:09 |
<gwicke> |
bounced cassandra on restbase1004 |
[production] |
| 15:25 |
<mutante> |
handing over adminship of the "test" mailman list to John F. Lewis (was: Thehelpfulone) due to inactivity |
[production] |
| 13:36 |
<springle> |
Synchronized wmf-config/db-eqiad.php: raise db1041 load (duration: 00m 13s) |
[production] |
| 12:58 |
<paravoid> |
manually dpkg -P ferm on potassium |
[production] |
| 12:52 |
<paravoid> |
rmmod all iptables/netfilter-related modules from potassium |
[production] |
| 11:23 |
<godog> |
bounce cassandra on restbase1004, heap space |
[production] |
| 11:12 |
<_joe_> |
mw1153 passed the smoke tests, repooling |
[production] |
| 11:08 |
<godog> |
bounce cassandra on restbase1004 and restbase1005 'cannot achieve consistency level quorum' |
[production] |
| 10:50 |
<godog> |
bounce cassandra on restbase1004, death by compaction |
[production] |
| 09:43 |
<ori> |
_joe_: starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:42 |
<ori> |
Nuked /var/lib/carbon/whisper/ResourceLoader on graphite[12]001. Data prior to rollout of I55f0c44cd considered bogus. |
[production] |
| 09:42 |
<ori> |
morebots, are you OK? |
[production] |
| 09:41 |
<godog> |
bounce nutcracker on silver |
[production] |
| 09:33 |
<_joe_> |
starting reimaging of mw1153, depooling it and scheduling downtime (at 9:21 UTC) |
[production] |
| 09:26 |
<hashar> |
upgraded plugins on jenkins and restarting it |
[production] |
| 09:06 |
<hashar> |
Jenkins registering jobs with Zuul |
[production] |
| 08:41 |
<hashar> |
Jenkins is migrating old build histories. Lot of disk IO happening |
[production] |
| 08:12 |
<hashar> |
shutdowning Jenkins for upgrade. |
[production] |
| 05:57 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 8 05:57:10 UTC 2015 (duration 57m 9s) |
[production] |
| 05:46 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1041, warm up (duration: 00m 13s) |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.26wmf13) at 2015-07-08 02:31:24+00:00 |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-08 02:16:50+00:00 |
[production] |
| 02:16 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 00m 48s) |
[production] |