|
2015-12-02
§
|
| 20:37 |
<Jeff_Green> |
enable mail queue monitoring for fundraising |
[production] |
| 19:13 |
<jynus> |
restarting mysql on db2067 to test a configuration change |
[production] |
| 18:07 |
<mutante> |
mw1136 - hhvm restart |
[production] |
| 14:45 |
<Coren> |
deploying cleanup of labs PAM configuration - this should be a functional noop but may cause some puppet noise |
[production] |
| 12:41 |
<akosiaris> |
restart cassandra on maps-test200{1,2,3,4}.codfw.wmnet |
[production] |
| 11:23 |
<moritzm> |
restarting cassandra on restbase100[78] (subsequently) (to effect openjdk security updates plus related libs) |
[production] |
| 11:06 |
<moritzm> |
restarting cassandra on restbase100[2-4] (subsequently) (to effect openjdk security updates plus related libs) |
[production] |
| 11:04 |
<moritzm> |
restart cassandra on restbase1001 (to effect openjdk security updates plus related libs) |
[production] |
| 10:21 |
<aude@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 27s) |
[production] |
| 10:20 |
<aude@tin> |
Synchronized dblists/arbitraryaccess.dblist: Enabling data access for wikinews, wikispecies and mediawiki.org (duration: 00m 30s) |
[production] |
| 10:14 |
<_joe_> |
clearing the job cache for ocg1003 |
[production] |
| 09:22 |
<_joe_> |
stopped ocg service on ocg1003 |
[production] |
| 08:38 |
<_joe_> |
planet1001 recovered as soon as I did get into console via gnt-instance |
[production] |
| 06:25 |
<ori@tin> |
Synchronized php-1.27.0-wmf.7/extensions/NavigationTiming: Idb675cdce: Add isHiDPI and isHttp2 properties; drop isHttps (T119014) (duration: 00m 50s) |
[production] |
| 05:53 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Dec 2 05:53:21 UTC 2015 (duration 53m 20s) |
[production] |
| 02:27 |
<mutante|away> |
labcontrol2001 - disable puppet, kill from puppet stored configs |
[production] |
| 02:26 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.7) (duration: 10m 47s) |
[production] |
| 00:29 |
<mutante> |
puppetstoredconfigclean.rb labcontrol2001.wikimedia.org fixes icinga config |
[production] |
|
2015-12-01
§
|
| 22:41 |
<mutante> |
stop/start ntp service on lvs2002 |
[production] |
| 20:54 |
<mutante> |
re-enabled puppet on cache::misc, replaced planet cert |
[production] |
| 20:19 |
<mutante> |
temp. disabling puppet on cache::misc |
[production] |
| 18:14 |
<twentyafterfour> |
restarted apache on iridium to deploy https://phabricator.wikimedia.org/rPHEXd724c51a4144f7de548546875097da57ee1b38d7 |
[production] |
| 17:05 |
<moritzm> |
restarting cassandra instances on restbase200[3-6] (to effect openjdk security updates and a few depending libs) |
[production] |
| 17:00 |
<moritzm> |
restarting cassandra instances on restbase2002 (to effect openjdk security updates and a few depending libs) |
[production] |
| 16:55 |
<moritzm> |
restarting cassandra instances on restbase2001 (to effect openjdk security updates and a few depending libs) |
[production] |
| 16:22 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for all accounts on eswiki. Part II [[gerrit:250474]] (duration: 00m 28s) |
[production] |
| 16:21 |
<thcipriani@tin> |
Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor for all accounts on eswiki. Part I [[gerrit:250474]] (duration: 01m 29s) |
[production] |
| 12:59 |
<godog> |
reimage mw2208 |
[production] |
| 10:50 |
<moritzm> |
restarting on cassandra instances on restbase1008 (to effect openjdk security updates and a few depending libs) |
[production] |
| 06:52 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Dec 1 06:52:40 UTC 2015 (duration 52m 39s) |
[production] |
| 02:26 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.7) (duration: 10m 14s) |
[production] |
| 02:18 |
<mutante> |
restarted hhvm on mw1147 |
[production] |
| 01:47 |
<ori@tin> |
Synchronized docroot and w: (no message) (duration: 00m 30s) |
[production] |
| 00:47 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.7/extensions/VisualEditor: SWAT (duration: 00m 29s) |
[production] |
| 00:47 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.7/resources/src/mediawiki/mediawiki.ForeignStructuredUpload.js: SWAT (duration: 00m 30s) |
[production] |
|
2015-11-30
§
|
| 21:55 |
<mutante> |
re-wrote l10nupdate cron; restarted cron service on tin |
[production] |
| 20:05 |
<apergos> |
re-enabled puppet on neodymium, minion testing concluded for now |
[production] |
| 19:47 |
<gwicke> |
running `nodetool decommission` on restbase1009 in preparation for the conversion to the multi-instance setup, per https://phabricator.wikimedia.org/T95253# |
[production] |
| 19:31 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: rm deprecated/unused rate limit log config (duration: 00m 28s) |
[production] |
| 17:27 |
<demon@tin> |
Synchronized php-1.27.0-wmf.7/extensions/WikimediaMaintenance/: need maint script errywhere (duration: 00m 28s) |
[production] |
| 16:51 |
<thcipriani@tin> |
Synchronized php-1.27.0-wmf.7/extensions/ContentTranslation/modules/draft/ext.cx.draft.js: SWAT: Add some extra information to save failure logging [[gerrit:255956]] (duration: 00m 28s) |
[production] |
| 16:38 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable QuickSurveys reader segmentation survey [[gerrit:255448]] (duration: 00m 28s) |
[production] |
| 16:30 |
<paravoid> |
mw1002 service hhvm restart |
[production] |
| 16:17 |
<paravoid> |
rolling back to kernel 3.19 on lvs2001/2/3 |
[production] |
| 15:29 |
<paravoid> |
stopping pybal on lvs2001/2/3 |
[production] |
| 15:21 |
<paravoid> |
switching lvs2004/5/6 traffic back to lvs2001/2/3 |
[production] |
| 15:13 |
<paravoid> |
switching lvs2001/2/3 traffic to lvs2004/5/6 and upgrading kernels |
[production] |
| 15:12 |
<_joe_> |
restarting HHVM on mw1147 too, same reason as mw1114 |
[production] |
| 15:10 |
<_joe_> |
restarting hhvm on mw1114, stuck in __pthread_cond_wait () [folly::EventBase::runInEventBaseThreadAndWait ()], apparently blocked in writing to stdout |
[production] |
| 15:02 |
<paravoid> |
switching traffic from lvs4002 to lvs4004; upgrading lvs4002's kernel |
[production] |