|
2016-05-30
§
|
| 12:35 |
<gehel> |
re-enabling puppet on elasticsearch codfw cluster (T133125) |
[production] |
| 12:07 |
<gehel> |
nginx restarted on elasticsearch codfw cluster (T133125) |
[production] |
| 11:42 |
<moritzm> |
upgrading hhvm in codfw (also picking up updated versions of icu and lcms) |
[production] |
| 10:57 |
<moritzm> |
upgrading hhvm on remaining canaries (also picking up updated versions of icu and lcms) |
[production] |
| 10:35 |
<hashar> |
Restarted Zuul. |
[production] |
| 10:31 |
<moritzm> |
upgrading hhvm on mw1017 (also picking up updated versions of icu and lcms) |
[production] |
| 10:30 |
<hashar> |
Zuul deadlocked :( |
[production] |
| 10:12 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Reduce db1071 load (duration: 00m 48s) |
[production] |
| 09:57 |
<moritzm> |
installing libidn security updates on jessie systems |
[production] |
| 09:09 |
<gehel> |
shutting down elasticsearch on codfw for upgrade (T133125) |
[production] |
| 08:44 |
<volans> |
Align thread_pool_max_threads to my.cnf value on 1 slave/shard in eqiad (db1065,db1076,db1078,db1040,db1026,db1061,db1039) T133333 |
[production] |
| 08:27 |
<gehel> |
starting elasticsearch upgrade on codfw (T133125) |
[production] |
| 02:30 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon May 30 02:30:46 UTC 2016 (duration 5m 52s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.3) (duration: 09m 04s) |
[production] |
|
2016-05-27
§
|
| 23:58 |
<ori> |
Forcing a Puppet run on logstash* |
[production] |
| 23:44 |
<legoktm> |
ran extensions/GlobalBlocking/fixGlobalBlockWhitelist.php for T56496 |
[production] |
| 23:15 |
<legoktm@tin> |
Synchronized php-1.28.0-wmf.3/includes/actions/RollbackAction.php: RollbackAction: Don't return true, causes '1' to be output (duration: 00m 34s) |
[production] |
| 22:50 |
<legoktm> |
mwscript initSiteStats.php --wiki=lrcwiki --update for T109635 |
[production] |
| 19:49 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.3/includes/Linker.php: Iba17ce55ff9 (duration: 00m 25s) |
[production] |
| 19:49 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.3/includes/actions/RollbackAction.php: Iba17ce55ff9 (duration: 00m 31s) |
[production] |
| 19:06 |
<paravoid> |
un-draining esams |
[production] |
| 18:15 |
<thcipriani> |
restarted zuul due to deadlock issue |
[production] |
| 17:10 |
<volans> |
Stop slave, stop mysql and shutdown es2017 and es2019 for hardware maintenance T130702 |
[production] |
| 16:34 |
<dcausse> |
elasticsearch in codfw: creating jamwiki index |
[production] |
| 16:34 |
<volans> |
Align runtime MySQL max_connections on codfw masters with the my.cnf ones T133333 |
[production] |
| 16:04 |
<paravoid> |
shutting down ms-fe3001/ms-fe3002 |
[production] |
| 15:26 |
<paravoid> |
draining esams for network maintenance |
[production] |
| 15:02 |
<krenair@tin> |
Synchronized php-1.28.0-wmf.3/extensions/CentralNotice/resources/subscribing: rv due to T136387 (duration: 00m 36s) |
[production] |
| 14:36 |
<paravoid> |
restarting pybal on lvs3003/lvs3004 |
[production] |
| 14:19 |
<mobrovac> |
change-prop deployed 3747ebd |
[production] |
| 13:54 |
<urandom> |
Bootstrapping restbase2003-b.codfw.wmnet : T134016 |
[production] |
| 12:27 |
<hashar> |
CI/Zuul deadlocked quickly due to a dependency set on a repository not known to Zuul |
[production] |
| 12:22 |
<volans> |
Align runtime MySQL configurations on codfw slaves with the my.cnf ones T133333 |
[production] |
| 11:27 |
<elukey> |
restarted jmxtrans on kafka10* hosts |
[production] |
| 10:11 |
<volans> |
restarting MySQL on db2038 to test change 286858 - T133333 |
[production] |
| 10:04 |
<moritzm> |
disable firejail test on mw1153, all went well, but rather revert back since it's Friday and enable this along with the other image scalers on Monday |
[production] |
| 09:14 |
<moritzm> |
enable firejail for image scaling on mw1153 as a canary |
[production] |
| 08:57 |
<volans> |
Set sync_binlog=1 on db2011 (m2) and monitoring it. T133333 |
[production] |
| 08:00 |
<elukey> |
restarted memcached on mc1009 to collect metrics for T129963 |
[production] |
| 07:04 |
<_joe_> |
updating HHVM on the remaining hosts (mira, wasat, snapshot1*) |
[production] |
| 02:33 |
<ejegg> |
rolled back payments from de0398a244094f2bad6bc70eefce8388a616e575 to e6807395d7687d521070b83d159b77b242e5c04f |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri May 27 02:31:01 UTC 2016 (duration 9m 12s) |
[production] |
| 02:21 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.3) (duration: 07m 50s) |
[production] |