|
2016-03-02
§
|
| 12:29 |
<bblack> |
restarted logstash on logstash1001 |
[production] |
| 12:27 |
<_joe_> |
puppet disabled on both scb1001/2, depooled scb1001 for moborovac to test and config manually patched on scb1002 so that it runs with the old code correctly |
[production] |
| 12:25 |
<mobrovac> |
mobileapps rolling back to 68e38ec7, problems found in the latest deploy for T113542 |
[production] |
| 12:00 |
<mobrovac> |
mobileapps stopping the service on scb1001 for debug purposes, T113542 |
[production] |
| 11:56 |
<_joe_> |
stopped puppet on scb1002, depooled scb1001 from mobileapps |
[production] |
| 11:36 |
<mobrovac> |
mobileapps deploying d384f1ba |
[production] |
| 11:09 |
<jynus> |
profiling db1023 and db1061 for 24 hours- 1/20th of the queries slightly slower |
[production] |
| 10:42 |
<moritzm> |
restarting graphite-web on graphite1001 (for django security update) |
[production] |
| 10:42 |
<hashar> |
Zuul should no more be caught in death loop due to Depends-On on an event-schemas change. Hole filled with https://gerrit.wikimedia.org/r/#/c/274356/ T128569 |
[production] |
| 10:36 |
<elukey> |
stopped Redis multi-instance on rdb1006 (Job Queue slave) as pre-step for Debian re-image |
[production] |
| 10:16 |
<gehel> |
elastic1004.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 09:43 |
<volans> |
Cloning es2005->es2014, es2007->es2016, es2009->es2018, see T127330 |
[production] |
| 09:30 |
<moritzm> |
installing nodejs updates on restbase* |
[production] |
| 09:19 |
<elukey> |
redis multi-instance stopped on rdb1004 (jobqueue slave) as pre-step for Debian re-image |
[production] |
| 09:16 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Depooling external storage DBs in codfw for migration: T127330 (duration: 01m 24s) |
[production] |
| 09:13 |
<hashar> |
Zuul went crazy / caught in a loop of doom. Same has Saturday. It went back magically at 08:32 UTC T128569 |
[production] |
| 08:48 |
<gehel> |
elastic1003.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 08:33 |
<moritzm> |
installing Django security updates |
[production] |
| 08:17 |
<_joe_> |
disabling puppet on all memcached hosts in preparation for enabling ipsec |
[production] |
| 07:35 |
<legoktm@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable $wgReferrerPolicy on private wikis (duration: 01m 01s) |
[production] |
| 06:45 |
<_joe_> |
rebooting serpens |
[production] |
| 03:04 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Mar 2 03:04:14 UTC 2016 (duration 8m 49s) |
[production] |
| 02:55 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.15) (duration: 09m 31s) |
[production] |
| 02:29 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 12m 32s) |
[production] |
| 00:45 |
<krenair@tin> |
Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 17s) |
[production] |
| 00:44 |
<krenair@tin> |
Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 16s) |
[production] |
| 00:31 |
<krenair@tin> |
Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s) |
[production] |
| 00:30 |
<krenair@tin> |
Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s) |
[production] |
| 00:26 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272926/ - prepare for VE default switch on dewiki (duration: 01m 17s) |
[production] |
| 00:12 |
<krenair@tin> |
Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/274129/ - +testwiki (duration: 01m 20s) |
[production] |
| 00:10 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/274129/ - VE SET on mediawikiwiki/testwiki (duration: 01m 21s) |
[production] |
| 00:04 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/271932/ - disable Gather on enwiki (duration: 01m 26s) |
[production] |
|
2016-03-01
§
|
| 23:57 |
<ebernhardson> |
upgrade elastic1002.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 23:17 |
<mutante> |
maps-test2001 - could not find dependency for postgres class is NOT related to my recent change. icinga crit since a long time |
[production] |
| 22:34 |
<mutante> |
re-enabled puppet runs on all mw* servers, mediawiki roles now in modules/role/manifests/mediawiki/ |
[production] |
| 22:27 |
<mutante> |
temp. disabling puppet runs on mw appservers to be extra safe during mediawiki module change |
[production] |
| 21:29 |
<gehel> |
elastic1001.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 20:29 |
<demon@tin> |
Finished scap: group0 to wmf.15 (duration: 31m 24s) |
[production] |
| 19:58 |
<demon@tin> |
Started scap: group0 to wmf.15 |
[production] |
| 19:19 |
<jynus> |
testing heartbeat in m5 (db1009, db2030) |
[production] |
| 19:14 |
<demon@tin> |
scap aborted: testwikis to wmf.15 and rebuild l10n (duration: 01m 19s) |
[production] |
| 19:14 |
<chasemp> |
clean out /var/log/atop and /var/log/account on iridium |
[production] |
| 19:13 |
<demon@tin> |
Started scap: testwikis to wmf.15 and rebuild l10n |
[production] |
| 18:53 |
<mutante> |
iridium - gzip /var/log/atop/atop_20160* |
[production] |
| 18:51 |
<mutante> |
iridium: apt-get clean for some more disk space |
[production] |
| 18:49 |
<subbu> |
finished deploying parsoid sha 1f7ed5d0 |
[production] |
| 18:44 |
<subbu> |
synced parsoid code; restarted parsoid on wtp1002 as a canary |
[production] |
| 18:41 |
<subbu> |
starting parsoid deploy |
[production] |
| 17:52 |
<gehel> |
elastic2024.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 17:18 |
<mobrovac> |
restbase rolling-restart restbase for https://gerrit.wikimedia.org/r/#/c/273974/ |
[production] |