1551-1600 of 10000 results (32ms)
2016-03-02 §
12:27 <_joe_> puppet disabled on both scb1001/2, depooled scb1001 for moborovac to test and config manually patched on scb1002 so that it runs with the old code correctly [production]
12:25 <mobrovac> mobileapps rolling back to 68e38ec7, problems found in the latest deploy for T113542 [production]
12:00 <mobrovac> mobileapps stopping the service on scb1001 for debug purposes, T113542 [production]
11:56 <_joe_> stopped puppet on scb1002, depooled scb1001 from mobileapps [production]
11:36 <mobrovac> mobileapps deploying d384f1ba [production]
11:09 <jynus> profiling db1023 and db1061 for 24 hours- 1/20th of the queries slightly slower [production]
10:42 <moritzm> restarting graphite-web on graphite1001 (for django security update) [production]
10:42 <hashar> Zuul should no more be caught in death loop due to Depends-On on an event-schemas change. Hole filled with https://gerrit.wikimedia.org/r/#/c/274356/ T128569 [production]
10:36 <elukey> stopped Redis multi-instance on rdb1006 (Job Queue slave) as pre-step for Debian re-image [production]
10:16 <gehel> elastic1004.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
09:43 <volans> Cloning es2005->es2014, es2007->es2016, es2009->es2018, see T127330 [production]
09:30 <moritzm> installing nodejs updates on restbase* [production]
09:19 <elukey> redis multi-instance stopped on rdb1004 (jobqueue slave) as pre-step for Debian re-image [production]
09:16 <volans@tin> Synchronized wmf-config/db-codfw.php: Depooling external storage DBs in codfw for migration: T127330 (duration: 01m 24s) [production]
09:13 <hashar> Zuul went crazy / caught in a loop of doom. Same has Saturday. It went back magically at 08:32 UTC T128569 [production]
08:48 <gehel> elastic1003.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
08:33 <moritzm> installing Django security updates [production]
08:17 <_joe_> disabling puppet on all memcached hosts in preparation for enabling ipsec [production]
07:35 <legoktm@tin> Synchronized wmf-config/InitialiseSettings.php: Disable $wgReferrerPolicy on private wikis (duration: 01m 01s) [production]
06:45 <_joe_> rebooting serpens [production]
03:04 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Mar 2 03:04:14 UTC 2016 (duration 8m 49s) [production]
02:55 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.15) (duration: 09m 31s) [production]
02:29 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.14) (duration: 12m 32s) [production]
00:45 <krenair@tin> Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 17s) [production]
00:44 <krenair@tin> Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ - try #2, this time with the submodule update (duration: 01m 16s) [production]
00:31 <krenair@tin> Synchronized portals: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s) [production]
00:30 <krenair@tin> Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/274316/ (duration: 01m 18s) [production]
00:26 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/272926/ - prepare for VE default switch on dewiki (duration: 01m 17s) [production]
00:12 <krenair@tin> Synchronized dblists/visualeditor-default.dblist: https://gerrit.wikimedia.org/r/#/c/274129/ - +testwiki (duration: 01m 20s) [production]
00:10 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/274129/ - VE SET on mediawikiwiki/testwiki (duration: 01m 21s) [production]
00:04 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/271932/ - disable Gather on enwiki (duration: 01m 26s) [production]
2016-03-01 §
23:57 <ebernhardson> upgrade elastic1002.eqiad.wmnet to elasticsearch 1.7.5 [production]
23:17 <mutante> maps-test2001 - could not find dependency for postgres class is NOT related to my recent change. icinga crit since a long time [production]
22:34 <mutante> re-enabled puppet runs on all mw* servers, mediawiki roles now in modules/role/manifests/mediawiki/ [production]
22:27 <mutante> temp. disabling puppet runs on mw appservers to be extra safe during mediawiki module change [production]
21:29 <gehel> elastic1001.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
20:29 <demon@tin> Finished scap: group0 to wmf.15 (duration: 31m 24s) [production]
19:58 <demon@tin> Started scap: group0 to wmf.15 [production]
19:19 <jynus> testing heartbeat in m5 (db1009, db2030) [production]
19:14 <demon@tin> scap aborted: testwikis to wmf.15 and rebuild l10n (duration: 01m 19s) [production]
19:14 <chasemp> clean out /var/log/atop and /var/log/account on iridium [production]
19:13 <demon@tin> Started scap: testwikis to wmf.15 and rebuild l10n [production]
18:53 <mutante> iridium - gzip /var/log/atop/atop_20160* [production]
18:51 <mutante> iridium: apt-get clean for some more disk space [production]
18:49 <subbu> finished deploying parsoid sha 1f7ed5d0 [production]
18:44 <subbu> synced parsoid code; restarted parsoid on wtp1002 as a canary [production]
18:41 <subbu> starting parsoid deploy [production]
17:52 <gehel> elastic2024.codfw.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) [production]
17:18 <mobrovac> restbase rolling-restart restbase for https://gerrit.wikimedia.org/r/#/c/273974/ [production]
17:05 <thcipriani@tin> Synchronized wmf-config/ProductionServices.php: SWAT: Add kafka1012.eqiad.wmnet back to the media-wiki config [[gerrit:273488]] (duration: 00m 39s) [production]