351-400 of 10000 results (51ms)
2018-10-05 §
07:20 <godog> temporarily stop prometheus on bast4001 to finalize data transfer - T179050 [production]
07:19 <jynus> stopping s3 replication on db1075 [production]
07:18 <jynus> stopping s5 replication on db1070 [production]
07:09 <moritzm> installing python3.4/2.7 security updates [production]
05:55 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T205599 - Ic28e00c30 (duration: 00m 57s) [production]
05:53 <_joe_> upgrading python-etcd on conf1004-6, restarting etcdmirror [production]
05:13 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Clarify db1092 status - T205514 (duration: 00m 57s) [production]
04:18 <krinkle@deploy1001> Synchronized php-1.32.0-wmf.24/includes/libs/filebackend/FileBackendStore.php: T205567 - I75f1eb6dc2cb (duration: 00m 56s) [production]
04:16 <krinkle@deploy1001> Synchronized php-1.32.0-wmf.24/extensions/CirrusSearch/includes/DataSender.php: I0769c50c (duration: 01m 01s) [production]
00:31 <mutante> LDAP: added user skvjold to group wmf (T204377) [production]
2018-10-04 §
22:51 <ejegg> updated fundraising CiviCRM from 944b954bac to ebc2e0076c [production]
21:27 <XioNoX> bounce phab1001 switch port - T201039 [production]
20:47 <ejegg> updated fundraising CiviCRM from ddf4865650 to 944b954bac [production]
20:23 <mforns@deploy1001> Finished deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 (duration: 00m 17s) [production]
20:22 <mforns@deploy1001> Started deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 [production]
20:10 <mforns@deploy1001> Finished deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 (duration: 14m 04s) [production]
19:56 <mforns@deploy1001> Started deploy [analytics/refinery@3eb9bf2]: deploying refinery together with refinery-source v0.0.76 [production]
19:30 <marxarelli> rise in fatals "Fatal error: entire web request took longer than 60 seconds and timed out in /srv/mediawiki/php-1.32.0-wmf.24/includes/Title.php" [production]
19:26 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.24 [production]
19:15 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@6dc89c0]: Bump cirrusSearchLinksUpdate concurrency to 50 (duration: 00m 53s) [production]
19:14 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@6dc89c0]: Bump cirrusSearchLinksUpdate concurrency to 50 [production]
18:49 <sbisson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:460202|]] (duration: 00m 59s) [production]
18:24 <XioNoX> bounce lvs1002:eth1 switch port [production]
18:23 <sbisson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:464510|Enable PageTriage/ORES on enwiki (T206149)]] (duration: 01m 01s) [production]
18:21 <bblack> lvs1002: puppet disabled, stopping pybal (fail to 1005) [production]
18:07 <_joe_> disabled notifications for etcd replication lag on conf1005, not in production [production]
17:47 <banyek> repooling labsb1010 (T195747) [production]
17:41 <_joe_> uploaded new python-etcd packages for jessie, stretch [production]
17:38 <XioNoX> asw2-b-eqiad recabling done - T201039 [production]
17:34 <elukey> pool kafka1002 (eventbus) after maintenance [production]
17:22 <elukey> re-enable ircecho after alarms shower [production]
17:15 <andrewbogott> triggering some alerts on labvirt1018 to figure out about alert thresholds [production]
17:06 <elukey> stop ircecho on einstenium - alarms shower [production]
17:02 <gtirloni> tools - published updated toollabs-* Docker images [production]
16:54 <ejegg> updated standalone SmashPig deploy from 82f9d49c23 to 5f21d3f2db [production]
16:52 <XioNoX> Step 3) Add missing links - T201039 [production]
16:45 <shdubsh> etherpad1001 running systemctl reset-failed [production]
16:41 <XioNoX> Connect/enable fpc2:0/51-fpc5:1/0 (5m DAC) - T201039 [production]
16:39 <XioNoX> Enable fpc5-fpc7 - T201039 [production]
16:33 <twentyafterfour> started phd on phab1001 and re-enabled puppet (I had it disabled to prevent starting phd during read-only) [production]
16:25 <twentyafterfour> phabricator is read-write [production]
16:21 <jynus> reloading dbproxy1003,8 [production]
16:16 <marostegui> Stop and reboot db1072 (phabricator master) for maintenance [production]
16:16 <twentyafterfour> phabricator is read-only [production]
16:14 <XioNoX> Enable all VC ports on FPC2 and FPC7 - T201039 [production]
16:13 <XioNoX> starting asw2-b-eqiad re-cabling - T201039 [production]
16:08 <twentyafterfour> logged downtime for phabricator in icinga, stopped phd queue processing in preparation for read-only mode [production]
16:07 <jynus> reloading haproxy @ dbproxy1005 [production]
16:00 <marostegui> Stop MySQL on db1073 for mariadb and kernel upgrade - T201039 T148507 [production]
15:58 <arturo> icinga downtime every server in the main cloudvps deployment for 2h T201039 [production]