4501-4550 of 10000 results (66ms)
2019-03-04 §
06:46 <marostegui> Stop MySQL on dbstore1002 for decommission T210478 T172410 T216491 T215589 [production]
06:38 <marostegui> Stop MySQL on labsdb1010 for mysql upgrade [production]
06:34 <gtirloni> downtimed cloudstore1008/9 (T209527) [production]
06:13 <marostegui> Upgrade MySQL on db2041 db2049 db2056 db2095 [production]
06:06 <marostegui> Run analyze table logging on db2038 and db2059 - T71222 [production]
06:05 <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 [production]
06:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1094:3314 for schema change (duration: 01m 11s) [production]
05:18 <kart_> Started manual run of unpublished ContentTranslation draft purge script (T217310) [production]
2019-03-03 §
12:26 <volans|off> restarted icinga on icinga2001, stale status file, too many open files [production]
10:44 <elukey> restart pdfrender on scb1003 [production]
2019-03-02 §
12:12 <gtirloni> labstore1006 started nfsd T217473 [production]
2019-03-01 §
20:45 <ejegg> turned off fundraising omnimail process unsubscribes job [production]
19:40 <XioNoX> pre-configure asw-a8 ports on asw2-a8-eqiad - T187960 [production]
19:32 <XioNoX> pre-configure asw-a7 ports on asw2-a7-eqiad - T187960 [production]
19:29 <XioNoX> pre-configure asw-a6 ports on asw2-a6-eqiad - T187960 [production]
19:17 <XioNoX> pre-configure asw-a5 ports on asw2-a5-eqiad - T187960 [production]
18:53 <robh> notebook1003 has unusually high load recently (23) and seemed to lag in reporting to icinga. no hardware failures, pinged about it in #wikimedia-analytics [production]
16:33 <jbond42> rolling security update of bind9 packages on jessie and trusty [production]
15:38 <ema> trafficserver_8.0.2-1wm1 uploaded to stretch-wikimedia [production]
15:02 <akosiaris> restore proton config values [production]
14:33 <hashar> Updating all debian-glue Jenkins job to properly take in account the BUILD_TIMEOUT parameter # T217403 [production]
13:24 <moritzm> removed sca* hosts from debmonitor database [production]
12:49 <akosiaris> lower max_render_queue_size: to 20 for proton on proton100{1,2} [production]
12:32 <akosiaris> restart proton1002, OOM showed up [production]
12:31 <akosiaris> restart proton on proton1001, counted 99 chromium processes left running since at least Jan 30 [production]
11:47 <jbond42> rebooting labsdb1005.codfw.wmnet [production]
11:17 <jbond42> rebooting labstore2004.codfw.wmnet [production]
11:05 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1094 (duration: 00m 50s) [production]
08:52 <godog> temporarily stop prometheus instances on prometheus2004 to take a snapshot [production]
07:44 <oblivian@deploy1001> Synchronized README: Test deploy for new scap configuration (duration: 00m 48s) [production]
07:39 <oblivian@deploy1001> Synchronized README: noop sync to test opcache-manager (duration: 00m 47s) [production]
07:31 <oblivian@deploy1001> Synchronized README: Test deploy for new scap configuration (duration: 00m 46s) [production]
07:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1094 after mysql upgrade (duration: 00m 47s) [production]
07:23 <_joe_> installed php 7.2 compatible packages on deploy1001,2001 [production]
07:12 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1094 after mysql upgrade (duration: 00m 47s) [production]
06:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1094 after mysql upgrade (duration: 00m 46s) [production]
06:48 <marostegui> Deploy schema change on s4 codfw, lag will appear on s4 codfw - T86342 [production]
06:43 <marostegui> Stop MySQL on db1094 for mysql upgrade [production]
06:40 <_joe_> upgrading php extensions on deploy* to versions compatible with php7.2 [production]
05:59 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 51s) [production]
00:12 <XioNoX> pre-configure asw-a3 ports on asw2-a3-eqiad - T187960 [production]
00:09 <thcipriani@deploy1001> Synchronized README: noop sync to test opcache-manager in scap 3.9.1-1 (duration: 00m 48s) [production]
2019-02-28 §
23:44 <XioNoX> pre-configure asw-a2 ports on asw2-a2-eqiad - T187960 [production]
23:31 <XioNoX> pre-configure asw-a1 ports on asw2-a1-eqiad - T187960 [production]
23:27 <bblack@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp107[678]\.eqiad\.wmnet [production]
23:07 <robh> decom cp1045-cp1055, all are role spare but may icinga alert for ping [production]
22:39 <ejegg> updated fundraising CiviCRM from c81fe7a4fd to 616c58cebe [production]
22:33 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Remove old translate config (duration: 00m 46s) [production]
22:29 <reedy@deploy1001> Synchronized wmf-config/CommonSettings.php: Disable some translate special pages again T217376 (duration: 00m 47s) [production]
22:29 <ottomata> replaying events from mediawki eventbus config outage - T217385 [production]