2401-2450 of 10000 results (50ms)
2018-07-09 §
08:31 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521 [production]
08:26 <marostegui> Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379 [production]
08:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s) [production]
08:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s) [production]
08:08 <marostegui> Deploy schema change on db1105:3311 T146591 T197891 T196379 [production]
08:07 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s) [production]
08:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s) [production]
07:54 <marostegui> Deploy schema change on db1099:3311 T146591 T197891 T196379 [production]
07:54 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s) [production]
07:49 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s) [production]
07:40 <marostegui> Deploy schema change on db1067 T146591 T197891 T196379 [production]
07:40 <ema> reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825 [production]
07:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s) [production]
07:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s) [production]
07:18 <marostegui> Deploy schema change on db1119 - T146591 T197891 T196379 [production]
07:18 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s) [production]
07:18 <elukey> update filter analytics-in4 on cr1/cr2 eqiad [production]
07:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s) [production]
06:54 <marostegui> Deploy schema change on db1089 - T146591 T197891 T196379 [production]
06:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s) [production]
06:41 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459 [production]
06:30 <marostegui> Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379 [production]
06:25 <elukey> restart hue on thorium to pick up new smtp changes - T196920 [production]
06:13 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459 [production]
06:12 <marostegui> Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379 [production]
06:03 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521 [production]
06:01 <marostegui> Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379 [production]
05:55 <marostegui> Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379 [production]
05:53 <marostegui> Deploy schema change on db1076 - T146591 T197891 T196379 [production]
05:41 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521 [production]
05:31 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459 [production]
05:31 <marostegui> Deploy schema change on db1090 - T146591 T197891 T196379 [production]
05:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s) [production]
05:11 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379 [production]
05:04 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459 [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s) [production]
02:44 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s) [production]
2018-07-08 §
20:37 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I81d566ba860 (duration: 00m 51s) [production]
20:30 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: Id94ca5f55c2c6 (duration: 00m 53s) [production]
19:10 <chasemp> disable puppet on labstore1007 to turn down ratelimits for nginx (load is rising and rising) [production]
11:27 <elukey> restart rsyslog on lithium - in:imtcp thread stuck at 99% cpu usage [production]
2018-07-07 §
23:25 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: Icbbccf495899 (duration: 00m 50s) [production]
23:23 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I001f68a1b39 (duration: 00m 53s) [production]
23:18 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: I1b42c260b5ee3 (duration: 01m 00s) [production]
13:02 <_joe_> rebooting lvs2001, panic logs from the ethernet card module [production]
2018-07-06 §
18:08 <foks> reset 2FA for User:Howcheng [production]
16:06 <chasemp> labcontrol1003:~# /sbin/reboot T198950 [production]
13:36 <moritzm> installing openldap security updates on trusty [production]
13:22 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1103:3312 - T197459 [production]