4751-4800 of 10000 results (46ms)
2018-07-09 §
10:12 <godog> powercycle ms-be1017 - can't login on ssh/console [production]
10:07 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:444558|Bumping portals to master (T128546)]] (duration: 00m 50s) [production]
10:06 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:444558|Bumping portals to master (T128546)]] (duration: 00m 51s) [production]
09:22 <ppchelko@deploy1001> Finished deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 (duration: 15m 32s) [production]
09:07 <ppchelko@deploy1001> Started deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 [production]
09:02 <vgutierrez> Bump AES128-SHA traffic redirection to 100% - T192555 [production]
08:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 49s) [production]
08:35 <jynus> updating password for labdb1004/5 for admin users [production]
08:31 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521 [production]
08:26 <marostegui> Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379 [production]
08:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s) [production]
08:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s) [production]
08:08 <marostegui> Deploy schema change on db1105:3311 T146591 T197891 T196379 [production]
08:07 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s) [production]
08:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s) [production]
07:54 <marostegui> Deploy schema change on db1099:3311 T146591 T197891 T196379 [production]
07:54 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s) [production]
07:49 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s) [production]
07:40 <marostegui> Deploy schema change on db1067 T146591 T197891 T196379 [production]
07:40 <ema> reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825 [production]
07:37 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s) [production]
07:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s) [production]
07:18 <marostegui> Deploy schema change on db1119 - T146591 T197891 T196379 [production]
07:18 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s) [production]
07:18 <elukey> update filter analytics-in4 on cr1/cr2 eqiad [production]
07:06 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s) [production]
06:54 <marostegui> Deploy schema change on db1089 - T146591 T197891 T196379 [production]
06:53 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s) [production]
06:41 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459 [production]
06:30 <marostegui> Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379 [production]
06:25 <elukey> restart hue on thorium to pick up new smtp changes - T196920 [production]
06:13 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459 [production]
06:12 <marostegui> Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379 [production]
06:03 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521 [production]
06:01 <marostegui> Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379 [production]
05:55 <marostegui> Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379 [production]
05:53 <marostegui> Deploy schema change on db1076 - T146591 T197891 T196379 [production]
05:41 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521 [production]
05:31 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459 [production]
05:31 <marostegui> Deploy schema change on db1090 - T146591 T197891 T196379 [production]
05:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s) [production]
05:11 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379 [production]
05:04 <marostegui> Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459 [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s) [production]
02:44 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s) [production]
02:33 <l10nupdate@deploy1001> scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s) [production]
2018-07-08 §
20:37 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: I81d566ba860 (duration: 00m 51s) [production]
20:30 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: Id94ca5f55c2c6 (duration: 00m 53s) [production]
19:10 <chasemp> disable puppet on labstore1007 to turn down ratelimits for nginx (load is rising and rising) [production]
11:27 <elukey> restart rsyslog on lithium - in:imtcp thread stuck at 99% cpu usage [production]