251-300 of 10000 results (36ms)
2018-07-10 §
10:27 <ppchelko@deploy1001> Finished deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out (duration: 06m 39s) [production]
10:20 <ppchelko@deploy1001> Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out [production]
10:20 <ppchelko@deploy1001> deploy aborted: Fix up the invalid Vary header (duration: 12m 50s) [production]
10:07 <ppchelko@deploy1001> Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header [production]
10:07 <moritzm> restarting thumbor on thumbor1001 to pick up exiv2 security updates [production]
10:03 <elukey> restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change [production]
09:34 <elukey> forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected [production]
09:25 <godog> swift eqiad-prod add ms-be1036 back gradually - T196873 [production]
09:05 <moritzm> installing libsoup security updates on jessie/stretch [production]
08:38 <moritzm> installing libjpeg-turbo security updates on trusty [production]
08:37 <godog> drain and restart cassandra-a on restbase2001 to test a restart [production]
08:34 <elukey> rolling restart of AQS to apply the new config [production]
08:09 <godog> disable puppet on hosts running cassandra before merging 444247 and 443114 [production]
08:08 <moritzm> installing openslp security updates on trusty [production]
07:58 <moritzm> installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while) [production]
07:33 <twentyafterfour> deploying fix for phabricator userpage having hard-coded my username [production]
06:53 <twentyafterfour> deployed rPHEX03173dd0097451f60faa3a1705abee58a9fe4c5f [production]
06:32 <marostegui> Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw - T187521 [production]
06:23 <marostegui> Deploy schema change on codfw s7 master (db2040) with replication, this will generate lag on s7 codfw T146591 T197891 T196379 [production]
06:17 <marostegui> Deploy schema change on s8 primary master (db1071) T146591 T197891 T196379 [production]
06:06 <marostegui> Deploy schema change on dbstore1002:s8 T146591 T197891 T196379 [production]
06:02 <marostegui> Deploy schema change on codfw s8 master (db2045) with replication, this will generate lag on s8 codfw T146591 T197891 T196379 [production]
05:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 50s) [production]
05:39 <marostegui> Deploy schema change on s4 primary master (db1068) T146591 T197891 T196379 [production]
05:36 <marostegui> Deploy schema change on db1091 T146591 T197891 T196379 [production]
05:36 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 50s) [production]
05:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 49s) [production]
05:27 <marostegui> Deploy schema change on db1081 T146591 T197891 T196379 [production]
05:27 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 50s) [production]
05:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s) [production]
05:21 <marostegui> Deploy schema change on db1121 with replication, this will generate lag on s4 labs hosts T146591 T197891 T196379 [production]
05:17 <marostegui> Optimize frwiki.wbc_entity_usage on s6 eqiad hosts T187521 [production]
05:16 <marostegui> Deploy schema change on db1103:3314 T146591 T197891 T196379 [production]
05:16 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s) [production]
05:10 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 50s) [production]
05:03 <marostegui> Deploy schema change on db1097:3314 T146591 T197891 T196379 [production]
05:03 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 51s) [production]
04:59 <marostegui> Optimize frwiki.wbc_entity_usage on s6 codfw, this will generate lag on s6 codfw - T187521 [production]
04:57 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s) [production]
04:48 <marostegui> Deploy schema change on db1084 T146591 T197891 T196379 [production]
04:48 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 52s) [production]
04:36 <marostegui> Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1066 (s2 primary master) - T187521 [production]
04:28 <marostegui> Deploy schema change on s1 primary master (db1052) T146591 T197891 T196379 [production]
02:44 <l10nupdate@deploy1001> ResourceLoader cache refresh completed at Tue Jul 10 02:44:29 UTC 2018 (duration 10m 18s) [production]
00:21 <twentyafterfour> deploying https://phabricator.wikimedia.org/rPHABc6f75c918afa1cc59472c5fe226539e093f6c3ef [production]
2018-07-09 §
23:58 <ejegg> updated payments-wiki from ed40f33c44 to 383f667171 [production]
23:58 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s) [production]
23:55 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6 (duration: 00m 50s) [production]
23:53 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813 (duration: 00m 50s) [production]
23:52 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4 (duration: 00m 50s) [production]