1851-1900 of 10000 results (72ms)
2018-12-17 §
08:21 <marostegui> Deploy schema change on db1077 with replication (lag will be generated on labsdb:s3) T86338 T202167 [production]
08:21 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1077 T86338 T202167 (duration: 00m 45s) [production]
08:16 <marostegui> Stop replication on labsdb1011 [production]
08:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1123 T86338 T202167 (duration: 00m 45s) [production]
07:58 <marostegui> Enable GTID on db2052 (s5 master) - T211973 [production]
06:48 <marostegui> Enable GTID on db2035 (s2 master) - T211973 [production]
06:36 <marostegui> Enable GTID on db2034 (x1 master) - T211973 [production]
06:36 <marostegui> Enable GTID on db2034 (x1 master) [production]
06:29 <marostegui> Enable replication consistency options on codfw masters - T211973 [production]
06:28 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1123 T86338 T202167 (duration: 00m 46s) [production]
06:28 <marostegui> Depool labsdb1010 - T86338 [production]
04:22 <tstarling@deploy1001> Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 44s) [production]
04:20 <tstarling@deploy1001> Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 59s) [production]
2018-12-16 §
09:52 <elukey> mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine) [production]
07:30 <marostegui> Reboot db1115 after OOM [production]
07:28 <marostegui> Stop MySQL on db1115 so tendril can get back to work [production]
2018-12-15 §
13:03 <volans|off> restarted pdfrender on scb1004 [production]
09:22 <elukey> mask + reset-failed kafkatee default instance on weblog1001 [production]
2018-12-14 §
23:59 <mutante> LDAP: added aezell to wmf group (T211945) for grafana access [production]
22:00 <XioNoX> increase accepted-prefix-limit for HE to 200000 [production]
20:38 <mutante> sulfur systemctl restart nagios-nrpe-server [production]
20:32 <andrew@deploy1001> Finished deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855 (duration: 03m 17s) [production]
20:29 <andrew@deploy1001> Started deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855 [production]
20:13 <otto@deploy1001> Finished deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833 (duration: 06m 04s) [production]
20:07 <otto@deploy1001> Started deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833 [production]
20:07 <otto@deploy1001> deploy aborted: (no justification provided) (duration: 00m 00s) [production]
20:07 <otto@deploy1001> Started deploy [analytics/refinery@ef1f7c6]: (no justification provided) [production]
17:22 <cmjohnson1> mw1272 down for h/w troubleshooting [production]
14:38 <marostegui> Enable GTID on db2039 (s6 codfw master) - T211973 [production]
14:03 <marostegui> Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973 [production]
13:59 <marostegui> Enable notifications for db1095 (s3 lag check)- T211973 [production]
13:57 <marostegui> Enable notifications for db2068 (s7 lag check)- T211973 [production]
13:47 <dcausse> elastic@codfw copying index data from the main cluster to psi & omega (test disk usage & import speed) [production]
13:42 <marostegui> Enable GTID on db1124:3318 - T211973 [production]
11:11 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2083 db2068 db2067 after mysql and kernel upgrade (duration: 00m 44s) [production]
10:51 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2083 db2068 db2067 for mysql and kernel upgrade (duration: 00m 45s) [production]
10:50 <marostegui> Stop MySQL on db2083, db2068 and db2067 for mysql and kernel upgrade [production]
10:19 <filippo@deploy1001> Synchronized wmf-config/logging.php: wmf-config/InitialiseSettings-labs.php (duration: 00m 44s) [production]
10:18 <filippo@deploy1001> Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s) [production]
09:33 <marostegui> Deploy schema change on db1095:3313 T86338 T202167 [production]
09:28 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2049, db2059, db2070 after mysql and kernel upgrade (duration: 00m 45s) [production]
09:21 <banyek> global user rename is in progress - T209488 [production]
08:58 <marostegui> Stop MySQL on db2049, db2059 and db2070 for mysql and kernel upgrade [production]
08:56 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2049, db2059, db2070 for mysql and kernel upgrade (duration: 00m 43s) [production]
08:50 <elukey> swap oxygen with weblog1001 [production]
08:48 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2084 and db2088 after mysql and kernel upgrade (duration: 00m 44s) [production]
08:47 <elukey> disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001) [production]
07:58 <marostegui> Deploy schema change on dbstore1002:s3 T86338 T202167 [production]
07:49 <marostegui> Upgrade mysql and kernel on db2084 and db2088 [production]
07:49 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2084 and db2088 for mysql and kernel upgrade (duration: 00m 45s) [production]