501-550 of 10000 results (42ms)
2017-07-10 §
12:22 <marostegui@tin> scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
12:12 <marostegui> Deploy alter table on db1091 - T168661 [production]
12:11 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1091 - T168661 (duration: 00m 42s) [production]
12:02 <marostegui> Upgrade db1102 to 10.1 and enable rbr triggers - T153743 [production]
12:02 <moritzm> installing bind security updates (we only have client libs/tools installed) [production]
11:39 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore original weight for db1097 - T168661 (duration: 00m 42s) [production]
11:36 <marostegui> Stop MySQL on db1102 for maintenance - T153743 [production]
11:16 <moritzm> installing libgcrypt and expat security updates [production]
11:16 <kartik@tin> Finished deploy [cxserver/deploy@c209bec]: Update cxserver to 3375da5 (duration: 02m 49s) [production]
11:13 <kartik@tin> Started deploy [cxserver/deploy@c209bec]: Update cxserver to 3375da5 [production]
10:28 <addshore> WMDE Summer campaign deploy slot DONE [production]
10:28 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:364184|WMDE Summer campaign - Add logging]] (fix spacing) NOOP (duration: 00m 43s) [production]
10:22 <addshore@tin> Synchronized php-1.30.0-wmf.7/extensions/WikimediaEvents: [[gerrit:364172|WMDE Summer campaign - Add hook]] (duration: 00m 42s) [production]
10:21 <addshore@tin> Synchronized php-1.30.0-wmf.7/extensions/CentralAuth: CentralAuth (undeployed patches) [[gerrit:363892]], [[gerrit:363893]], [[gerrit:363891]] & revert [[gerrit:364182]] T169261 (duration: 00m 47s) [production]
10:13 <addshore> reverting https://gerrit.wikimedia.org/r/#/c/363891 as it is sitting on tin undeployed T169261 [production]
09:59 <moritzm> rebooting mc2* servers for kernel update [production]
09:54 <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:362380|WMDE Summer campaign - Add logging]] (duration: 00m 45s) [production]
09:10 <marostegui> Compress innodb on wikidata on dbstore2001 [production]
09:00 <moritzm> rebooting mw1168 (video scaler) for kernel update [production]
08:52 <moritzm> rebooting mwlog2001 for kernel update [production]
08:35 <moritzm> rebooting ms1001 for kernel update [production]
08:29 <moritzm> rebooting francium for kernel update [production]
08:17 <marostegui> Deploy alter table on db1097 - T168661 [production]
08:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 46s) [production]
08:16 <marostegui@tin> scap failed: average error rate on 2/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
08:03 <marostegui> Drop database l10nwiki on s2 - T119811 [production]
07:53 <moritzm> rebooting hafnium for kernel update [production]
07:18 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=cp3009.* [production]
07:13 <moritzm> reboot netmon1001 for kernel update [production]
06:11 <marostegui> Deploy alter table on s1 - db1080 and db1067 - T166204 [production]
06:00 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1080, depool db1067 - T166204 (duration: 00m 42s) [production]
05:59 <marostegui@tin> scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
02:27 <l10nupdate@tin> scap failed: average error rate on 2/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/3888cca979647b9381a7739b0bdbc88e for details) [production]
2017-07-08 §
22:14 <bd808> Deleted ukwikimedia records in CentralAuth localuser and localnames tables for T170005. [production]
2017-07-07 §
21:54 <legoktm@tin> Synchronized php-1.30.0-wmf.7/extensions/CentralAuth/: Fix handling of password hash upgrade on login - T169261 (duration: 00m 45s) [production]
21:52 <demon@tin> Synchronized wmf-config/interwiki.php: Updating interwiki cache, T169979 (duration: 00m 43s) [production]
15:07 <marostegui> Stop MySQL on db1102 for MariaDB upgrade [production]
15:00 <dcausse> deleting commonswiki_file_1499379383 on elastic@eqiad (failed reindex) [production]
12:20 <elukey> restart mysql on dbstore1002 - high swap used [production]
11:40 <moritzm> rebooting rdb* servers in codfw for kernel update [production]
10:30 <gehel> restarting elastic1043 (corrupted statistics) [production]
09:42 <gehel> unbanning elastic1020 and 1026 from elasticsearch eqiad [production]
09:37 <gehel> restarting elastic1036 (corrupted statistics) [production]
09:30 <TabbyCat> Global rename of Idh0854 → Garam has finished (T167031) [production]
09:24 <moritzm> installing NTP security updates on trusty hosts [production]
09:23 <akosiaris> schedule a month's worth of downtime for ores100X [production]
08:56 <moritzm> restarting HHVM on app server canaries to pick up libgcrypt and expat updates [production]
08:54 <_joe_> reenabling puppet across the fleet [production]
08:52 <_joe_> restarting apache on all puppetmaster, after a successful puppet run [production]
08:39 <_joe_> disabling puppet across the fleet for enabling directory environments in puppet [production]