3201-3250 of 10000 results (50ms)
2017-05-09 §
07:31 <marostegui> Stop replication at the same position on db1050 and db2028 - T147166 T130067 [production]
07:27 <marostegui> Disable replication codfw > eqiad on s6 - T147166 T130067 [production]
07:12 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool es2019 - T149526 (duration: 00m 39s) [production]
07:11 <elukey> reboot kafka1014 for kernel upgrades [production]
07:01 <_joe_> installing the new version of python-service-checker across the fleet [production]
06:37 <marostegui> Run pt-table-checksum on s7.cawiki - T163190 [production]
06:01 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2038 - T162539 T163548 (duration: 00m 41s) [production]
05:54 <marostegui> Deploy alter table on wikidatawiki.wb_terms on codfw master db2023 - https://phabricator.wikimedia.org/T162539 - https://phabricator.wikimedia.org/T163548 [production]
02:27 <l10nupdate@tin> ResourceLoader cache refresh completed at Tue May 9 02:27:46 UTC 2017 (duration 5m 58s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.21) (duration: 08m 17s) [production]
2017-05-08 §
23:21 <bd808@tin> Synchronized wmf-config/wikitech.php: Revert "Disable creation of new forms on wikitech" (T53642) (duration: 01m 10s) [production]
22:54 <bd808@tin> Finished deploy [striker/deploy@00e8545]: openstack: Role modifications require global admin rights (T164787) (duration: 00m 27s) [production]
22:54 <bd808@tin> Started deploy [striker/deploy@00e8545]: openstack: Role modifications require global admin rights (T164787) [production]
22:17 <bd808> Deleted 2fa for user Mdann52 on wikitech after verifying account ownership via ssh file creation. T164804 [production]
22:01 <andrewbogott> rebooting labservices1002 to mess with the bios [production]
21:55 <bblack> depooled cp3035 (memory issues - already schedule for FE restart to fix, which will repool when it's reached in the list...) [production]
21:25 <mobrovac@tin> Finished deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 (duration: 01m 39s) [production]
21:24 <mobrovac@tin> Started deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 [production]
21:22 <mobrovac@tin> Finished deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 (duration: 03m 51s) [production]
21:18 <mobrovac@tin> Started deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 [production]
21:17 <mobrovac@tin> Finished deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 (duration: 00m 38s) [production]
21:17 <mobrovac@tin> Started deploy [graphoid/deploy@a288409]: Switched to npm-stored graph-shared, fix mapsnapshot - T164046 [production]
21:12 <mobrovac@tin> Finished deploy [restbase/deploy@c70a1e1]: Remove the mobile-text end point - T158128 (duration: 06m 23s) [production]
21:06 <mobrovac@tin> Started deploy [restbase/deploy@c70a1e1]: Remove the mobile-text end point - T158128 [production]
21:05 <arlolra@tin> Finished deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 (duration: 02m 43s) [production]
21:02 <arlolra@tin> Started deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 [production]
20:48 <arlolra@tin> Finished deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 (duration: 01m 36s) [production]
20:47 <arlolra@tin> Started deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 [production]
20:37 <gehel> silencing elasticsearch shard incinga check, recovery after upgrade is going to take a long time - T161908 [production]
20:34 <arlolra@tin> Finished deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 (duration: 04m 50s) [production]
20:30 <arlolra@tin> Started deploy [parsoid/deploy@0459ae3]: Updating Parsoid to 9d8badc8 [production]
20:27 <gehel> restarted kibana on logstash cluster - T161908 [production]
20:21 <gehel> upgrading kibana on logstash cluster - T161908 [production]
20:02 <gehel> restarting elasticsearch on logstash cluster after upgrade - T161908 [production]
19:47 <gehel> logstash / elasticsearch downtime coming up - T161908 [production]
19:34 <bd808> Deployment of Striker for T162508 complete; will continue debug keystone issue that is preventing Tool Labs membership requests from being approved [production]
19:34 <bblack> restarted varnishxcache service on cp3031, was malfunctioning and sending crazy stats to grafana... [production]
19:28 <gehel> starting ELK (logstash) upgrade - T161908 [production]
19:17 <bd808@tin> Finished deploy [striker/deploy@3836477]: Implement Tool Labs membership application and processing (T162508) (duration: 00m 32s) [production]
19:17 <bd808@tin> Started deploy [striker/deploy@3836477]: Implement Tool Labs membership application and processing (T162508) [production]
19:15 <bd808> Forced puppet run on californium to provision new striker config settings [production]
19:07 <bd808> Applied database migration for T162508 to striker database on m5-master [production]
18:58 <MaxSem> Restarted tilerator and tileratorui across the cluster [production]
18:52 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: T164621 (duration: 00m 39s) [production]
18:50 <bblack> running varnish frontend restarts to fix memory sizing on 256G+ hosts over the next ~4.5 h (mostly text+upload hosts) [production]
18:49 <bblack> cp4006 repooled (frontend restarted) [production]
18:45 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: T164498 (duration: 00m 39s) [production]
18:44 <bblack> running varnish frontend restarts to fix memory sizing on 96GB and 192GB hosts over the next ~45m (mostly maps+misc hosts) [production]
18:41 <catrope@tin> Synchronized php-1.29.0-wmf.21/extensions/Popups/: T163198 (duration: 00m 39s) [production]
18:40 <bblack@neodymium> conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet [production]