251-300 of 10000 results (39ms)
2017-05-03 §
08:53 <_joe_> rebooting restbase1018 T163280 [production]
08:24 <_joe_> deactivating restbase1018-vg for RAID failover and rebuild T163280 [production]
08:01 <hashar> Rolling back Jenkins 2.46.2 -> 2.46.1 - T144106 [production]
07:53 <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106 [production]
07:42 <_joe_> rebuilding RAIDs on restbase1018 T163280 [production]
07:35 <hashar> Restarting Nodepool to catch up with python-jenkins 0.4.14 [production]
07:35 <moritzm> updated python-jenkins on labnodepool1001 to 0.4.14 (needed by latest Jenkins LTS) [production]
02:48 <l10nupdate@naos> ResourceLoader cache refresh completed at Wed May 3 02:48:33 UTC 2017 (duration 5m 21s) [production]
02:43 <l10nupdate@naos> scap sync-l10n completed (1.29.0-wmf.21) (duration: 14m 02s) [production]
01:41 <mutante> kubernetes - puppet fails because "E: Unable to locate package cni [production]
2017-05-02 §
23:42 <TimStarling> EtcdConfig changes all reverted [production]
23:17 <tstarling@puppetmaster1001> conftool action : set/@read-only.yaml; selector: name=ReadOnly,scope=eqiad [production]
23:07 <TimStarling> scap pull on mw2017 and mwdebug1001 for etcd testing [production]
23:00 <TimStarling> locking scap on naos for deployment of EtcdConfig https://gerrit.wikimedia.org/r/#/c/351132/ [production]
22:57 <_joe_> upgrading python-conftool across the fleet [production]
22:38 <mutante> gerrit (cobalt/gerrit2001) - deployed firewall change to allow ssh between gerrit servers for clustering, new iptables rules exist now (T152525) [production]
21:52 <jynus> running previously failed alter tables on s3-eqiad T163912 [production]
21:33 <jynus> creating missing math table on bdwikimedia (s3) [production]
20:04 <hashar> Restarting Jenkins for plugin rollback [production]
17:51 <bblack> codfw->eqiad switchback: end-user edge traffic back to normal @ eqiad ( https://gerrit.wikimedia.org/r/#/c/351330/ ) - 10 minute TTL for bulk traffic pattern shift starts now. [production]
17:50 <mobrovac@naos> Finished deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 (duration: 06m 04s) [production]
17:48 <papaul> new db servers signing puppet certs,salt-key, initial run [production]
17:44 <mobrovac@naos> Started deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 [production]
17:40 <END> (PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) [production]
17:39 <mobrovac@naos> Finished deploy [restbase/deploy@6adb0f2]: (no justification provided) (duration: 01m 34s) [production]
17:38 <START> - Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) [production]
17:37 <mobrovac@naos> Started deploy [restbase/deploy@6adb0f2]: (no justification provided) [production]
17:37 <END> (PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:37 <START> - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:35 <END> (PASS) - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <MediaWiki> read-only period ends at: 2017-05-02 17:35:48.111079 (switchdc/volans@neodymium) [production]
17:35 <START> - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <oblivian@puppetmaster1001> conftool action : set/val=test; selector: name=ReadOnly,scope=codfw [production]
17:33 <END> (PASS) - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:33 <START> - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:32 <END> (PASS) - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) [production]
17:32 <START> - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) [production]
17:31 <END> (PASS) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:31 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:23 <END> (FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:23 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:20 <END> (PASS) - Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) [production]
17:17 <START> - Switch traffic flow to the appservers from eqiad to codfw - t05_switch_traffic (switchdc/volans@neodymium) [production]
17:08 <END> (FAIL) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:08 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/volans@neodymium) [production]
17:05 <catrope@naos> Synchronized php-1.29.0-wmf.21/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTarget.js: T164157 (duration: 01m 00s) [production]
17:03 <END> (FAIL) - Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) [production]
17:03 <START> - Set core DB masters in read-only mode in eqiad, ensure all masters are read-only - t03_coredb_masters_readonly (switchdc/volans@neodymium) [production]
16:58 <END> (FAIL) - Set MediaWiki in read-only mode in eqiad - t02_start_mediawiki_readonly (switchdc/volans@neodymium) [production]
16:57 <MediaWiki> read-only period starts at: 2017-05-02 16:57:37.952132 (switchdc/volans@neodymium) [production]