501-550 of 10000 results (37ms)
2017-05-03 §
11:02 <hashar> Restarting Nodepool [production]
10:58 <moritzm> upgrading nodepool on labnodepool1001 to a package including https://gerrit.wikimedia.org/r/351608 [production]
10:18 <END> (PASS) - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/oblivian@neodymium) [production]
10:17 <START> - Switch MediaWiki master datacenter and read-write discovery records from eqiad to codfw - t05_switch_datacenter (switchdc/oblivian@neodymium) [production]
10:14 <END> (PASS) - Set MediaWiki in read-write mode in codfw (db-codfw config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
10:14 <START> - Set MediaWiki in read-write mode in codfw (db-codfw config already merged and git pulled) - t08_stop_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
10:14 <END> (PASS) - Set MediaWiki in read-only mode in eqiad (db-eqiad config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
10:13 <START> - Set MediaWiki in read-only mode in eqiad (db-eqiad config already merged and git pulled) - t02_start_mediawiki_readonly (switchdc/oblivian@neodymium) [production]
10:13 <_joe_> testing reverted steps of switchdc, non-dry-run --dc-from eqiad --dc-to codfw (should be noop) [production]
10:05 <moritzm> installing icu security updates on trusty (jessie already fixed) [production]
09:50 <marostegui> Restart db1097 to change its binlog to STATEMENT - T155099 [production]
09:19 <elukey> reboot mc[1019-1036].eqiad.wmnet for kernel upgrades [production]
09:18 <moritzm> rebooting restbase1018 for update to Linux 4.9 [production]
09:05 <godog> rebuild mismounted FSes on ms-be1035 - T163673 [production]
08:53 <_joe_> rebooting restbase1018 T163280 [production]
08:24 <_joe_> deactivating restbase1018-vg for RAID failover and rebuild T163280 [production]
08:01 <hashar> Rolling back Jenkins 2.46.2 -> 2.46.1 - T144106 [production]
07:53 <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106 [production]
07:42 <_joe_> rebuilding RAIDs on restbase1018 T163280 [production]
07:35 <hashar> Restarting Nodepool to catch up with python-jenkins 0.4.14 [production]
07:35 <moritzm> updated python-jenkins on labnodepool1001 to 0.4.14 (needed by latest Jenkins LTS) [production]
02:48 <l10nupdate@naos> ResourceLoader cache refresh completed at Wed May 3 02:48:33 UTC 2017 (duration 5m 21s) [production]
02:43 <l10nupdate@naos> scap sync-l10n completed (1.29.0-wmf.21) (duration: 14m 02s) [production]
01:41 <mutante> kubernetes - puppet fails because "E: Unable to locate package cni [production]
2017-05-02 §
23:42 <TimStarling> EtcdConfig changes all reverted [production]
23:17 <tstarling@puppetmaster1001> conftool action : set/@read-only.yaml; selector: name=ReadOnly,scope=eqiad [production]
23:07 <TimStarling> scap pull on mw2017 and mwdebug1001 for etcd testing [production]
23:00 <TimStarling> locking scap on naos for deployment of EtcdConfig https://gerrit.wikimedia.org/r/#/c/351132/ [production]
22:57 <_joe_> upgrading python-conftool across the fleet [production]
22:38 <mutante> gerrit (cobalt/gerrit2001) - deployed firewall change to allow ssh between gerrit servers for clustering, new iptables rules exist now (T152525) [production]
21:52 <jynus> running previously failed alter tables on s3-eqiad T163912 [production]
21:33 <jynus> creating missing math table on bdwikimedia (s3) [production]
20:04 <hashar> Restarting Jenkins for plugin rollback [production]
17:51 <bblack> codfw->eqiad switchback: end-user edge traffic back to normal @ eqiad ( https://gerrit.wikimedia.org/r/#/c/351330/ ) - 10 minute TTL for bulk traffic pattern shift starts now. [production]
17:50 <mobrovac@naos> Finished deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 (duration: 06m 04s) [production]
17:48 <papaul> new db servers signing puppet certs,salt-key, initial run [production]
17:44 <mobrovac@naos> Started deploy [restbase/deploy@6adb0f2]: Include displaytitle and page_id in the summary output and bump the content type version - T163729 T164079 [production]
17:40 <END> (PASS) - Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) [production]
17:39 <mobrovac@naos> Finished deploy [restbase/deploy@6adb0f2]: (no justification provided) (duration: 01m 34s) [production]
17:38 <START> - Start MediaWiki jobrunners, videoscalers and maintenance in codfw - t09_start_maintenance (switchdc/volans@neodymium) [production]
17:37 <mobrovac@naos> Started deploy [restbase/deploy@6adb0f2]: (no justification provided) [production]
17:37 <END> (PASS) - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:37 <START> - Restore the TTL of all the MediaWiki read-write discovery records and cleanup confd stale files - t09_restore_ttl (switchdc/volans@neodymium) [production]
17:35 <END> (PASS) - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <MediaWiki> read-only period ends at: 2017-05-02 17:35:48.111079 (switchdc/volans@neodymium) [production]
17:35 <START> - Set MediaWiki in read-write mode in codfw - t08_stop_mediawiki_readonly (switchdc/volans@neodymium) [production]
17:35 <oblivian@puppetmaster1001> conftool action : set/val=test; selector: name=ReadOnly,scope=codfw [production]
17:33 <END> (PASS) - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:33 <START> - Set core DB masters in read-write mode in codfw, ensure masters in eqiad are read-only - t07_coredb_masters_readwrite (switchdc/volans@neodymium) [production]
17:32 <END> (PASS) - Switch the Redis masters from eqiad to codfw and invert the replication - t06_redis (switchdc/volans@neodymium) [production]