751-800 of 10000 results (41ms)
2017-04-18 §
11:37 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:35 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed [production]
11:35 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters [production]
11:34 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(codfw, eqiad) Successfully completed [production]
11:34 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters [production]
11:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed [production]
11:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records [production]
11:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
11:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) [production]
11:30 <switchdc> (volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed [production]
11:30 <switchdc> (volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode [production]
11:18 <switchdc> (volans@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed [production]
11:18 <switchdc> (volans@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication [production]
11:14 <moritzm> upgrading logstash* to Linux 4.9 [production]
10:58 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed [production]
10:56 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter [production]
10:56 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed [production]
10:55 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter [production]
10:48 <switchdc> (volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute [production]
10:48 <switchdc> (volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode [production]
10:43 <switchdc> (volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Successfully completed [production]
10:43 <switchdc> (volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) [production]
10:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute [production]
10:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Successfully completed [production]
10:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records [production]
10:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed [production]
10:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts [production]
10:28 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Failed to execute [production]
10:28 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records [production]
10:26 <switchdc> (volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed [production]
10:26 <switchdc> (volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts [production]
10:25 <volans> Final test of switchdc steps in the codfw->eqiad configuration, only idempotent changes, T160178 [production]
10:25 <moritzm> installing wireshark security updates [production]
10:20 <moritzm> uploaded HHVM 3.18.2+wmf2 for jessie-wikimedia/experimental (includes fix for T162354) [production]
09:52 <oblivian:> Setting zotero in codfw UP [production]
09:50 <_joe_> testing switchover script for services, will act on zotero in codfw [production]
09:45 <_joe_> adding 60G to the ocg output partition on ocg1003 [production]
09:17 <oblivian@neodymium> conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw [production]
09:03 <volans> upgrading conftool to v0.4.1 on neodymium/sarin [production]
07:48 <_joe_> uploaded python-conftool 0.4.1 to jessie-wikimedia [production]
07:42 <_joe_> cleaning up orphaned COW images in /var/cache/pbuilder/build/ on copper [production]
06:16 <marostegui> For the record: restarted s7 instance on db1069 - T163183 [production]
00:36 <catrope@tin> Synchronized php-1.29.0-wmf.20/extensions/MobileFrontend/resources/mobile.mainMenu/mainmenu.less: T163059 (duration: 03m 07s) [production]
2017-04-17 §
23:37 <mutante> runnin rmmod acpi_pad on the 16 R320 via cumin, since blacklisting in puppet does not actively remove, confirmed unloaded. (16/16) success ratio (>= 100.0% threshold) for command: 'lsmod|grep -c acpi_pad ||:' (T162850) [production]
23:33 <mutante> running puppet via cumin on all 16 Dell PowerEdge R320, adding blacklist file for acpi_pad kernel module. 15/16 success, all but tin (T162850) [production]
22:46 <catrope@tin> Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 03m 01s) [production]
22:42 <mutante> tin - load average going down, acpi_pad processes gone, cpu usage low again (T163158) [production]
22:40 <mutante> tin - rmmod acpi_pad (T163158) [production]
22:08 <catrope@tin> Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 16m 23s) [production]