|
2017-04-18
§
|
| 11:34 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_tendril(codfw, eqiad) Update Tendril configuration for the new masters |
[production] |
| 11:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Successfully completed |
[production] |
| 11:33 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(codfw, eqiad) Restore the TTL of all the MediaWiki discovery records |
[production] |
| 11:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Successfully completed |
[production] |
| 11:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-write mode (db_to config already merged and git pulled) |
[production] |
| 11:30 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) Successfully completed |
[production] |
| 11:30 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(codfw, eqiad) set core DB masters in read-write mode |
[production] |
| 11:18 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t06_redis(codfw, eqiad) Successfully completed |
[production] |
| 11:18 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t06_redis(codfw, eqiad) Switch the Redis replication |
[production] |
| 11:14 |
<moritzm> |
upgrading logstash* to Linux 4.9 |
[production] |
| 10:58 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Successfully completed |
[production] |
| 10:56 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(codfw, eqiad) Switch traffic flow to the appservers in the new datacenter |
[production] |
| 10:56 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Successfully completed |
[production] |
| 10:55 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t05_switch_datacenter(codfw, eqiad) Switch MediaWiki configuration to the new datacenter |
[production] |
| 10:48 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) Failed to execute |
[production] |
| 10:48 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t03_coredb_masters_readonly(codfw, eqiad) set core DB masters in read-only mode |
[production] |
| 10:43 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Successfully completed |
[production] |
| 10:43 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t02_start_mediawiki_readonly(codfw, eqiad) Set MediaWiki in read-only mode (db_from config already merged and git pulled) |
[production] |
| 10:33 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Failed to execute |
[production] |
| 10:33 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t01_stop_maintenance(codfw, eqiad) Stop MediaWiki maintenance in the old master DC |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Successfully completed |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed |
[production] |
| 10:31 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts |
[production] |
| 10:28 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Failed to execute |
[production] |
| 10:28 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts |
[production] |
| 10:25 |
<volans> |
Final test of switchdc steps in the codfw->eqiad configuration, only idempotent changes, T160178 |
[production] |
| 10:25 |
<moritzm> |
installing wireshark security updates |
[production] |
| 10:20 |
<moritzm> |
uploaded HHVM 3.18.2+wmf2 for jessie-wikimedia/experimental (includes fix for T162354) |
[production] |
| 09:52 |
<oblivian:> |
Setting zotero in codfw UP |
[production] |
| 09:50 |
<_joe_> |
testing switchover script for services, will act on zotero in codfw |
[production] |
| 09:45 |
<_joe_> |
adding 60G to the ocg output partition on ocg1003 |
[production] |
| 09:17 |
<oblivian@neodymium> |
conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw |
[production] |
| 09:03 |
<volans> |
upgrading conftool to v0.4.1 on neodymium/sarin |
[production] |
| 07:48 |
<_joe_> |
uploaded python-conftool 0.4.1 to jessie-wikimedia |
[production] |
| 07:42 |
<_joe_> |
cleaning up orphaned COW images in /var/cache/pbuilder/build/ on copper |
[production] |
| 06:16 |
<marostegui> |
For the record: restarted s7 instance on db1069 - T163183 |
[production] |
| 00:36 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/MobileFrontend/resources/mobile.mainMenu/mainmenu.less: T163059 (duration: 03m 07s) |
[production] |
|
2017-04-17
§
|
| 23:37 |
<mutante> |
runnin rmmod acpi_pad on the 16 R320 via cumin, since blacklisting in puppet does not actively remove, confirmed unloaded. (16/16) success ratio (>= 100.0% threshold) for command: 'lsmod|grep -c acpi_pad ||:' (T162850) |
[production] |
| 23:33 |
<mutante> |
running puppet via cumin on all 16 Dell PowerEdge R320, adding blacklist file for acpi_pad kernel module. 15/16 success, all but tin (T162850) |
[production] |
| 22:46 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 03m 01s) |
[production] |
| 22:42 |
<mutante> |
tin - load average going down, acpi_pad processes gone, cpu usage low again (T163158) |
[production] |
| 22:40 |
<mutante> |
tin - rmmod acpi_pad (T163158) |
[production] |
| 22:08 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 16m 23s) |
[production] |
| 19:16 |
<mutante> |
tegmen test ircecho stop/start service to confirm it's fine on jessie/prod icinga role (that's the passive server) |
[production] |
| 19:02 |
<demon@tin> |
Synchronized wmf-config/: Pruning some old extension message files, co-master sync (duration: 01m 52s) |
[production] |
| 18:58 |
<demon@tin> |
Pruned MediaWiki: 1.29.0-wmf.15 (duration: 00m 14s) |
[production] |
| 18:46 |
<maxsem@tin> |
Finished deploy [tilerator/deploy@001811e]: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only (duration: 00m 19s) |
[production] |