1701-1750 of 10000 results (39ms)
2017-04-19 ยง
17:03 <filippo@naos> Started deploy [prometheus/jmx_exporter@7327459]: test deploy from naos [production]
17:02 <godog> bounce tcpircbot on einsteinium to pick up changes [production]
17:02 <_joe_> running manally enwiki refreshLinks jobs to catch up a bit [production]
16:59 <papaul> power balancing on mw2215 [production]
16:58 <Amir1> ladsgroup@naos:~$ mwscript extensions/ORES/maintenance/CleanDuplicateScores.php --wiki=enwiki froze [production]
16:49 <Amir1> ladsgroup@naos:~$ mwscript extensions/ORES/maintenance/CleanDuplicateScores.php --wiki=enwiki (T163337) [production]
16:33 <godog> deploy.fixurl on G@deployment_target:* after deployment server switchover [production]
16:20 <gehel> disabling deprecation warning logs on elasticsearch eqiad - T163345 [production]
16:19 <jynus> setting db2033 as read write [production]
16:13 <godog> run puppet on naos.codfw.wmnet - new deployment server [production]
16:09 <MediaWiki_> test [production]
16:03 <gehel> disabling deprecation warning logs on elasticsearch codfw - T163345 [production]
15:55 <User___> test [production]
15:51 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: dc=codfw,cluster=elasticsearch,name=elastic2020.* [production]
15:49 <jynus> shutting down db2033 (x1-master) [production]
15:48 <oblivian@puppetmaster1001> conftool action : set/pooled=inactive; selector: dc=codfw,cluster=appserver,name=mw2256.* [production]
15:48 <jynus@tin> Synchronized wmf-config/db-codfw.php: Failing over x1-master (duration: 00m 41s) [production]
15:46 <gehel@puppetmaster1001> conftool action : set/pooled=no; selector: name=elastic2020.codfw.wmnet [production]
15:42 <jynus@tin> Synchronized wmf-config/InitialiseSettings.php: Disable cx_translation- it is causing an outage on x1 (duration: 02m 44s) [production]
15:40 <dzahn@puppetmaster2001> conftool action : set/pooled=no; selector: name=mw2256.codfw.wmnet [production]
15:32 <mutante> mw2256 went down and showed " PANIC: double fault, error_code: 0x0" [production]
15:16 <jynus@tin> Synchronized wmf-config/db-codfw.php: Pool db2055 as an additional API server (duration: 01m 02s) [production]
15:11 <_joe_> ran cumin 'R:class = role::mediawiki::jobrunner and *.eqiad.wmnet' 'systemctl reset-failed' manually [production]
15:07 <godog> start swiftrepl on ms-fe1005 for codfw switchover [production]
15:04 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_restart_parsoid(eqiad, codfw) Successfully completed [production]
14:53 <akosiaris@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet,service=apache2 [production]
14:53 <akosiaris@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet,service=nginx [production]
14:48 <akosiaris@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw2256.codfw.wmnet,service=nginx [production]
14:48 <akosiaris@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw2256.codfw.wmnet,service=apache2 [production]
14:46 <gehel> banning elastic2020 from codfw cluster - T149006 [production]
14:46 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_restart_parsoid(eqiad, codfw) Rolling restart parsoid in eqiad and codfw [production]
14:44 <oblivian@tin> Synchronized wmf-config/ProductionServices.php: Fix redis locks (duration: 02m 24s) [production]
14:41 <akosiaris> powercycle mw2256 [production]
14:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_tendril(eqiad, codfw) Successfully completed [production]
14:33 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_tendril(eqiad, codfw) Update Tendril configuration for the new masters [production]
14:33 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_start_maintenance(eqiad, codfw) Successfully completed [production]
14:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_start_maintenance(eqiad, codfw) Start MediaWiki maintenance in the new master DC [production]
14:31 <switchdc> (volans@sarin) END TASK - switchdc.stages.t09_restore_ttl(eqiad, codfw) Successfully completed [production]
14:31 <switchdc> (volans@sarin) START TASK - switchdc.stages.t09_restore_ttl(eqiad, codfw) Restore the TTL of all the MediaWiki discovery records [production]
14:30 <switchdc> (volans@sarin) END TASK - switchdc.stages.t08_stop_mediawiki_readonly(eqiad, codfw) Successfully completed [production]
14:30 <switchdc> (volans@sarin) MediaWiki read-only period ends at: 2017-04-19 14:30:05.678665 [production]
14:30 <root@tin> Synchronized wmf-config/db-codfw.php: Set MediaWiki in read-write mode in datacenter codfw (duration: 00m 18s) [production]
14:29 <switchdc> (volans@sarin) START TASK - switchdc.stages.t08_stop_mediawiki_readonly(eqiad, codfw) Set MediaWiki in read-write mode (db_to config already merged and git pulled) [production]
14:28 <switchdc> (volans@sarin) END TASK - switchdc.stages.t07_coredb_masters_readwrite(eqiad, codfw) Successfully completed [production]
14:28 <switchdc> (volans@sarin) START TASK - switchdc.stages.t07_coredb_masters_readwrite(eqiad, codfw) set core DB masters in read-write mode [production]
14:25 <switchdc> (volans@sarin) END TASK - switchdc.stages.t06_redis(eqiad, codfw) Successfully completed [production]
14:25 <switchdc> (volans@sarin) START TASK - switchdc.stages.t06_redis(eqiad, codfw) Switch the Redis replication [production]
14:25 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_traffic(eqiad, codfw) Successfully completed [production]
14:22 <switchdc> (volans@sarin) START TASK - switchdc.stages.t05_switch_traffic(eqiad, codfw) Switch traffic flow to the appservers in the new datacenter [production]
14:22 <switchdc> (volans@sarin) END TASK - switchdc.stages.t05_switch_datacenter(eqiad, codfw) Successfully completed [production]