|
2018-01-11
ยง
|
| 14:57 |
<chasemp> |
reboot tools-exec-1401 again... |
[tools] |
| 14:53 |
<chasemp> |
reboot tools-exec-1401 |
[tools] |
| 14:47 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=ms-fe1007.eqiad.wmnet |
[production] |
| 14:47 |
<godog> |
continue swift frontend eqiad roll-restart, ms-fe1007 / ms-fe1008 |
[production] |
| 14:46 |
<chasemp> |
install metltdown kernel and reboot workers 1011-1016 as jessie pilot |
[tools] |
| 14:46 |
<joal> |
Deploy refinery onto HDFS |
[analytics] |
| 14:45 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Promote db2040 as the new codfw-s7 master (duration: 01m 22s) |
[production] |
| 14:40 |
<moritzm> |
rolling reboot of prometheus in codfw for kernel security update |
[production] |
| 14:37 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: mw1271.eqiad.wmnet |
[production] |
| 14:36 |
<joal@tin> |
Finished deploy [analytics/refinery@ed8ecbc]: Patching interlanguage link and manually add a jar to our collection (duration: 04m 10s) |
[production] |
| 14:36 |
<jynus> |
running scap pull on mw1271 |
[production] |
| 14:33 |
<joal> |
Deploy refinery with Scap |
[analytics] |
| 14:32 |
<joal@tin> |
Started deploy [analytics/refinery@ed8ecbc]: Patching interlanguage link and manually add a jar to our collection |
[production] |
| 14:26 |
<moritzm> |
powercycling mw1271 |
[production] |
| 14:25 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:17 |
<jynus> |
upgrade and restart db2029 |
[production] |
| 14:16 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:403584|Create extendedconfirmed for kowiki (T184675)]] (duration: 01m 23s) |
[production] |
| 14:13 |
<akosiaris> |
set migration_downtime to 2000ms for seaborgium |
[production] |
| 14:07 |
<joal> |
Manually restarting banner streaming job to prevent alerting |
[analytics] |
| 14:05 |
<hashar> |
Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403654 |
[releng] |
| 14:01 |
<moritzm> |
reboot hafnium for kernel security update |
[production] |
| 14:00 |
<moritzm> |
reboot tungsten for kernel security update |
[production] |
| 13:58 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1099:3318 weight (duration: 01m 15s) |
[production] |
| 13:56 |
<jynus> |
perform master switchover of s7 codfw |
[production] |
| 13:45 |
<hashar> |
Migrate composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403647 |
[releng] |
| 13:42 |
<moritzm> |
rebooting ores2* for kernel security update |
[production] |
| 13:34 |
<jynus> |
upgrade and restart db2077 |
[production] |
| 13:34 |
<moritzm> |
rebooting bast2001 for kernel security update |
[production] |
| 13:31 |
<moritzm> |
migrating instances off ganeti1001 for subsequent reboot for kernel security update |
[production] |
| 13:27 |
<moritzm> |
failover the ganeti master in eqiad to ganeti1004 |
[production] |
| 13:23 |
<joal> |
Killing banner-streaming job to have it auto-restarted from cron |
[analytics] |
| 12:39 |
<volans> |
Icinga failover back to einsteinium completed - T170353 |
[production] |
| 12:38 |
<moritzm> |
rearmed keyholder on naos |
[production] |
| 12:36 |
<moritzm> |
migrating instances off ganeti1007 for subsequent reboot for kernel security update |
[production] |
| 12:34 |
<moritzm> |
rebooting naos for kernel security update |
[production] |
| 12:28 |
<volans> |
Start Icinga failover back to einsteinium - T170353 |
[production] |
| 12:15 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 with low weight (duration: 01m 44s) |
[production] |
| 12:07 |
<marostegui> |
Stop replication in sync db1089 db1099:3311 - T162807 |
[production] |
| 12:03 |
<moritzm> |
migrating instances off ganeti1006 for subsequent reboot for kernel security update |
[production] |
| 11:45 |
<elukey> |
re-run webrequest-load-wf-text-2018-1-11-8 (failed due to reboots) |
[analytics] |
| 11:39 |
<joal> |
rerun mediacounts-load-wf-2018-1-11-8 |
[analytics] |
| 11:33 |
<moritzm> |
migrating instances off ganeti1005 for subsequent reboot for kernel security update |
[production] |
| 11:14 |
<moritzm> |
migrating instances off ganeti1004 for subsequent reboot for kernel security update |
[production] |
| 11:07 |
<moritzm> |
reboot remaining job runners in eqiad for kernel security update (along with update to HHVM 3.18.6) |
[production] |
| 11:02 |
<akosiaris> |
upload cg3_1.0.0~r12254-1+wmf1_amd64 to apt.wikimedia.org/jessie-wikimedia/main |
[production] |
| 11:02 |
<moritzm> |
migrating instances off ganeti1003 for subsequent reboot for kernel security update |
[production] |
| 10:56 |
<akosiaris> |
upload apertium_3.4.2~r68466-3+wmf1_amd64to apt.wikimedia.org/jessie-wikimedia/main T181464 |
[production] |
| 10:54 |
<akosiaris> |
set kvm:migration_downtime to 30ms for both eqiad/codfw ganeti clusters. Then set migration_downtime 30000 for nitrogen/nihal |
[production] |
| 10:52 |
<moritzm> |
rearmed keyholder on tin |
[production] |
| 10:48 |
<joal> |
Restarting banner-streaming job after hadoop nodes reboot |
[analytics] |