|
2021-04-20
ยง
|
| 23:09 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on furud.codfw.wmnet with reason: REIMAGE |
[production] |
| 23:05 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:03 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on flerovium.eqiad.wmnet with reason: REIMAGE |
[production] |
| 22:20 |
<bd808> |
`clush -w @all -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` |
[tools] |
| 22:19 |
<bd808> |
`clush -w @exec -b "sudo exiqgrep -z -i | xargs sudo exim -Mt"` |
[tools] |
| 22:14 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:10 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:52 |
<bd808> |
Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad1.wikimedia.cloud`. Was using wrong domain name in prior update. |
[tools] |
| 21:49 |
<bstorm> |
tagged the latest maintain-kubeusers and deployed to toolforge (with kustomize changes to rbac) after testing in toolsbeta T280300 |
[tools] |
| 21:46 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:44 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1016.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:27 |
<bd808> |
Update hiera `profile::toolforge::active_mail_relay: tools-mail-03.tools.eqiad.wmflabs`. was -2 which is decommed. |
[tools] |
| 20:52 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=ruwiki # T279853 |
[production] |
| 20:48 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd1020.wikimedia.org |
[production] |
| 20:41 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=viwiki # T279853 |
[production] |
| 20:36 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd1020.wikimedia.org |
[production] |
| 20:36 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=ukwiki # T279853 |
[production] |
| 20:35 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd[1017-1019].wikimedia.org |
[production] |
| 20:34 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=tewiki # T279853 |
[production] |
| 20:32 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=svwiki # T279853 |
[production] |
| 20:30 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=srwiki # T279853 |
[production] |
| 20:29 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=rowiki # T279853 |
[production] |
| 20:27 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=hywiki # T279853 |
[production] |
| 20:22 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=huwiki # T279853 |
[production] |
| 20:21 |
<andrewbogott> |
reboot cloudservices1003 |
[admin] |
| 20:21 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=hrwiki # T279853 |
[production] |
| 20:18 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=hewiki # T279853 |
[production] |
| 20:16 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=frwiktionary # T279853 |
[production] |
| 20:16 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd[1017-1019].wikimedia.org |
[production] |
| 20:15 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=euwiki # T279853 |
[production] |
| 20:13 |
<andrewbogott> |
reboot cloudservices1004 |
[admin] |
| 20:13 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=bnwiki # T279853 |
[production] |
| 20:08 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:03 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:58 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:56 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:28 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudcephosd1016.wikimedia.org |
[production] |
| 19:01 |
<bstorm> |
updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244 |
[toolsbeta] |
| 18:34 |
<Urbanecm> |
mwscript extensions/GrowthExperiments/maintenance/migrateMentorMenteeRelationship.php --wiki=idwiki # T279853 |
[production] |
| 18:33 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd1016.wikimedia.org |
[production] |
| 18:29 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.1/extensions/GrowthExperiments/: 4d1969d: 1fbb8e9: MentorStore: Set wasPosted to true in command line mode (T275773) (duration: 00m 59s) |
[production] |
| 17:26 |
<XioNoX> |
boot cr1-codfw:fpc1 - T277341 |
[production] |
| 17:16 |
<papaul> |
Adding a MPC7E to cr1-codfw |
[production] |
| 16:32 |
<arturo> |
merging change to core route firewall https://gerrit.wikimedia.org/r/c/operations/homer/public/+/681316 (T272587) |
[production] |
| 16:15 |
<andrewbogott> |
updating core routers config with https://gerrit.wikimedia.org/r/c/operations/homer/public/+/681315 |
[production] |
| 15:51 |
<elukey> |
move analytics-hive.eqiad.wmnet back to an-coord1001 (test on an-coord1002 successful) |
[analytics] |
| 15:38 |
<ottomata> |
deployed refiner to hdfs |
[analytics] |
| 15:38 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host eventlog1003.eqiad.wmnet |
[production] |
| 15:21 |
<urbanecm@deploy1002> |
Synchronized docroot/noc/conf/debug.json: dc6647b9c674429c0811116e0caca7639b766e77: remove mwdebug1003 from list of debug servers (T267248) (duration: 00m 58s) |
[production] |
| 15:20 |
<urbanecm@deploy1002> |
Synchronized debug.json: dc6647b9c674429c0811116e0caca7639b766e77: remove mwdebug1003 from list of debug servers (T267248) (duration: 00m 57s) |
[production] |