|
2021-05-11
ยง
|
| 20:52 |
<legoktm> |
upgraded mailman3 on lists1001 |
[production] |
| 20:37 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host people2002.codfw.wmnet |
[production] |
| 20:24 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 06m 57s) |
[production] |
| 20:17 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] |
[production] |
| 20:17 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@270c753] (thin): Regular analytics weekly train THIN [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 00m 05s) |
[production] |
| 20:17 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@270c753] (thin): Regular analytics weekly train THIN [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] |
[production] |
| 20:17 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@270c753]: Regular analytics weekly train [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 17m 01s) |
[production] |
| 20:00 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@270c753]: Regular analytics weekly train [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] |
[production] |
| 19:59 |
<mforns> |
repeating refinery deployment (matching source v0.1.11) - missed unmerged change |
[analytics] |
| 19:55 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host people2002.codfw.wmnet |
[production] |
| 19:53 |
<mforns> |
finished refinery deployment (matching source v0.1.11) |
[analytics] |
| 19:46 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@7e0598d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 09m 45s) |
[production] |
| 19:37 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@7e0598d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] |
[production] |
| 19:33 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.5 |
[production] |
| 19:29 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@7e0598d] (thin): Regular analytics weekly train THIN [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 00m 07s) |
[production] |
| 19:29 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@7e0598d] (thin): Regular analytics weekly train THIN [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] |
[production] |
| 19:28 |
<mforns@deploy1002> |
Finished deploy [analytics/refinery@7e0598d]: Regular analytics weekly train [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 45m 45s) |
[production] |
| 18:54 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1011.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:53 |
<otto@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Migrate VirtualPageView to EventPlatform on testwiki - T238138 (duration: 01m 09s) |
[production] |
| 18:52 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1011.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:43 |
<mforns@deploy1002> |
Started deploy [analytics/refinery@7e0598d]: Regular analytics weekly train [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] |
[production] |
| 18:41 |
<mforns> |
starting refinery deployment (matching source v0.1.11) |
[analytics] |
| 18:20 |
<dancy@deploy1002> |
Finished scap: testwikis wikis to 1.37.0-wmf.5 (duration: 09m 43s) |
[production] |
| 18:10 |
<dancy@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.5 |
[production] |
| 18:00 |
<andrewbogott> |
adding 'trove' service project in advance of deploying trove in eqiad1 |
[admin] |
| 17:36 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) (duration: 01m 25s) |
[production] |
| 17:35 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:35 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) |
[production] |
| 17:34 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) (duration: 02m 27s) |
[production] |
| 17:33 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:32 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) |
[production] |
| 17:31 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@2604d7b]: testing default policy deployment in codfw1dev (duration: 01m 59s) |
[production] |
| 17:29 |
<andrew@deploy1002> |
Started deploy [horizon/deploy@2604d7b]: testing default policy deployment in codfw1dev |
[production] |
| 17:26 |
<mforns> |
deployed refinery-source v0.1.11 |
[analytics] |
| 17:20 |
<mutante> |
the backend for people.wikimedia.org switched from people1002 to people1003, the people.wikimedia.org CNAME has been updated. MOTD is about to be updated to inform users. |
[production] |
| 17:18 |
<legoktm> |
disabled pipermail redirects on lists.wikimedia.org |
[production] |
| 17:17 |
<Majavah> |
shutdown and delete tools-checker-03 T278540 |
[tools] |
| 17:14 |
<Majavah> |
move floating ip 185.15.56.61 to tools-checker-04 |
[tools] |
| 17:12 |
<Majavah> |
add tools-checker-04 as a grid submit host T278540 |
[tools] |
| 17:07 |
<dancy@deploy1002> |
scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) |
[production] |
| 16:58 |
<Majavah> |
add tools-checker-04 to toollabs::checker_hosts hiera key T278540 |
[tools] |
| 16:49 |
<Majavah> |
creating tools-checker-04 with buster T278540 |
[tools] |
| 16:32 |
<Majavah> |
carefully shutdown tools-k8s-haproxy-1 T252239 |
[tools] |
| 16:29 |
<Majavah> |
carefully shutdown tools-k8s-haproxy-2 T252239 |
[tools] |
| 16:12 |
<jynus> |
restarting bacula-dir on backup1001, stuck process |
[production] |
| 15:59 |
<dancy@deploy1002> |
rebuilt and synchronized wikiversions files: (no justification provided) |
[production] |
| 15:58 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mwlog1001.eqiad.wmnet |
[production] |
| 15:55 |
<bstorm> |
restart haproxy on dbproxy1018/9 to remove old config |
[production] |
| 15:47 |
<herron@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mwlog1001.eqiad.wmnet |
[production] |
| 15:38 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mwlog2001.codfw.wmnet |
[production] |