|
2021-04-01
§
|
| 10:49 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ganeti1011.eqiad.wmnet |
[production] |
| 10:39 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2002-dev.codfw.wmnet |
[production] |
| 10:33 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudcephosd2002-dev.codfw.wmnet |
[production] |
| 10:33 |
<dcaro@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcephosd2001-dev.codfw.wmnet |
[production] |
| 10:26 |
<dcaro@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudcephosd2001-dev.codfw.wmnet |
[production] |
| 09:07 |
<hashar> |
contint2001: compressing files with 4 parallel executions: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -print0|xargs -0 -P4 gzip |
[production] |
| 09:01 |
<hashar> |
contint2001: compressing all fresnel trace--trace.json files: sudo -u jenkins find /srv/jenkins/builds/mediawiki-fresnel-patch-docker -name "*trace.json" -exec gzip {} \+ # T249268 |
[production] |
| 08:52 |
<moritzm> |
drain ganeti1011 |
[production] |
| 08:35 |
<moritzm> |
failover Ganeti master in eqiad to ganeti1009 |
[production] |
| 08:25 |
<moritzm> |
installing ldb security updates |
[production] |
| 08:12 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 08:12 |
<kharlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 08:10 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 08:10 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 08:09 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 07:58 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 07:58 |
<kharlan@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 07:55 |
<kharlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 06:37 |
<elukey> |
powercycle cp1087 (no ssh, no tty via serial console) - T278729 |
[production] |
| 06:35 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet |
[production] |
| 02:50 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2386.codfw.wmnet with reason: REIMAGE |
[production] |
| 02:48 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2386.codfw.wmnet with reason: REIMAGE |
[production] |
| 02:21 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2387.codfw.wmnet with reason: REIMAGE |
[production] |
| 02:18 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2387.codfw.wmnet with reason: REIMAGE |
[production] |
| 02:16 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2384.codfw.wmnet with reason: REIMAGE |
[production] |
| 02:13 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2384.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:53 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2385.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:52 |
<Reedy> |
`echo "https://www.mediawiki.org/static/images/footer/poweredby_mediawiki_176x62.png" | mwscript purgeList.php --wiki=enwiki` T268230 |
[production] |
| 01:51 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2385.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:51 |
<Reedy> |
`echo "https://www.mediawiki.org/favicon.ico" | mwscript purgeList.php --wiki=enwiki` T268230 |
[production] |
| 01:37 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2384.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:35 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2384.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:24 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2383.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:22 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2383.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:12 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2382.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:10 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2382.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:56 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:54 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:46 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2380.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:44 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2380.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:32 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2379.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:30 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2379.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:12 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 00:08 |
<legoktm> |
uploaded mailman3 3.2.1-1+wmf1, postorius 1.2.4-1+wmf1 to apt.wikimedia.org |
[production] |
| 00:07 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |