|
2021-08-03
ยง
|
| 19:45 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 19:43 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti-test2002.codfw.wmnet with reason: REIMAGE |
[production] |
| 19:42 |
<otto@deploy1002> |
Started deploy [analytics/refinery@ea78871]: Regular analytics weekly train [analytics/refinery@ea78871] |
[production] |
| 19:36 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:31 |
<ryankemper> |
T285355 `ryankemper@an-web1001:~$ sudo run-puppet-agent` to establish `role(analytics_cluster::webserver)` on the host in preparation for upcoming cutover from `thorium`->`an-web1001` |
[production] |
| 19:31 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@aceb561] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@aceb561] (duration: 05m 40s) |
[production] |
| 19:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:27 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: revert group0 wikis to 1.37.0-wmf.16 |
[production] |
| 19:25 |
<otto@deploy1002> |
Started deploy [analytics/refinery@aceb561] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@aceb561] |
[production] |
| 19:14 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.17 |
[production] |
| 19:01 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@aceb561] (thin): Regular analytics weekly train THIN [analytics/refinery@aceb561] (duration: 00m 07s) |
[production] |
| 19:01 |
<otto@deploy1002> |
Started deploy [analytics/refinery@aceb561] (thin): Regular analytics weekly train THIN [analytics/refinery@aceb561] |
[production] |
| 19:00 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@aceb561]: Regular analytics weekly train [analytics/refinery@aceb561] (duration: 16m 25s) |
[production] |
| 18:47 |
<Amir1> |
running mwscript migrateUserGroup.php --wiki=idwiki editor reviewer (T286853) |
[production] |
| 18:44 |
<otto@deploy1002> |
Started deploy [analytics/refinery@aceb561]: Regular analytics weekly train [analytics/refinery@aceb561] |
[production] |
| 18:29 |
<dduvall@deploy1002> |
Finished scap: testwikis wikis to 1.37.0-wmf.17 (duration: 36m 44s) |
[production] |
| 18:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:06 |
<ebernhardson@deploy1002> |
Finished deploy [search/mjolnir/deploy@f0f70d1]: T286642 fixes to bulk daemon prioritization (duration: 00m 48s) |
[production] |
| 18:05 |
<ebernhardson@deploy1002> |
Started deploy [search/mjolnir/deploy@f0f70d1]: T286642 fixes to bulk daemon prioritization |
[production] |
| 17:52 |
<dduvall@deploy1002> |
Started scap: testwikis wikis to 1.37.0-wmf.17 |
[production] |
| 16:59 |
<hashar> |
Gerrit has been upgraded |
[production] |
| 16:47 |
<dancy@deploy1002> |
Finished deploy [gerrit/gerrit@244120b]: Gerrit to 3.3.5 on gerrit1001 (duration: 00m 07s) |
[production] |
| 16:47 |
<dancy@deploy1002> |
Started deploy [gerrit/gerrit@244120b]: Gerrit to 3.3.5 on gerrit1001 |
[production] |
| 16:45 |
<urbanecm> |
Start server side upload for 1 video file (T287957) |
[production] |
| 16:45 |
<hashar> |
Stopping Gerrit for upgrade |
[production] |
| 16:43 |
<volans> |
upgraded spicerack to 0.0.57-1+deb10u1 on cumin1001 |
[production] |
| 16:36 |
<dancy@deploy1002> |
Finished deploy [gerrit/gerrit@244120b]: Gerrit to 3.3.5 on gerrit2001 (duration: 00m 10s) |
[production] |
| 16:36 |
<dancy@deploy1002> |
Started deploy [gerrit/gerrit@244120b]: Gerrit to 3.3.5 on gerrit2001 |
[production] |
| 16:27 |
<hashar> |
Going to upgrade Gerrit 3.3 (scheduled maintenance) |
[production] |
| 16:18 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:14 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:00 |
<dcausse@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' . |
[production] |
| 15:55 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:50 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:26 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:25 |
<moritzm> |
prune testvm2001 from Ganeti and clean up from Netbox (stuck in some inconsistent state the decom cookbook can't handle) T286206 |
[production] |
| 15:14 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2002.codfw.wmnet |
[production] |
| 15:01 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.makevm for new host testvm2002.codfw.wmnet |
[production] |
| 14:56 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2001.codfw.wmnet |
[production] |
| 14:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet |
[production] |
| 14:32 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:27 |
<ottomata> |
chown dumpsgen and chmod 644 /data/xmldatadumps/public/*/20210801/dumpstatus.json on labstore1006 and labstore1007 (it was only readable by root causing an analytics import job to fail), ping apergos |
[production] |
| 14:23 |
<ottomata> |
chown dumpsgen and chmod 644 /data/xmldatadumps/public/lezwiki/20210801/dumpstatus.json on labstore1006 and labstore1007 (it was only readable by root causing an analytics import job to fail), ping apergos |
[production] |
| 14:13 |
<ottomata> |
chown dumpsgen and chmod 644 dumpsdata1003:/data/xmldatadumps/public/lezwiki/20210801/dumpstatus.json (it was only readable by root causing an analytics import job to fail), ping apergos |
[production] |
| 12:47 |
<moritzm> |
restarting Tomcat on idp1001 |
[production] |
| 12:05 |
<moritzm> |
installing libgcrypt20 security updates |
[production] |