|
2021-11-04
ยง
|
| 20:01 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 19:29 |
<dduvall> |
1.38.0-wmf.7 on all wikis. no new errors or increase in error rates (refs T293948) |
[production] |
| 19:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:16 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 18:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17703 and previous config saved to /var/cache/conftool/dbconfig/20211104-182655-root.json |
[production] |
| 18:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17701 and previous config saved to /var/cache/conftool/dbconfig/20211104-181151-root.json |
[production] |
| 18:11 |
<legoktm> |
upgrading to scap 4.0.3 on canaries again (T294966) |
[production] |
| 18:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:08 |
<legoktm> |
uploaded scap 4.0.3-2 to apt.wm.o for buster/stretch (T294966) |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:06 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 03s) |
[production] |
| 18:05 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 04s) |
[production] |
| 17:58 |
<Amir1> |
Upgrade db1153 T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1153 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17700 and previous config saved to /var/cache/conftool/dbconfig/20211104-175606-ladsgroup.json |
[production] |
| 17:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17699 and previous config saved to /var/cache/conftool/dbconfig/20211104-175429-root.json |
[production] |
| 17:50 |
<volans> |
restarted puppetdb.service on puppetdb2002 |
[production] |
| 17:47 |
<ryankemper> |
T288620 [Elastic] Rebooting `elastic1049.eqiad.wmnet` to uptake new gelf settings change |
[production] |
| 17:46 |
<hnowlan> |
enabling puppet on C:cassandra after profile::java transition |
[production] |
| 17:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17698 and previous config saved to /var/cache/conftool/dbconfig/20211104-173926-root.json |
[production] |
| 17:33 |
<Amir1> |
Upgrade db1152 T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1152 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17697 and previous config saved to /var/cache/conftool/dbconfig/20211104-172950-ladsgroup.json |
[production] |
| 17:29 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:24 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:23 |
<ryankemper> |
T294961 [WCQS] Installed kernel version `Linux 5.10.0-0.bpo.9-amd64` on all wcqs* hosts |
[production] |
| 16:48 |
<ryankemper> |
T294961 [WCQS] Power cycled all 6 wcqs* hosts via the mgmt console (`racadm serveraction powercycle`) |
[production] |
| 16:42 |
<mutante> |
scandium (parsoid::testing) - purging MW font packages |
[production] |
| 16:08 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 (duration: 16m 06s) |
[production] |
| 16:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2143 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17696 and previous config saved to /var/cache/conftool/dbconfig/20211104-160047-root.json |
[production] |
| 15:52 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 |
[production] |
| 15:50 |
<jbond> |
disable puppet fleet wide to deploy a puppet change |
[production] |
| 15:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2143 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17695 and previous config saved to /var/cache/conftool/dbconfig/20211104-154543-root.json |
[production] |
| 15:37 |
<Amir1> |
Upgrade db2143 T295026 |
[production] |
| 15:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 |
[production] |
| 15:31 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 |
[production] |
| 15:30 |
<XioNoX> |
drain codfw-ulsfo link |
[production] |
| 15:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2143 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17694 and previous config saved to /var/cache/conftool/dbconfig/20211104-152919-ladsgroup.json |
[production] |
| 15:26 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet |
[production] |
| 15:26 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet |
[production] |
| 15:11 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet |
[production] |
| 15:05 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet |
[production] |
| 15:04 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 15:03 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 14:50 |
<XioNoX> |
disable cr1-codfw:et-0/0/0 |
[production] |
| 14:49 |
<hashar> |
Upgrading CI Jenkins |
[production] |
| 14:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet |
[production] |