|
2021-11-04
ยง
|
| 23:34 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:22 |
<cjming@deploy1002> |
Synchronized php-1.38.0-wmf.7/extensions/RelatedArticles: Backport: [[gerrit:736830|Fix loading of related articles via IntersectionObserver (T223844)]] (duration: 00m 55s) |
[production] |
| 23:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:19 |
<mutante> |
wtp1025, wtp1026, parse2001, parse2002 (parsoid-canary): purging mediawiki font packages (T294378) |
[production] |
| 23:16 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:736522|Allow bureaucrats to grant and revoke the importer rights to enwikiversity (T294930)]] (duration: 00m 56s) |
[production] |
| 23:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 21:26 |
<bblack> |
cpNNNN: manual (cumin) removal of outdated digicert-2020 ocsp configuration and output files, to avoid icinga alerts and clean up |
[production] |
| 20:01 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 20:01 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 19:29 |
<dduvall> |
1.38.0-wmf.7 on all wikis. no new errors or increase in error rates (refs T293948) |
[production] |
| 19:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:16 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 18:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17703 and previous config saved to /var/cache/conftool/dbconfig/20211104-182655-root.json |
[production] |
| 18:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17701 and previous config saved to /var/cache/conftool/dbconfig/20211104-181151-root.json |
[production] |
| 18:11 |
<legoktm> |
upgrading to scap 4.0.3 on canaries again (T294966) |
[production] |
| 18:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:08 |
<legoktm> |
uploaded scap 4.0.3-2 to apt.wm.o for buster/stretch (T294966) |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:06 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 03s) |
[production] |
| 18:05 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 04s) |
[production] |
| 17:58 |
<Amir1> |
Upgrade db1153 T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1153 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17700 and previous config saved to /var/cache/conftool/dbconfig/20211104-175606-ladsgroup.json |
[production] |
| 17:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17699 and previous config saved to /var/cache/conftool/dbconfig/20211104-175429-root.json |
[production] |
| 17:50 |
<volans> |
restarted puppetdb.service on puppetdb2002 |
[production] |
| 17:47 |
<ryankemper> |
T288620 [Elastic] Rebooting `elastic1049.eqiad.wmnet` to uptake new gelf settings change |
[production] |
| 17:46 |
<hnowlan> |
enabling puppet on C:cassandra after profile::java transition |
[production] |
| 17:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17698 and previous config saved to /var/cache/conftool/dbconfig/20211104-173926-root.json |
[production] |
| 17:33 |
<Amir1> |
Upgrade db1152 T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1152 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17697 and previous config saved to /var/cache/conftool/dbconfig/20211104-172950-ladsgroup.json |
[production] |
| 17:29 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:24 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:23 |
<ryankemper> |
T294961 [WCQS] Installed kernel version `Linux 5.10.0-0.bpo.9-amd64` on all wcqs* hosts |
[production] |
| 16:48 |
<ryankemper> |
T294961 [WCQS] Power cycled all 6 wcqs* hosts via the mgmt console (`racadm serveraction powercycle`) |
[production] |
| 16:42 |
<mutante> |
scandium (parsoid::testing) - purging MW font packages |
[production] |
| 16:08 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 (duration: 16m 06s) |
[production] |
| 16:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2143 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17696 and previous config saved to /var/cache/conftool/dbconfig/20211104-160047-root.json |
[production] |
| 15:52 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 |
[production] |
| 15:50 |
<jbond> |
disable puppet fleet wide to deploy a puppet change |
[production] |
| 15:45 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2143 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17695 and previous config saved to /var/cache/conftool/dbconfig/20211104-154543-root.json |
[production] |
| 15:37 |
<Amir1> |
Upgrade db2143 T295026 |
[production] |
| 15:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 |
[production] |
| 15:31 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 |
[production] |
| 15:30 |
<XioNoX> |
drain codfw-ulsfo link |
[production] |