|
2021-11-04
ยง
|
| 20:01 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 19:29 |
<dduvall> |
1.38.0-wmf.7 on all wikis. no new errors or increase in error rates (refs T293948) |
[production] |
| 19:25 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:16 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.7 refs T293948 |
[production] |
| 18:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17703 and previous config saved to /var/cache/conftool/dbconfig/20211104-182655-root.json |
[production] |
| 18:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1153 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17701 and previous config saved to /var/cache/conftool/dbconfig/20211104-181151-root.json |
[production] |
| 18:11 |
<legoktm> |
upgrading to scap 4.0.3 on canaries again (T294966) |
[production] |
| 18:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:08 |
<legoktm> |
uploaded scap 4.0.3-2 to apt.wm.o for buster/stretch (T294966) |
[production] |
| 18:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:06 |
<jdrewniak@deploy1002> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 03s) |
[production] |
| 18:05 |
<jdrewniak@deploy1002> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:736795| Bumping portals to master (T128546)]] (duration: 01m 04s) |
[production] |
| 17:58 |
<Amir1> |
Upgrade db1153 T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:57 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1153.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1153 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17700 and previous config saved to /var/cache/conftool/dbconfig/20211104-175606-ladsgroup.json |
[production] |
| 17:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17699 and previous config saved to /var/cache/conftool/dbconfig/20211104-175429-root.json |
[production] |
| 17:50 |
<volans> |
restarted puppetdb.service on puppetdb2002 |
[production] |
| 17:50 |
<hashar> |
Updating docker-pkg files on contint for https://gerrit.wikimedia.org/r/c/integration/config/+/736630/ |
[releng] |
| 17:47 |
<ryankemper> |
T288620 [Elastic] Rebooting `elastic1049.eqiad.wmnet` to uptake new gelf settings change |
[production] |
| 17:46 |
<hnowlan> |
enabling puppet on C:cassandra after profile::java transition |
[production] |
| 17:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1152 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17698 and previous config saved to /var/cache/conftool/dbconfig/20211104-173926-root.json |
[production] |
| 17:33 |
<Amir1> |
Upgrade db1152 T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1152.eqiad.wmnet with reason: Maintenance T295026 |
[production] |
| 17:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1152 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17697 and previous config saved to /var/cache/conftool/dbconfig/20211104-172950-ladsgroup.json |
[production] |
| 17:29 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:24 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:23 |
<ryankemper> |
T294961 [WCQS] Installed kernel version `Linux 5.10.0-0.bpo.9-amd64` on all wcqs* hosts |
[production] |
| 16:48 |
<ryankemper> |
T294961 [WCQS] Power cycled all 6 wcqs* hosts via the mgmt console (`racadm serveraction powercycle`) |
[production] |
| 16:42 |
<majavah> |
provisioning cloud-cumin-03 as bullseye |
[cloudinfra] |
| 16:42 |
<mutante> |
scandium (parsoid::testing) - purging MW font packages |
[production] |
| 16:39 |
<razzi> |
add "can sql json on superset" permission to Alpha role on superset.wikimedia.org |
[analytics] |
| 16:39 |
<wm-bot> |
Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:39 |
<wm-bot> |
Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:37 |
<wm-bot> |
Finished rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance |
[admin] |
| 16:34 |
<wm-bot> |
Rebooting node cloudcephosd1023.eqiad.wmnet - cookbook ran by arturo@endurance |
[admin] |
| 16:33 |
<wm-bot> |
Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:33 |
<wm-bot> |
Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:17 |
<wm-bot> |
Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:17 |
<wm-bot> |
Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:16 |
<wm-bot> |
Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance |
[admin] |
| 16:14 |
<razzi> |
drop and restore superset_staging database to test permissions as they are in production |
[analytics] |
| 16:13 |
<wm-bot> |
Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance |
[admin] |
| 16:12 |
<wm-bot> |
Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:12 |
<wm-bot> |
Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance |
[admin] |
| 16:08 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 (duration: 16m 06s) |
[production] |
| 16:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db2143 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17696 and previous config saved to /var/cache/conftool/dbconfig/20211104-160047-root.json |
[production] |
| 16:00 |
<wm-bot> |
Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance |
[admin] |