|
2022-01-17
§
|
| 09:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1131 (T285149)', diff saved to https://phabricator.wikimedia.org/P18756 and previous config saved to /var/cache/conftool/dbconfig/20220117-095837-marostegui.json |
[production] |
| 09:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18755 and previous config saved to /var/cache/conftool/dbconfig/20220117-095830-marostegui.json |
[production] |
| 09:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18754 and previous config saved to /var/cache/conftool/dbconfig/20220117-094325-marostegui.json |
[production] |
| 09:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2144.codfw.wmnet with OS bullseye |
[production] |
| 09:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2143.codfw.wmnet with OS bullseye |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P18753 and previous config saved to /var/cache/conftool/dbconfig/20220117-092820-marostegui.json |
[production] |
| 09:23 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1017.eqiad.wmnet with OS bullseye |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18752 and previous config saved to /var/cache/conftool/dbconfig/20220117-091316-marostegui.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1113:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18751 and previous config saved to /var/cache/conftool/dbconfig/20220117-091308-marostegui.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18750 and previous config saved to /var/cache/conftool/dbconfig/20220117-091300-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18749 and previous config saved to /var/cache/conftool/dbconfig/20220117-085756-marostegui.json |
[production] |
| 08:53 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbproxy1017.eqiad.wmnet with OS bullseye |
[production] |
| 08:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18748 and previous config saved to /var/cache/conftool/dbconfig/20220117-084251-marostegui.json |
[production] |
| 08:36 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM schema1003.eqiad.wmnet |
[production] |
| 08:34 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM schema1003.eqiad.wmnet |
[production] |
| 08:32 |
<hashar> |
Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro |
[releng] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18747 and previous config saved to /var/cache/conftool/dbconfig/20220117-082746-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T285149)', diff saved to https://phabricator.wikimedia.org/P18746 and previous config saved to /var/cache/conftool/dbconfig/20220117-082638-marostegui.json |
[production] |
| 08:26 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:21 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM schema1004.eqiad.wmnet |
[production] |
| 08:17 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM schema1004.eqiad.wmnet |
[production] |
| 07:19 |
<elukey> |
launch webrequest bundle from 2022-01-16T01:00 (first hour missing for text) - 0003712-220113112502223-oozie-oozi-B |
[analytics] |
| 07:17 |
<elukey> |
kill webrequest bundle, text coordinator failed (logs/info/etc.. https://hue.wikimedia.org/hue/jobbrowser/#!id=0024621-210701181527401-oozie-oozi-B) |
[analytics] |
| 07:13 |
<elukey> |
umount/mount /mnt/hdfs on an-coord1001 to pick up java upgrades |
[analytics] |
| 06:59 |
<wm-bot> |
<legoktm> Updating uatu to v0.1.10 |
[tools.mjolnir] |
| 06:59 |
<elukey> |
`systemctl reset-failed ifup@ens5.service` on an-test-client1001 and kafka-test1010 |
[production] |
| 06:27 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1016.eqiad.wmnet with OS bullseye |
[production] |
| 05:57 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host dbproxy1016.eqiad.wmnet with OS bullseye |
[production] |
|
2022-01-16
§
|
| 19:53 |
<wm-bot> |
<lucaswerkmeister> deployed 504c5481e9 (update Spanish verbs) |
[tools.lexeme-forms] |
| 18:19 |
<wm-bot> |
<lucaswerkmeister> deployed 68234bd17d (Odia adjectives and adverbs) |
[tools.lexeme-forms] |
| 16:43 |
<elukey> |
`elukey@an-launcher1002:~$ sudo systemctl reset-failed eventlogging_to_druid_network_internal_flows-sanitization_daily.service eventlogging_to_druid_network_internal_flows_daily.service eventlogging_to_druid_network_internal_flows_hourly.service |
[analytics] |
| 08:21 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync on production |
[production] |
| 08:20 |
<oblivian@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply on staging |
[production] |
| 08:20 |
<oblivian@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply on production |
[production] |
| 08:18 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: sync on production |
[production] |
| 08:17 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply on staging |
[production] |
| 08:17 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply on production |
[production] |
| 07:19 |
<legoktm> |
deleting extdist-05 instance, no longer needed |
[extdist] |
| 07:18 |
<legoktm> |
switching extdist.wmflabs.org domain to point to extdist-06 instance (bullseye/cinder) - T293055 |
[extdist] |