|
2023-01-04
ยง
|
| 14:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on puppetdb2002.codfw.wmnet with reason: maintenance |
[production] |
| 14:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P42780 and previous config saved to /var/cache/conftool/dbconfig/20230104-140936-marostegui.json |
[production] |
| 14:08 |
<urbanecm@deploy1002> |
urbanecm and stang: Backport for [[gerrit:870920|Revert "trwiki: Add 20 years celebration logos" (T325823)]], [[gerrit:870988|kuwiki: Install SandboxLink (T325469)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 14:06 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:870920|Revert "trwiki: Add 20 years celebration logos" (T325823)]], [[gerrit:870988|kuwiki: Install SandboxLink (T325469)]] |
[production] |
| 14:04 |
<XioNoX> |
fix inconsistent mtu on mr1-ulsfo - T315838 |
[production] |
| 14:02 |
<marostegui> |
dbmaint deploy schema change on s3 T326221 |
[production] |
| 14:02 |
<moritzm> |
updating buster nodes running 5.10 to 5.10.158-2~deb10u1 (only rollout of the new kernel, no reboots) |
[production] |
| 14:02 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:874844|Update interwiki cache]] (duration: 08m 00s) |
[production] |
| 13:58 |
<marostegui> |
dbmaint deploy schema change on s7 T326221 |
[production] |
| 13:57 |
<marostegui> |
dbmaint deploy schema change on s8 T326221 |
[production] |
| 13:57 |
<marostegui> |
dbmaint deploy schema change on s6 T326221 |
[production] |
| 13:56 |
<marostegui> |
dbmaint deploy schema change on s5 T326221 |
[production] |
| 13:55 |
<marostegui> |
dbmaint deploy schema change on s4 T326221 |
[production] |
| 13:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2149 (T326011)', diff saved to https://phabricator.wikimedia.org/P42779 and previous config saved to /var/cache/conftool/dbconfig/20230104-135429-marostegui.json |
[production] |
| 13:54 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:874844|Update interwiki cache]] |
[production] |
| 13:54 |
<marostegui> |
dbmaint deploy schema change on s2 T326221 |
[production] |
| 13:53 |
<marostegui> |
dbmaint deploy schema change on s1 T326221 |
[production] |
| 13:52 |
<urbanecm@deploy1002> |
Finished scap: Creating gorwiktionary (T326137), fixing aswikiquote logo (T321246) (duration: 07m 52s) |
[production] |
| 13:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2149 (T326011)', diff saved to https://phabricator.wikimedia.org/P42778 and previous config saved to /var/cache/conftool/dbconfig/20230104-134544-marostegui.json |
[production] |
| 13:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 13:45 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 13:44 |
<XioNoX> |
repool esams-eqiad link for mtu change - T315838 |
[production] |
| 13:44 |
<urbanecm@deploy1002> |
Started scap: Creating gorwiktionary (T326137), fixing aswikiquote logo (T321246) |
[production] |
| 13:41 |
<XioNoX> |
drain esams-eqiad link for mtu change - T315838 |
[production] |
| 13:39 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:874883|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]], [[gerrit:874882|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]] (duration: 38m 23s) |
[production] |
| 13:38 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 13:38 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 13:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127 (T326011)', diff saved to https://phabricator.wikimedia.org/P42777 and previous config saved to /var/cache/conftool/dbconfig/20230104-133830-marostegui.json |
[production] |
| 13:33 |
<XioNoX> |
fix missmatch MTU on pfw3-codfw - T315838 |
[production] |
| 13:31 |
<urbanecm> |
New wiki creation will run over by a couple of minutes |
[production] |
| 13:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P42776 and previous config saved to /var/cache/conftool/dbconfig/20230104-132323-marostegui.json |
[production] |
| 13:15 |
<XioNoX> |
fix missmatch MTU on cloudsw switches - T315838 |
[production] |
| 13:11 |
<btullis@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1001" |
[production] |
| 13:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P42775 and previous config saved to /var/cache/conftool/dbconfig/20230104-130816-marostegui.json |
[production] |
| 13:00 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:874883|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]], [[gerrit:874882|Add messages for Gorontalo Wiktionary (gorwiktionary) (T326137)]] |
[production] |
| 12:58 |
<urbanecm@deploy1002> |
Finished scap: Creating shnwikibooks (T321248) (duration: 07m 38s) |
[production] |
| 12:55 |
<moritzm> |
installing emacs security updates |
[production] |
| 12:54 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 100%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42774 and previous config saved to /var/cache/conftool/dbconfig/20230104-125330-root.json |
[production] |
| 12:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2127 (T326011)', diff saved to https://phabricator.wikimedia.org/P42773 and previous config saved to /var/cache/conftool/dbconfig/20230104-125310-marostegui.json |
[production] |
| 12:51 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage |
[production] |
| 12:50 |
<urbanecm@deploy1002> |
Started scap: Creating shnwikibooks (T321248) |
[production] |
| 12:48 |
<urbanecm@deploy1002> |
Finished scap: Creating guwwikiquote (T321247) (duration: 07m 44s) |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2127 (T326011)', diff saved to https://phabricator.wikimedia.org/P42772 and previous config saved to /var/cache/conftool/dbconfig/20230104-124424-marostegui.json |
[production] |
| 12:44 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 12:44 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2127.codfw.wmnet with reason: Maintenance |
[production] |
| 12:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42771 and previous config saved to /var/cache/conftool/dbconfig/20230104-124403-marostegui.json |
[production] |
| 12:41 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 12:41 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 12:41 |
<urbanecm@deploy1002> |
Started scap: Creating guwwikiquote (T321247) |
[production] |