|
2022-09-14
ยง
|
| 13:20 |
<kartik@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:832145|Enable Content/Section translation on WPs with new MT support from Google (T313296)]] (duration: 03m 39s) |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:13 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:10 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:10 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:09 |
<kartik@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:831872|Enable Section Translation in Odia Wikipedia (T313300)]] (duration: 03m 55s) |
[production] |
| 13:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 12:56 |
<aqu> |
~1hago sudo systemctl start refinery-sqoop-mediawiki-production-daily.service ; sudo systemctl start refinery-import-siteinfo-dumps.service ; sudo systemctl start refinery-import-page-current-dumps.service ; sudo systemctl start refinery-import-page-history-dumps.service |
[analytics] |
| 12:54 |
<jayme> |
imported rsyslog 8.2208.0-1~bpo11+1 into bullseye-wikimedia component/rsyslog-k8s - T289766 |
[production] |
| 11:59 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:59 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1189.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T314041)', diff saved to https://phabricator.wikimedia.org/P34725 and previous config saved to /var/cache/conftool/dbconfig/20220914-115920-ladsgroup.json |
[production] |
| 11:49 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqdfw,cr2-eqdfw IPv6 |
[production] |
| 11:49 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.remove-downtime for cr2-eqdfw,cr2-eqdfw IPv6 |
[production] |
| 11:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P34723 and previous config saved to /var/cache/conftool/dbconfig/20220914-114413-ladsgroup.json |
[production] |
| 11:34 |
<btullis> |
remounted all remaining /mnt/hdfs mount points, except stat1005 which is busy |
[analytics] |
| 11:29 |
<topranks> |
rebooting cr2-eqdfw to complete upgrade |
[production] |
| 11:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P34721 and previous config saved to /var/cache/conftool/dbconfig/20220914-112907-ladsgroup.json |
[production] |
| 11:14 |
<topranks> |
Shutting down internet transit and peering on cr2-eqdfw in advance of upgrade reboot |
[production] |
| 11:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1179 (T314041)', diff saved to https://phabricator.wikimedia.org/P34719 and previous config saved to /var/cache/conftool/dbconfig/20220914-111400-ladsgroup.json |
[production] |
| 11:12 |
<btullis> |
remounted /mnt/hdfs on an-coord100[1-2] |
[analytics] |
| 11:09 |
<btullis> |
remounted /mnt/hdfs on an-airflow1001 |
[analytics] |
| 11:02 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: router upgrade |
[production] |
| 11:02 |
<cmooney@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: router upgrade |
[production] |
| 11:01 |
<wm-bot2> |
Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus |
[admin] |
| 11:01 |
<topranks> |
Prepping to upgrade JunOS on cr2-eqdfw. Adjusting OSPF costs to force traffic via alternate POPs. |
[production] |
| 10:58 |
<wm-bot2> |
Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:57 |
<wm-bot2> |
Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:57 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34717 and previous config saved to /var/cache/conftool/dbconfig/20220914-103810-root.json |
[production] |
| 10:27 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:26 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 10:25 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 10:24 |
<kharlan@deploy1002> |
Synchronized php-1.40.0-wmf.1/extensions/WikimediaEvents/includes/BlockMetrics/BlockMetricsHooks.php: Backport: [[gerrit:831969|BlockMetrics: Update to new event schema version (T306018)]] (duration: 03m 48s) |
[production] |
| 10:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34715 and previous config saved to /var/cache/conftool/dbconfig/20220914-102305-root.json |
[production] |
| 10:18 |
<moritzm> |
import routinator 0.11.3-1bullseye to thirdparty/routinator |
[production] |
| 10:09 |
<wm-bot2> |
Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:09 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Repooling for warm up after upgrade', diff saved to https://phabricator.wikimedia.org/P34714 and previous config saved to /var/cache/conftool/dbconfig/20220914-100800-root.json |
[production] |
| 10:06 |
<wm-bot2> |
Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:06 |
<wm-bot2> |
Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus |
[admin] |
| 10:00 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=no; selector: cluster=wikireplicas-a,name=dbproxy1018.eqiad.wmnet |
[production] |
| 09:59 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=yes; selector: cluster=wikireplicas-a,name=dbproxy1019.eqiad.wmnet |
[production] |
| 09:58 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=inactive; selector: cluster=wikireplicas-b,name=dbproxy1018.eqiad.wmnet |
[production] |
| 09:57 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=no; selector: cluster=wikireplicas-b,name=dbproxy1018.eqiad.wmnet |
[production] |
| 09:57 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=yes; selector: cluster=wikireplicas-b,name=dbproxy1019.eqiad.wmnet |
[production] |
| 09:53 |
<ladsgroup@cumin1001> |
conftool action : set/pooled=no; selector: cluster=wikireplicas-b,name=dbproxy1019.eqiad.wmnet |
[production] |