|
2021-01-25
§
|
| 11:39 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:658242| Bumping portals to master (T128546)]] (duration: 00m 58s) |
[production] |
| 11:35 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh-clients (exit_code=0) for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001 |
[production] |
| 11:33 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro-from-cdh-clients for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001 |
[production] |
| 11:11 |
<godog> |
thanos delete old orphaned blocks with replica=unset label |
[production] |
| 10:58 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host scandium.eqiad.wmnet |
[production] |
| 10:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host scandium.eqiad.wmnet |
[production] |
| 10:44 |
<godog> |
swift decrease weight for ms-be20[16,18,20,22] - T272837 |
[production] |
| 10:00 |
<moritzm> |
installing imagemagick security updates on stretch |
[production] |
| 09:55 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1002.wikimedia.org |
[production] |
| 09:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host urldownloader1002.wikimedia.org |
[production] |
| 09:49 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2002.wikimedia.org |
[production] |
| 09:48 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host urldownloader2002.wikimedia.org |
[production] |
| 09:40 |
<godog> |
bounce apache2 on logstash1024, stuck on high cpu |
[production] |
| 09:21 |
<marostegui@deploy1001> |
Synchronized wmf-config/etcd.php: Add x2 to the mapping array T269324 (duration: 00m 58s) |
[production] |
| 09:17 |
<moritzm> |
installing samba security updates on stretch |
[production] |
| 09:15 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Add x2 to the mapping array T269324 (duration: 01m 01s) |
[production] |
| 09:06 |
<ema> |
cp3054: install varnish 6.0.1-1wm2 -- 6.0.1 without https://github.com/varnishcache/varnish-cache/pull/2705 T264398 |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1142 (re)pooling @ 100%: After upgrading its kernel', diff saved to https://phabricator.wikimedia.org/P13944 and previous config saved to /var/cache/conftool/dbconfig/20210125-084715-root.json |
[production] |
| 08:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1142 (re)pooling @ 75%: After upgrading its kernel', diff saved to https://phabricator.wikimedia.org/P13943 and previous config saved to /var/cache/conftool/dbconfig/20210125-083211-root.json |
[production] |
| 08:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1142 (re)pooling @ 50%: After upgrading its kernel', diff saved to https://phabricator.wikimedia.org/P13942 and previous config saved to /var/cache/conftool/dbconfig/20210125-081708-root.json |
[production] |
| 08:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1142 (re)pooling @ 25%: After upgrading its kernel', diff saved to https://phabricator.wikimedia.org/P13941 and previous config saved to /var/cache/conftool/dbconfig/20210125-080204-root.json |
[production] |
| 07:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1142 for kernel upgrade', diff saved to https://phabricator.wikimedia.org/P13940 and previous config saved to /var/cache/conftool/dbconfig/20210125-073322-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add x2 eqiad to dbctl T269324', diff saved to https://phabricator.wikimedia.org/P13939 and previous config saved to /var/cache/conftool/dbconfig/20210125-064419-marostegui.json |
[production] |
| 06:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Populate x2 eqiad hosts into dbctl T269324', diff saved to https://phabricator.wikimedia.org/P13938 and previous config saved to /var/cache/conftool/dbconfig/20210125-064305-marostegui.json |
[production] |
|
2021-01-23
§
|
| 22:21 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) (duration: 00m 06s) |
[production] |
| 22:21 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) |
[production] |
| 04:05 |
<ryankemper> |
Depooled `wdqs1013` (it has ~50 mins of lag to catch up on, and also the bad gateway above) |
[production] |
| 04:03 |
<ryankemper> |
Restarted `wdqs-blazegraph` on `wdqs1013`: `sudo systemctl restart wdqs-blazegraph` |
[production] |
| 01:52 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2332.codfw.wmnet |
[production] |
| 01:52 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2328.codfw.wmnet |
[production] |
| 01:51 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2332.codfw.wmnet |
[production] |
| 01:51 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2328.codfw.wmnet |
[production] |
| 01:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2330.codfw.wmnet |
[production] |
| 01:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2334.codfw.wmnet |
[production] |
| 01:48 |
<foks> |
reset user email for Davey2010 |
[production] |
| 01:44 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1413.eqiad.wmnet |
[production] |
| 01:43 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 01:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2330.codfw.wmnet |
[production] |
| 01:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2334.codfw.wmnet |
[production] |
| 01:40 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet |
[production] |
| 01:39 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1413.eqiad.wmnet |
[production] |
| 00:46 |
<legoktm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Switch enwiki to use enwiki20 "Option A" logo variant (T272526) (duration: 00m 57s) |
[production] |
| 00:36 |
<legoktm@deploy1001> |
Synchronized static/images/project-logos/: Add enwiki20 "Option A" fixed logos (T272526) (duration: 00m 59s) |
[production] |
|
2021-01-22
§
|
| 22:41 |
<reedy@deploy1001> |
Synchronized invalid.json: (no justification provided) (duration: 00m 58s) |
[production] |
| 20:07 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1268.eqiad.wmnet with reason: REIMAGE |
[production] |
| 20:05 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2330.codfw.wmnet with reason: REIMAGE |
[production] |
| 20:05 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2332.codfw.wmnet with reason: REIMAGE |
[production] |
| 20:03 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2328.codfw.wmnet with reason: REIMAGE |
[production] |
| 20:01 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1413.eqiad.wmnet with reason: REIMAGE |
[production] |