|
2021-03-30
§
|
| 16:40 |
<effie> |
enable puppet on mw* hosts |
[production] |
| 16:10 |
<mutante> |
mw1296 - started ferm |
[production] |
| 16:10 |
<mutante> |
mw1308 - started ferm |
[production] |
| 16:07 |
<akosiaris> |
split jobrunners/videoscalers clusters in conftool. mw12* become videoscalers, mw13* become jobrunners, killing ffmpeg on mw13* |
[production] |
| 16:07 |
<mutante> |
mw1309 - systemctl start ferm |
[production] |
| 16:07 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=jobrunner,name=mw12.* |
[production] |
| 16:06 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=videoscaler,name=mw13.* |
[production] |
| 16:06 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=videoscaler,name=mw12.* |
[production] |
| 15:59 |
<akosiaris> |
depool a number of hosts from videoscalers |
[production] |
| 15:59 |
<akosiaris@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=videoscaler,name=mw12.* |
[production] |
| 15:55 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: name=mw1308.eqiad.wmnet,service=jobrunner |
[production] |
| 15:55 |
<legoktm@deploy1002> |
conftool action : set/pooled=no; selector: name=mw1307.eqiad.wmnet,service=jobrunner |
[production] |
| 15:42 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet |
[production] |
| 15:29 |
<hnowlan> |
moving all test tables out of cassandra directories on aqs hosts |
[production] |
| 14:59 |
<effie> |
disable puppet on mediawiki servers to deploy 663565 |
[production] |
| 14:58 |
<Urbanecm> |
Move Help talk:Help talk:Getting started --> Help talk:Getting started via moveBatch.php on enwiki (T278350) |
[production] |
| 14:32 |
<arturo> |
manually start update-openstack-mirror.service on sodium (T278505) |
[production] |
| 13:02 |
<jbond42> |
rollout lxml update T278822 |
[production] |
| 12:55 |
<jbond42> |
update spamassasin on lists,otrs and mx T278820 |
[production] |
| 12:39 |
<Amir1> |
ssh -p 29418 gerrit.wikimedia.org replication start wikidata/query-builder --wait (T277060) |
[production] |
| 12:38 |
<jbond42> |
update python(3)-pygments |
[production] |
| 12:36 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet |
[production] |
| 12:14 |
<Urbanecm> |
mwmaint1002: Downloading multiple big files (total filesize estimated 150 GB, downloaded and processed in batches) for server-side uploads |
[production] |
| 11:21 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:675751|Disable legacy javascript global variables in group1]], Some increase in client errors is expected (T72470) (duration: 01m 11s) |
[production] |
| 09:58 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet |
[production] |
| 09:52 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet |
[production] |
| 09:42 |
<hnowlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:41 |
<hnowlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:35 |
<hnowlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:35 |
<hnowlan@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:05 |
<hnowlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 09:04 |
<hnowlan@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 08:36 |
<jynus> |
mariadb upgrade of all buster source backup hosts to 10.4.18 T250666 |
[production] |
| 08:05 |
<dcausse> |
refreshing wdqs entities (T278693) |
[production] |
| 07:37 |
<elukey> |
restart-php7.2-fpm on mw1304, jobrunner completely overwhelmed by ffmpeg/transcode jobs (not publishing metrics, erroring out for memcached timeouts) - T278734 |
[production] |
| 07:28 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.36 - T274940 |
[production] |
| 06:06 |
<elukey> |
powercycle cp1087 (no ssh, no mgmt console tty) |
[production] |
| 06:04 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet |
[production] |
|
2021-03-29
§
|
| 19:06 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet |
[production] |
| 17:47 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:37 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:15 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet |
[production] |
| 16:11 |
<hnowlan> |
depooled aqs1004 for transfer of large tables to aqs1010 |
[production] |
| 15:53 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:47 |
<jbond@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:45 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:39 |
<jbond@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:26 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:24 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on parse2001.codfw.wmnet with reason: REIMAGE |
[production] |
| 13:03 |
<ema> |
cp4027: rollback luajit experiment https://github.com/apache/trafficserver/issues/7423#issuecomment-809354214 |
[production] |