751-800 of 10000 results (42ms)
2021-03-30 §
16:07 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,cluster=jobrunner,name=mw12.* [production]
16:06 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,cluster=videoscaler,name=mw13.* [production]
16:06 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: dc=eqiad,cluster=videoscaler,name=mw12.* [production]
15:59 <akosiaris> depool a number of hosts from videoscalers [production]
15:59 <akosiaris@cumin1001> conftool action : set/pooled=no; selector: dc=eqiad,cluster=videoscaler,name=mw12.* [production]
15:55 <legoktm@deploy1002> conftool action : set/pooled=no; selector: name=mw1308.eqiad.wmnet,service=jobrunner [production]
15:55 <legoktm@deploy1002> conftool action : set/pooled=no; selector: name=mw1307.eqiad.wmnet,service=jobrunner [production]
15:42 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet [production]
15:29 <hnowlan> moving all test tables out of cassandra directories on aqs hosts [production]
14:59 <effie> disable puppet on mediawiki servers to deploy 663565 [production]
14:58 <Urbanecm> Move Help talk:Help talk:Getting started --> Help talk:Getting started via moveBatch.php on enwiki (T278350) [production]
14:32 <arturo> manually start update-openstack-mirror.service on sodium (T278505) [production]
13:02 <jbond42> rollout lxml update T278822 [production]
12:55 <jbond42> update spamassasin on lists,otrs and mx T278820 [production]
12:39 <Amir1> ssh -p 29418 gerrit.wikimedia.org replication start wikidata/query-builder --wait (T277060) [production]
12:38 <jbond42> update python(3)-pygments [production]
12:36 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet [production]
12:14 <Urbanecm> mwmaint1002: Downloading multiple big files (total filesize estimated 150 GB, downloaded and processed in batches) for server-side uploads [production]
11:21 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:675751|Disable legacy javascript global variables in group1]], Some increase in client errors is expected (T72470) (duration: 01m 11s) [production]
09:58 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet [production]
09:52 <aborrero@cumin1001> START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet [production]
09:42 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
09:41 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
09:35 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
09:35 <hnowlan@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
09:05 <hnowlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
09:04 <hnowlan@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
08:36 <jynus> mariadb upgrade of all buster source backup hosts to 10.4.18 T250666 [production]
08:05 <dcausse> refreshing wdqs entities (T278693) [production]
07:37 <elukey> restart-php7.2-fpm on mw1304, jobrunner completely overwhelmed by ffmpeg/transcode jobs (not publishing metrics, erroring out for memcached timeouts) - T278734 [production]
07:28 <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.36 - T274940 [production]
06:06 <elukey> powercycle cp1087 (no ssh, no mgmt console tty) [production]
06:04 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp1087.eqiad.wmnet [production]
2021-03-29 §
19:06 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet [production]
17:47 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:37 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:15 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=aqs1004.eqiad.wmnet [production]
16:11 <hnowlan> depooled aqs1004 for transfer of large tables to aqs1010 [production]
15:53 <jbond@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:47 <jbond@cumin1001> START - Cookbook sre.dns.netbox [production]
15:45 <jbond@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:39 <jbond@cumin1001> START - Cookbook sre.dns.netbox [production]
13:26 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on parse2001.codfw.wmnet with reason: REIMAGE [production]
13:24 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on parse2001.codfw.wmnet with reason: REIMAGE [production]
13:03 <ema> cp4027: rollback luajit experiment https://github.com/apache/trafficserver/issues/7423#issuecomment-809354214 [production]
12:36 <ema> cp4027: re-enable JIT compilation in all ats-be lua scripts -- https://github.com/apache/trafficserver/issues/7423 [production]
11:57 <ema> cp4027: re-enable JIT compilation in normalize-path.lua -- https://github.com/apache/trafficserver/issues/7423 [production]
11:32 <ema> cp4027: install libluajit 2.1.0~beta3+dfsg-6wm1 with P15083 applied -- https://github.com/apache/trafficserver/issues/7423 [production]
09:59 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pki2001.codfw.wmnet with reason: REIMAGE [production]
09:57 <jbond@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on pki2001.codfw.wmnet with reason: REIMAGE [production]