|
2020-10-16
§
|
| 19:00 |
<DannyS712> |
dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=enwiki suppress | T265755 |
[releng] |
| 18:59 |
<DannyS712> |
dannys712@deployment-deploy01:~$ mwscript maintenance/emptyUserGroup.php --wiki=enwiki autoreview | T265755 |
[releng] |
| 18:54 |
<bstorm> |
restarting webservice to try and clear a mysterious 404 |
[tools.sge-status] |
| 18:45 |
<James_F> |
Zuul: Switch various FR things from php70 to mwgate-composer-php73-docker |
[releng] |
| 18:27 |
<James_F> |
Zuul: [cloud/toolforge/buildpacks] Add CI |
[releng] |
| 18:26 |
<James_F> |
Zuul: Add in-wikimedia-fundraising-production and assert shape |
[releng] |
| 18:15 |
<legoktm> |
building new shellcheck docker image |
[releng] |
| 17:43 |
<thcipriani> |
restarting gerrit due to gc thrashing |
[production] |
| 17:26 |
<James_F> |
[DonationInterface] Switch special FR job from REL1_31-php70 to REL1_35-php73 |
[releng] |
| 17:06 |
<James_F> |
[FundraisingEmailUnsubscribe] Switch special FR job from REL1_31-php70 to REL1_35-php73 |
[releng] |
| 16:25 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@89b308c]: prevent creation of VMs with non-ceph flavors (duration: 04m 08s) |
[production] |
| 16:21 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@89b308c]: prevent creation of VMs with non-ceph flavors |
[production] |
| 15:36 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 15:36 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 15:11 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 15:01 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:41 |
<effie> |
pooling mw2279.codfw.wmnet T264698 |
[production] |
| 12:11 |
<jiji@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:09 |
<jiji@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:35 |
<reedy@deploy1001> |
Synchronized php-1.36.0-wmf.13/extensions/ProofreadPage/: Revert excessive escaping T265571 (duration: 01m 12s) |
[production] |
| 09:29 |
<arturo> |
[codfw1dev] still some DNS weirdness, investigating |
[admin] |
| 09:25 |
<arturo> |
[codfw1dev] hard-rebooting bastion-codfw1dev-02, seems in bad shape, doesn't even wake up in the virsh console |
[admin] |
| 09:23 |
<ema> |
text@esams (except for cp3050/cp3052): upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 |
[production] |
| 09:19 |
<ema> |
upload@esams: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 |
[production] |
| 09:18 |
<arturo> |
[codfw1dev] live-hacked cloudservices2002-dev /etc/powerdns/recursor.conf file to include cloud-codfw1dev-floating CIDR (185.15.57.0/29) while https://gerrit.wikimedia.org/r/c/operations/puppet/+/634050 is in review, so VMs with a floating IP can query the DNS recursor (T261724) |
[admin] |
| 09:08 |
<ema> |
upload@eqsin: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 |
[production] |
| 09:03 |
<XioNoX> |
eqsin, push CR 634473 |
[production] |
| 09:01 |
<ema> |
text@eqsin: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 |
[production] |
| 09:01 |
<arturo> |
[codfw1dev] basic network connectivity seems stable after cleaning up everything related to address scopes (T261724) |
[admin] |
| 08:53 |
<ema> |
upload@codfw: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 |
[production] |
| 08:52 |
<XioNoX> |
add BGP_IXP_RS_in to eqsin RS BGP sessions |
[production] |
| 08:48 |
<ema> |
text@codfw: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 |
[production] |
| 08:29 |
<ema> |
upload@eqiad: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 |
[production] |
| 08:24 |
<ema> |
text@eqiad: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 |
[production] |
| 08:21 |
<hashar> |
Nuking castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm |
[releng] |
| 08:09 |
<elukey> |
reboot stat1005/stat1008 to pick up correct GPU settings |
[production] |
| 08:08 |
<ema> |
upload@ulsfo: upgrade varnish to 6.0.6-1wm2, restart varnishkafka-webrequest T264074 |
[production] |
| 07:59 |
<ema> |
text@ulsfo: upgrade varnish to 6.0.6-1wm2, restart varnishkafka instances T264074 |
[production] |
| 07:48 |
<hashar> |
Disabling integration-agent-docker-1020 (the sole agent using Ceph): it is too slow # T260916 T265615 |
[releng] |
| 07:44 |
<hashar> |
arming keyholder on integration-cumin |
[releng] |
| 07:19 |
<dcausse@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@27d0b01]: cirrus namespace map: Align output columns with table (duration: 04m 22s) |
[production] |
| 07:15 |
<dcausse@deploy1001> |
Started deploy [wikimedia/discovery/analytics@27d0b01]: cirrus namespace map: Align output columns with table |
[production] |
| 06:57 |
<XioNoX> |
enable cr2-eqdfw:xe-0/1/2 |
[production] |
| 02:14 |
<eileen> |
civicrm revision changed from 585eb835d8 to 3c3dcf80ae, config revision is f76d7849bc |
[production] |
| 01:01 |
<ryankemper> |
Cleaning up a dangling no-longer-puppet-managed udev elasticsearch-readahead rule across all cirrus instances: `sudo cumin -b 36 C:profile::elasticsearch::cirrus 'sudo rm -fv /etc/udev/rules.d/elasticsearch-readahead.rules && sudo /sbin/udevadm control --reload && sudo /sbin/udevadm trigger'` |
[production] |
| 00:56 |
<cdanis@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 00:56 |
<cdanis@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 00:14 |
<Krinkle> |
https://meetbot.toolforge.org/ says 503, No webservice. `webservice status` says "Your webservice of type php7.3 is running on backend kubernetes". Do a restart. |
[tools.meetbot] |