|
2021-08-16
§
|
| 23:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:20 |
<urbanecm> |
Evening B&C window done |
[production] |
| 23:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 23:18 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a14868bbdf442eede5711576c4b4da51df0ccd77: Enable NewUserMessage on hiwiktionary (T287091) (duration: 01m 00s) |
[production] |
| 23:15 |
<eileen> |
civicrm revision changed from 1e32084622 to ba0c7705bb, config revision is 7bdc78073d |
[production] |
| 22:13 |
<bblack> |
dns[1235]002: upgrade gdnsd package to 3.8.0-1~wmf1 |
[production] |
| 21:31 |
<bblack> |
authdns1001: upgrade gdnsd package to 3.8.0-1~wmf1 |
[production] |
| 21:28 |
<bblack> |
dns4002: upgrade gdnsd package to 3.8.0-1~wmf1 |
[production] |
| 20:38 |
<bstorm@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 20:38 |
<bstorm@cumin1001> |
Added views for new wiki: labswiki T287442 |
[production] |
| 20:37 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 20:36 |
<bstorm@cumin1001> |
END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99) |
[production] |
| 20:36 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 20:35 |
<bstorm@cumin1001> |
END (FAIL) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=99) |
[production] |
| 20:35 |
<bstorm@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 18:48 |
<dancy> |
Restarted Jenkins due to stuck jobs. |
[production] |
| 18:17 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:15 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:52 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1453.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:49 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1453.eqiad.wmnet with reason: REIMAGE |
[production] |
| 17:34 |
<cmjohnson1> |
installing new line card in slot1 cr2-eqiad T277339 |
[production] |
| 17:33 |
<ladsgroup@deploy1002> |
Synchronized php-1.37.0-wmf.18/extensions/SpamBlacklist/includes/SpamBlacklistHooks.php: Backport: [[gerrit:712965|Try to use EditStash before re-rendering (T288639)]] (duration: 00m 59s) |
[production] |
| 17:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 17:25 |
<XioNoX> |
cr1-eqiad> request chassis fpc offline slot 5 - T277339 |
[production] |
| 17:17 |
<cmjohnson1> |
installing new line card in slot1 cr1-eqiad T277339 |
[production] |
| 17:11 |
<ejegg> |
updated fundraising CiviCRM from f3895dc907 to 1e32084622 |
[production] |
| 17:08 |
<XioNoX> |
asw2-a-eqiad> request virtual-chassis vc-port set pic-slot 1 member 8 port 1 - T288834 |
[production] |
| 17:05 |
<XioNoX> |
asw2-a-eqiad> request virtual-chassis vc-port delete pic-slot 1 member 8 port 1 - T288834 |
[production] |
| 16:40 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:37 |
<cwhite> |
restart logstash on logstash1008 |
[production] |
| 16:36 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:01 |
<mutante> |
LDAP - added user tandic to nda group (T288527) |
[production] |
| 15:37 |
<ryankemper> |
[WDQS] Re-pooled `codfw`: `ryankemper@puppetmaster1001:~$ sudo -i confctl --quiet --object-type discovery select 'dnsdisc=wdqs,name=codfw' set/pooled=true` |
[production] |
| 14:42 |
<mutante> |
miscweb - deploying new microsite for Wikidata Query Builder subpage (T266703) |
[production] |
| 14:41 |
<mutante> |
mw1455 - works fine after a reimage, unknown why it didnt last time, but ok :) |
[production] |
| 14:11 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1455.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:08 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1455.eqiad.wmnet with reason: REIMAGE |
[production] |
| 13:53 |
<mutante> |
mw1455 - mysteriously showing a bunch of issues in icinga, broken packages, envoy, memcached etc, after recent fresh install, trying another reimage (T273915) |
[production] |
| 13:50 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:48 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:42 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711515|Remove $wmgWikibaseFineGrainedLuaTracking (T288612)]] (duration: 00m 58s) |
[production] |
| 13:42 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:40 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue |
[production] |
| 13:40 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue |
[production] |
| 13:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:40 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:711514|Stop setting $wgWBClientSettings['fineGrainedLuaTracking'] (T288612)]] (duration: 00m 58s) |
[production] |
| 13:37 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:711513|Remove $wmgWikibaseClientUseTermsTableSearchFields (T288612)]] (beta, 2/2) (duration: 00m 59s) |
[production] |
| 13:36 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:711513|Remove $wmgWikibaseClientUseTermsTableSearchFields (T288612)]] (prod, 1/2) (duration: 00m 59s) |
[production] |