|
2019-12-11
ยง
|
| 19:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1097:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9858 and previous config saved to /var/cache/conftool/dbconfig/20191211-195306-marostegui.json |
[production] |
| 19:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3314 after schema change T233135', diff saved to https://phabricator.wikimedia.org/P9857 and previous config saved to /var/cache/conftool/dbconfig/20191211-195130-marostegui.json |
[production] |
| 19:43 |
<awight> |
Morning SWAT complete |
[production] |
| 19:43 |
<eileen> |
re-enabled dedupe (off from last night benevity import attempt) |
[production] |
| 19:43 |
<eileen> |
process-control config revision is 8c073ae64a |
[production] |
| 19:34 |
<awight@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Cite: SWAT: [[gerrit:556372|Lazily fetch user interface language to prevent cache split (take 2) (T240426, T239988)]] (duration: 00m 40s) |
[production] |
| 19:33 |
<awight> |
Overriding scap canaries for T240426 |
[production] |
| 19:24 |
<awight@deploy1001> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 19:18 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: eaa4c2c: Remove unused wmgCheckUserForceSummary (duration: 01m 01s) |
[production] |
| 19:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: c8fe811: Use wgCheckUserForceSummary instead of wmgCheckUserForceSummary (duration: 01m 02s) |
[production] |
| 19:11 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: c8fe811: Use wgCheckUserForceSummary instead of wmgCheckUserForceSummary (duration: 01m 02s) |
[production] |
| 19:10 |
<urbanecm@deploy1001> |
sync-file aborted: SWAT: c8fe811: Use wgCheckUserForceSummary instead of wmgCheckUserForceSummary (T239936) (duration: 00m 02s) |
[production] |
| 19:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 9ca31f4: Enable CheckUser Special:Investigate page on testwiki (T239936) (duration: 01m 02s) |
[production] |
| 18:53 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.10/extensions/MachineVision: Fix no-JS warning message (T240210) (duration: 01m 02s) |
[production] |
| 16:43 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.10/extensions/Wikibase/lib/includes/Store/Sql/SqlEntityInfoBuilder.php: Consider any type of empty value as uncached in SqlEntityInfoBuilder (T237984) (duration: 01m 03s) |
[production] |
| 16:39 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Wikibase/lib/includes/Store/Sql/SqlEntityInfoBuilder.php: Consider any type of empty value as uncached in SqlEntityInfoBuilder (T237984) (duration: 01m 03s) |
[production] |
| 16:28 |
<ema@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be |
[production] |
| 16:28 |
<ema> |
cp1075 ats-be: temporarily switch to plain HTTP for api and appservers (apache directly instead of nginx) |
[production] |
| 16:24 |
<ema@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=ats-be |
[production] |
| 15:23 |
<rzl@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 15:21 |
<rzl@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 15:14 |
<rzl@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 14:45 |
<rlazarus> |
updating envoyproxy to 1.12.2 on all eqiad T238050 |
[production] |
| 14:43 |
<rlazarus> |
updating envoyproxy to 1.12.2 on all codfw T238050 |
[production] |
| 14:19 |
<rlazarus> |
updating envoyproxy to 1.12.2 on mwmaint, restbase T238050 |
[production] |
| 14:00 |
<rlazarus> |
uploaded envoyproxy-1.12.2 to reprepro |
[production] |
| 13:37 |
<awight> |
EU SWAT complete |
[production] |
| 13:25 |
<andrew-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Cite: SWAT: [[gerrit:556367|Revert "Lazily fetch user interface language to prevent cache split" ()]] (duration: 01m 02s) |
[production] |
| 12:54 |
<andrew-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.10/extensions/Cite: SWAT: [[gerrit:556351|Use messagelocalizer in CiteErrorReporter (T239988)]] (duration: 01m 04s) |
[production] |
| 12:38 |
<andrew-wmde@deploy1001> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 12:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 7651c1a: GrowthExperiments: Configure testwiki to use local search & config (T235717) (duration: 01m 02s) |
[production] |
| 12:03 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.10/extensions/Wikibase/data-access: [[gerrit:556353|Fix idlookup dropping pageids (T236691 T240410)]] (duration: 01m 03s) |
[production] |
| 12:00 |
<moritzm> |
installing git security updates |
[production] |
| 11:57 |
<jbond42> |
draining kubernetes2003 to restart calico-node |
[production] |
| 11:55 |
<jbond42> |
draining kubernetes2002 to restart calico-node |
[production] |
| 11:52 |
<jbond42> |
draining kubernetes2001 to restart calico-node |
[production] |
| 11:36 |
<jbond42> |
draining kubernetes1004.eqiad.wmnet to restart calico-node |
[production] |
| 11:31 |
<jbond42> |
draining kubernetes1005.eqiad.wmnet to restart calico-node |
[production] |
| 11:27 |
<jbond42> |
draining kubernetes1006.eqiad.wmnet to restart calico-node |
[production] |
| 10:51 |
<jbond42> |
draining kubernetes1003.eqiad.wmnet to restart calico-node |
[production] |
| 10:48 |
<jbond42> |
draining kubernetes1002.eqiad.wmnet to restart calico-node |
[production] |
| 10:45 |
<marostegui> |
Deploy schema change on db1103:3314 |
[production] |
| 10:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3314 for schema change T233135', diff saved to https://phabricator.wikimedia.org/P9851 and previous config saved to /var/cache/conftool/dbconfig/20191211-104506-marostegui.json |
[production] |
| 10:39 |
<jbond42> |
draining kubernetes1001.eqiad.wmnet to restart calico-node |
[production] |
| 10:34 |
<Nikerabbit> |
Finished running Translate/refresh-translatable-pages.php --jobqueue for Translate wikis - T235027 T235188 |
[production] |
| 10:03 |
<ema> |
cp-ats: apply set_server_resp_no_store patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/556201/ to all hosts T227432 |
[production] |
| 09:45 |
<ema@cumin1001> |
conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet,service=ats-be |
[production] |
| 09:44 |
<ema> |
cp1075: repool ats-be after successful set_server_resp_no_store test P9849 T227432 |
[production] |
| 09:33 |
<godog> |
roll-restart logstash in codfw/eqiad after https://gerrit.wikimedia.org/r/c/operations/puppet/+/556173 |
[production] |
| 09:25 |
<ema@cumin1001> |
conftool action : set/pooled=no; selector: name=cp1075.eqiad.wmnet,service=ats-be |
[production] |