|
2020-01-21
ยง
|
| 16:45 |
<XioNoX> |
install software upgrade on pfw3a-eqiad (primary, no restart yet) |
[production] |
| 16:35 |
<XioNoX> |
install software upgrade on pfw3b-eqiad (secondary, no restart yet) |
[production] |
| 16:15 |
<vgutierrez> |
copied prometheus-varnishkafka-exporter from stretch to buster on apt.w.o - T242093 |
[production] |
| 16:02 |
<vgutierrez> |
uploaded libvmod-tbf 2.0.91-2wm to apt.w.o (buster) - T242093 |
[production] |
| 14:57 |
<vgutierrez> |
uploaded libvmod-re2 1.3.1-3 to apt.w.o (buster) - T242093 |
[production] |
| 14:56 |
<vgutierrez> |
uploaded libvmod-netmapper 1.7-3 to apt.w.o (buster) - T242093 |
[production] |
| 14:39 |
<moritzm> |
stopping/masking tor on torrelay1001 T243288 |
[production] |
| 14:38 |
<effie> |
Rolling restart all eqiad mw api servers |
[production] |
| 14:37 |
<vgutierrez> |
uploaded varnish-modules 0.12-1+wmf2 to apt.w.o (buster) - T242093 |
[production] |
| 14:36 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:36 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:35 |
<_joe_> |
restart pybal on low-traffic eqiad to pick up new configuration |
[production] |
| 14:33 |
<cdanis@cumin2001> |
conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=apache2,name=mw13.* |
[production] |
| 14:33 |
<cdanis@cumin2001> |
conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=nginx,name=mw13.* |
[production] |
| 14:30 |
<cdanis@cumin2001> |
conftool action : set/weight=15; selector: cluster=api_appserver,dc=eqiad,service=nginx,name=mw12[23].* |
[production] |
| 14:24 |
<_joe_> |
restarting pybal on lvs low-traffic in codfw |
[production] |
| 14:02 |
<oblivian@puppetmaster1001> |
conftool action : set/weight=10:pooled=yes; selector: service=kubesvc,cluster=kubernetes |
[production] |
| 13:24 |
<marostegui> |
Clean up some gerrit grants on db1132 (m2 master) T233714 |
[production] |
| 13:00 |
<mvolz@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'zotero' for release 'production' . |
[production] |
| 12:29 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 00m 58s) |
[production] |
| 12:28 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 01m 00s) |
[production] |
| 12:21 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'zotero' for release 'production' . |
[production] |
| 12:19 |
<vgutierrez> |
upgrading pybal on esams and eqiad - T169765 |
[production] |
| 12:12 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 00m 59s) |
[production] |
| 12:07 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 01m 12s) |
[production] |
| 11:56 |
<vgutierrez> |
upgrading pybal on eqsin and codfw - T169765 |
[production] |
| 11:54 |
<vgutierrez> |
restarting pybal instancs on eqsin |
[production] |
| 11:52 |
<_joe_> |
restarting etcd on conf2003 to test new pybal reconnection. Issues expected for pybal in eqsin, but not in ulsfo |
[production] |
| 11:44 |
<jbond42> |
importing puppet-master packages to component/puppet5 |
[production] |
| 11:39 |
<mvolz@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 11:23 |
<vgutierrez> |
Updating pybal to 1.15.7 on ulsfo load balancers - T169765 |
[production] |
| 11:23 |
<vgutierrez> |
uploaded pybal 1.15.7 to apt.w.o (stretch) - T169765 |
[production] |
| 11:22 |
<mvolz@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 10:47 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . |
[production] |
| 10:40 |
<mvolz@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'zotero' for release 'staging' . |
[production] |
| 10:38 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'zotero' for release 'staging' . |
[production] |
| 10:36 |
<godog> |
roll-restart thumbor after https://gerrit.wikimedia.org/r/c/operations/puppet/+/566069 |
[production] |
| 10:05 |
<volans@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:05 |
<volans@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:34 |
<oblivian@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 07:29 |
<oblivian@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . |
[production] |
| 07:23 |
<_joe_> |
adding TLS to citoid in production |
[production] |
| 07:20 |
<oblivian@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . |
[production] |
| 06:28 |
<marostegui> |
Remove the following users from phabricator database: 'phadmin'@'10.64.48.21' 'phuser'@'10.64.48.21' 'phstats'@'10.64.48.21' 'phmanifest'@'10.64.48.21' T238957 |
[production] |
| 06:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1087', diff saved to https://phabricator.wikimedia.org/P10233 and previous config saved to /var/cache/conftool/dbconfig/20200121-061932-marostegui.json |
[production] |
| 06:18 |
<marostegui> |
Aborted upgrade on db1087 (wiki dumps are running) |
[production] |
| 06:18 |
<marostegui> |
Upgrade db1087 |
[production] |
| 06:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1087 for upgrade', diff saved to https://phabricator.wikimedia.org/P10232 and previous config saved to /var/cache/conftool/dbconfig/20200121-061756-marostegui.json |
[production] |
| 06:05 |
<marostegui> |
Stop replication on db1107 |
[production] |
| 05:58 |
<marostegui> |
Stop MySQL on es2021 to clone es2022 - T243052 |
[production] |