|
2020-01-21
ยง
|
| 22:53 |
<jeh> |
deactivated image debian-9.1-stretch (deprecated 2017-11-16) |
[openstack] |
| 22:10 |
<Krenair> |
deployment-prep added jbond as admin, T243226 |
[releng] |
| 22:09 |
<mutante> |
gerrit - rsyncing 'git' and 'plugin' data dirs and /var/lib/gerrit2/review_site/ from gerrit1001 to gerrit1002 WITH --delete T239151 |
[releng] |
| 21:48 |
<mutante> |
gerrit - rsyncing git data from gerrit1001 to gerrit1002 (T239151) |
[releng] |
| 20:09 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.16 |
[production] |
| 19:59 |
<mutante> |
puppet-compilers: syncing facts from puppetmasters to 3 compiler instances |
[production] |
| 19:55 |
<XioNoX> |
restart mr1-esams for software upgrade - T242097 |
[production] |
| 19:54 |
<mutante> |
syncing puppet facts from local puppet repo |
[puppet-diffs] |
| 19:46 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@1ca3071]: Add separate rule for machine vision jobs T241072 (duration: 01m 11s) |
[production] |
| 19:45 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@1ca3071]: Add separate rule for machine vision jobs T241072 |
[production] |
| 19:40 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:39 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:39 |
<XioNoX> |
mr1-esams> request system software add /var/tmp/junos-srxsme-18.2R3-S2... - T242097 |
[production] |
| 19:39 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:38 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:25 |
<bstorm_> |
hard rebooting tools-sgeexec-0913/14/35 because they aren't even on the network |
[tools] |
| 19:22 |
<XioNoX> |
cr3-knams# set routing-options ppm no-delegate-processing - T240659 |
[production] |
| 19:17 |
<bstorm_> |
depooled and rebooted tools-sgeexec-0914 because it was acting funny |
[tools] |
| 19:16 |
<marxarelli> |
updating 105 jobs affected by https://gerrit.wikimedia.org/r/c/integration/config/+/566332 |
[releng] |
| 19:01 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 19:00 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 19:00 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 18:59 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 18:50 |
<brennen@deploy1001> |
Finished scap: testwiki to php-1.35.0-wmf.16 and rebuild l10n cache (duration: 30m 27s) |
[production] |
| 18:30 |
<bstorm_> |
depooling and rebooting tools-sgeexec-[0911,0913,0919,0921,0924,0931,0933,0935,0939,0941].tools.eqiad.wmflabs |
[tools] |
| 18:19 |
<brennen@deploy1001> |
Started scap: testwiki to php-1.35.0-wmf.16 and rebuild l10n cache |
[production] |
| 17:45 |
<XioNoX> |
add dwisehaupt user to pfw/fasw - T242758 |
[production] |
| 17:44 |
<ebernhardson@deploy1001> |
Finished deploy [search/mjolnir/deploy@986769c]: bulk_daemon: Treat model exists as unrecoverable failure (duration: 05m 42s) |
[production] |
| 17:43 |
<bstorm_> |
remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects |
[admin] |
| 17:39 |
<ebernhardson@deploy1001> |
Started deploy [search/mjolnir/deploy@986769c]: bulk_daemon: Treat model exists as unrecoverable failure |
[production] |
| 17:37 |
<bstorm_> |
re-exported NFS from labstore1006/7 |
[production] |
| 17:33 |
<ebernhardson@deploy1001> |
Finished deploy [wikimedia/discovery/analytics@ae77f9d]: Deploy ores_drafttopics dag (duration: 00m 22s) |
[production] |
| 17:32 |
<ebernhardson@deploy1001> |
Started deploy [wikimedia/discovery/analytics@ae77f9d]: Deploy ores_drafttopics dag |
[production] |
| 17:21 |
<bstorm_> |
rebooting toolschecker to recover stale nfs handle |
[tools] |
| 17:20 |
<brennen> |
starting branch cut for T233864 |
[production] |
| 17:08 |
<XioNoX> |
restart pfw3-eqiad for software upgrade |
[production] |
| 16:45 |
<XioNoX> |
install software upgrade on pfw3a-eqiad (primary, no restart yet) |
[production] |
| 16:35 |
<XioNoX> |
install software upgrade on pfw3b-eqiad (secondary, no restart yet) |
[production] |
| 16:15 |
<vgutierrez> |
copied prometheus-varnishkafka-exporter from stretch to buster on apt.w.o - T242093 |
[production] |
| 16:02 |
<vgutierrez> |
uploaded libvmod-tbf 2.0.91-2wm to apt.w.o (buster) - T242093 |
[production] |
| 14:57 |
<vgutierrez> |
uploaded libvmod-re2 1.3.1-3 to apt.w.o (buster) - T242093 |
[production] |
| 14:56 |
<vgutierrez> |
uploaded libvmod-netmapper 1.7-3 to apt.w.o (buster) - T242093 |
[production] |
| 14:39 |
<moritzm> |
stopping/masking tor on torrelay1001 T243288 |
[production] |
| 14:38 |
<effie> |
Rolling restart all eqiad mw api servers |
[production] |
| 14:37 |
<vgutierrez> |
uploaded varnish-modules 0.12-1+wmf2 to apt.w.o (buster) - T242093 |
[production] |
| 14:36 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:36 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:35 |
<_joe_> |
restart pybal on low-traffic eqiad to pick up new configuration |
[production] |
| 14:33 |
<cdanis@cumin2001> |
conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=apache2,name=mw13.* |
[production] |
| 14:33 |
<cdanis@cumin2001> |
conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=nginx,name=mw13.* |
[production] |