|
2020-05-28
ยง
|
| 16:10 |
<James_F> |
Zuul: Move MediaModeration to production section and enforce T247943 |
[releng] |
| 16:01 |
<bstorm_> |
kubectl upgraded to 1.16.10 on all bastions T246122 |
[tools] |
| 15:58 |
<arturo> |
upgrading tools-k8s-worker-[1..10] to 1.16.10 (T246122) |
[tools] |
| 15:41 |
<arturo> |
upgrading tools-k8s-control-3 to 1.16.10 (T246122) |
[tools] |
| 15:37 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] (duration: 00m 10s) |
[production] |
| 15:37 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182] (thin): Three hotfixes (THIN) [analytics/refinery@203d182] |
[production] |
| 15:17 |
<arturo> |
upgrading tools-k8s-control-2 to 1.16.10 (T246122) |
[tools] |
| 15:09 |
<arturo> |
upgrading tools-k8s-control-1 to 1.16.10 (T246122) |
[tools] |
| 15:05 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] (duration: 25m 59s) |
[production] |
| 15:02 |
<moritzm> |
installing exim4 security updates on jessie (stretch/buster already fixed) |
[production] |
| 14:49 |
<arturo> |
cleanup /etc/apt/sources.list.d/ directory in all tools-k8s-* VMs |
[tools] |
| 14:39 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@203d182]: Three hotfixes [analytics/refinery@203d182] |
[production] |
| 14:33 |
<andrew@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:30 |
<andrew@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:26 |
<ZI_Jony> |
restarted CVNBot14 |
[cvn] |
| 14:01 |
<ema> |
atskafka 0.8 uploaded to buster-wikimedia T253551 |
[production] |
| 13:49 |
<godog> |
roll-restart prometheus k8s-staging to enable thanos upload - T252186 |
[production] |
| 13:36 |
<hashar> |
Restarting CI Jenkins for plugin rollback |
[production] |
| 13:36 |
<andrewbogott> |
rebuilding cloudservices2002-dev with Buster |
[admin] |
| 13:31 |
<hashar> |
Rolling back Xunit Jenkins plugin # T194096 |
[releng] |
| 11:49 |
<moritzm> |
installing unbound security updates |
[production] |
| 11:27 |
<arturo> |
cleanup livehackings |
[toolsbeta] |
| 11:27 |
<arturo> |
merging change to front-proxy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) |
[tools] |
| 11:03 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Add db2138 to s2+s4 T252985', diff saved to https://phabricator.wikimedia.org/P11330 and previous config saved to /var/cache/conftool/dbconfig/20200528-110333-kormat.json |
[production] |
| 10:36 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:34 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:31 |
<arturo> |
livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) |
[toolsbeta] |
| 10:30 |
<arturo> |
livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 |
[toolsbeta] |
| 10:30 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 10:21 |
<RhinosF1> |
Known inconsistency in git - unable to pull changes (NOW RESOLVED) |
[tools.zppixbot] |
| 10:20 |
<RhinosF1> |
Known inconsistency in git - unable to pull changes |
[tools.zppixbot] |
| 10:07 |
<wm-bot> |
<rhinosf1> tweaking stuff for testing |
[tools.zppixbot-test] |
| 10:02 |
<mutante> |
gerrit1002 (test server) - chown -R gerrit2:gerrit2 /var/lib/gerrit/review_site ; restarted gerrit service, now the service is not in restart loop anymore, gerrit-ssh is listening too, just not accepting publickey (T239151) |
[production] |
| 09:51 |
<XioNoX> |
failover VRRP in ulsfo |
[production] |
| 09:41 |
<XioNoX> |
re-activate peering/transit on cr2-eqdfw - T243080 |
[production] |
| 09:35 |
<mutante> |
restarting gerrit on gerrit1002 after fixing db_pass to the readonly one (T243800) |
[production] |
| 09:33 |
<XioNoX> |
restart cr2-eqdfw for upgrade - T243080 |
[production] |
| 09:30 |
<XioNoX> |
deactivate peering/transit on cr2-eqdfw - T243080 |
[production] |
| 09:25 |
<_joe_> |
updating ACLs on all etcd servers |
[production] |
| 09:22 |
<XioNoX> |
install new Junos on cr2-eqdfw - T243080 |
[production] |
| 09:16 |
<XioNoX> |
rollback cr2-eqord ospf/bgp - T243080 |
[production] |
| 09:07 |
<XioNoX> |
restart cr2-eqord for upgrade - T243080 |
[production] |
| 09:05 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 08:50 |
<_joe_> |
upgrading etcd ACLs (adding new users) to conf1004 |
[production] |
| 08:50 |
<XioNoX> |
install new Junos on cr2-eqord - T243080 |
[production] |
| 08:46 |
<XioNoX> |
deactivate peering/transit on cr2-eqord - T243080 |
[production] |
| 08:45 |
<XioNoX> |
de-pref all OSPF links to cr2-eqord - T243080 |
[production] |
| 08:13 |
<marostegui> |
Pool db1141 into labsdb analytics role - T249188 |
[production] |
| 07:33 |
<gilles@deploy1001> |
Synchronized static/images: T252108 Deploying optimised static PNGs (duration: 01m 39s) |
[production] |
| 07:31 |
<gilles@deploy1001> |
Synchronized static/apple-touch: T252108 Deploying optimised static PNGs (duration: 01m 12s) |
[production] |