2351-2400 of 10000 results (27ms)
2020-06-08 §
11:49 <XioNoX> reboot cr2-codfw:re1 (backup) - T243080 [production]
11:45 <moritzm> restarting slapd on ldap-corp* for Gnu TLS security update [production]
11:43 <moritzm> rolling restart of Apache on Kibana/7 host to pick up Gnu TLS security update [production]
11:41 <XioNoX> de-pref cr2-codfw OSPF - T243080 [production]
11:39 <XioNoX> deactivate cr2-codfw transit/peering - T243080 [production]
11:38 <XioNoX> fail vrrp master from cr2 to cr1 - T243080 [production]
11:32 <XioNoX> cr1-codfw set OSPF metrics back to normal - T243080 [production]
11:30 <XioNoX> cr1-codfw re-enable transit/peering - T243080 [production]
11:29 <XioNoX> cr1-codfw add graceful-restart - T243080 [production]
11:28 <XioNoX> cr1-codfw add graceful-switchover - T243080 [production]
11:18 <Lucas_WMDE> EU SWAT done [production]
11:16 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:602981|Remove Wikibase idBlacklist setting (T254686)]], part 2 (duration: 00m 56s) [production]
11:15 <XioNoX> cr1-codfw> request chassis routing-engine master switch - T243080 [production]
11:15 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:602981|Remove Wikibase idBlacklist setting (T254686)]], part 1 (duration: 00m 56s) [production]
11:11 <XioNoX> reboot cr1-codfw:re0 (backup) - T243080 [production]
11:09 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:601409|Enable GrowthExperiments guidance everywhere behind feature flag (T253794)]] (duration: 00m 57s) [production]
11:05 <marostegui> Install events on es1 T254689 [production]
11:05 <XioNoX> install Junos on cr1-codfw:re0 (backup) - T243080 [production]
10:56 <XioNoX> do cr1-codfw RE mastership switch - T243080 [production]
10:51 <XioNoX> reboot cr1-codfw:re1 (backup) - T243080 [production]
10:46 <XioNoX> install Junos on cr1-codfw:re1 (backup) - T243080 [production]
10:43 <XioNoX> deactivate cr1-codfw transit/peering - T243080 [production]
10:41 <XioNoX> bump all cr1-codfw OSPF metrics - T243080 [production]
10:41 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:603408| Bumping portals to master (603408)]] (duration: 00m 57s) [production]
10:40 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:603408| Bumping portals to master (603408)]] (duration: 01m 09s) [production]
10:39 <XioNoX> depool codfw - T243080 [production]
09:46 <moritzm> installing gnutls28 security updates on buster (older releases not affected) [production]
09:32 <qchris> Turning on puppet on gerrit1002 again to avoid starting to lag too far behind [production]
08:17 <XioNoX> push T250136 to eqsin - T250136 [production]
08:09 <XioNoX> push T250136 to eqiad - T250136 [production]
08:07 <moritzm> upgrading mw1349-mw1383 to PHP 7.2.31 [production]
08:07 <mutante> stat1006 moved broken jupyter-dedcode-singleuser.service out of /run/systemd/transient. systemctl reset-failed [production]
08:02 <XioNoX> push T250136 to codfw - T250136 [production]
07:58 <XioNoX> push T250136 to eqord/eqdfw - T250136 [production]
07:58 <mutante> stat1006 bash[40607]: /bin/bash: line 0: exec: jupyterhub-singleuser: not found [production]
07:57 <mutante> ran puppet on all stat* hosts for an access request (dcipoletti was added) - stat1006 systemd state broke right after, jupyter-dedcode-singleuser.service failed [production]
07:46 <XioNoX> push T250136 to esams/knams - T250136 [production]
07:42 <XioNoX> cr4-ulsfo protocols bgp group Transit4 family inet any -> unicast - T250136 [production]
07:39 <XioNoX> cr3-ulsfo protocols bgp group Transit4 family inet any -> unicast - T250136 [production]
07:37 <moritzm> installing nodejs security updates [production]
07:05 <marostegui> Stop MySQL on labsdb1012 to clone labsdb1011 T249188 [production]
05:22 <marostegui> Upgrade db1077 to 10.4.13 to test events memory leak [production]
04:45 <_joe_> de-firewalling mc1029 [production]
04:27 <_joe_> firewallingf off memcached on mc1029 [production]
2020-06-05 §
16:45 <elukey@deploy1001> Finished deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 (duration: 00m 11s) [production]
16:45 <elukey@deploy1001> Started deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 [production]
16:29 <bd808> Testing stashbot following hard restart of service. It was having LDAP connection failure problems. [production]
16:00 <AndyRussG> Turned off Fundraising job recurring_smashpig_charge [production]
15:54 <cdanis> enabling & rerunning puppet on netflow* T254574 [production]
15:39 <cdanis> disabling puppet on netflow* and trying I6598d8f8 on netflow3001 first T254574 [production]