|
2019-03-15
§
|
| 09:19 |
<jiji@cumin1001> |
conftool action : set/weight=12; selector: dc=eqiad,service=cxserver,cluster=scb,name=kubernetes.* |
[production] |
| 09:18 |
<jiji@cumin1001> |
conftool action : set/weight=15; selector: dc=codfw,service=cxserver,cluster=scb,name=kubernetes.* |
[production] |
| 09:17 |
<jijiki> |
Ramp up cxserver k8s traffic to 50% - T213195 |
[production] |
| 08:57 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1103 (duration: 00m 50s) |
[production] |
| 08:52 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 47s) |
[production] |
| 08:22 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s) |
[production] |
| 07:28 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1091 (duration: 00m 49s) |
[production] |
| 07:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) |
[production] |
| 07:05 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) |
[production] |
| 07:01 |
<kart_> |
Finished manual run of unpublished ContentTranslation draft purge script (T217818) |
[production] |
| 06:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1091 (duration: 00m 48s) |
[production] |
| 06:44 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1091 (duration: 00m 48s) |
[production] |
| 06:04 |
<marostegui> |
Upgrade db1091 |
[production] |
| 06:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 50s) |
[production] |
| 04:01 |
<kart_> |
Started manual run of unpublished ContentTranslation draft purge script (T217818) |
[production] |
| 02:25 |
<gtirloni> |
activated TLS termination using Let's Encrypt on paws-proxy-02 |
[paws] |
| 02:25 |
<gtirloni> |
removed webproxies and created new A records pointing directly to paws-proxy-02 |
[paws] |
| 01:25 |
<ejegg> |
re-enabled ingenico audit parser |
[production] |
| 01:25 |
<ejegg> |
updated fundraising CiviCRM from 41efa14fb0 to a2316be94f |
[production] |
|
2019-03-14
§
|
| 23:52 |
<bd808> |
Disabled job queues and rescheduled continuous jobs away from tools-exec-14{21,22,23,24,25,26,27,28,29,30,31,32} (T217152) |
[tools] |
| 23:48 |
<Krinkle> |
Abort job quibble-vendor-mysql-hhvm-docker/39874/ for mwext-CentralAuth (stuck after 59 minutes) |
[releng] |
| 23:28 |
<bd808> |
Deleted tools-bastion-05 (T217152) |
[tools] |
| 23:20 |
<bd808> |
Deleted living-style-guide and lsg-01 (T204502) |
[design] |
| 23:18 |
<bd808> |
Deleted dumps-stats and bugzilla (T204503) |
[dumps] |
| 23:15 |
<bd808> |
Deleted vrt (T204508) |
[getstarted] |
| 23:13 |
<bd808> |
Deleted drmf, math-ru, mathoid2 (T204509) |
[math] |
| 23:11 |
<bd808> |
Deleted osm-serv and osmit-due (T204527) |
[osmit] |
| 23:09 |
<bd808> |
Deleted queryr-db-01 and queryr-api-01 (T204683) |
[queryrapi] |
| 23:06 |
<bd808> |
Deleted telnet2.telnet.eqiad.wmflabs (T204694) |
[telnet] |
| 23:05 |
<bd808> |
Deleted danny-b.wildcat.eqiad.wmflabs (T204703) |
[wildcat] |
| 22:54 |
<ejegg> |
temporarily disabled Ingenico WX audit parsing |
[production] |
| 22:30 |
<bd808> |
Removed obsolete submit hosts from Trusty grid config |
[tools] |
| 22:20 |
<bd808> |
Removed tools-webgrid-lighttpd-142{0,1,2,5} from the grid and shutdown instances via horizon (T217152) |
[tools] |
| 22:10 |
<bd808> |
Depooled tools-webgrid-lighttpd-142{0,1,2,5} (T217152) |
[tools] |
| 22:05 |
<cdanis> |
cdanis@icinga2001.wikimedia.org ~ % sudo systemctl restart icinga.service |
[production] |
| 21:58 |
<cdanis> |
cdanis@icinga2001.wikimedia.org ~ % sudo systemctl restart nsca.service |
[production] |
| 21:55 |
<bd808> |
Removed submit host flag from tools-bastion-05.tools.eqiad.wmflabs, removed floating ip, and shutdown instance via horizon (T217152) |
[tools] |
| 21:48 |
<bd808> |
Removed tools-exec-14{33,34,35,36,37,38,39,40,41,42} from the grid and shutdown instances via horizon (T217152) |
[tools] |
| 21:38 |
<gtirloni> |
rebooted tools-sgewebgrid-generic-0904 (T218341) |
[tools] |
| 21:32 |
<gtirloni> |
rebooted tools-exec-1020 (T218341) |
[tools] |
| 21:25 |
<hashar> |
Manually triggered tests for 12 ContentTranslation changes that had label:verified=-1 # T216689 |
[releng] |
| 21:23 |
<gtirloni> |
rebooted tools-sgeexec-0919, tools-sgeexec-0934, tools-worker-1018 (T218341) |
[tools] |
| 21:21 |
<hashar> |
Updated quibble-vendor-mysql-hhvm-docker with latest libc6 hopefully fixing HHVM segfault within libpthread # T216689 |
[releng] |
| 21:19 |
<bd808> |
Killed jobs still running on tools-exec-14{33,34,35,36,37,38,39,40,41,42}.tools.eqiad.wmflabs 2 weeks after being depooled (T217152) |
[tools] |
| 21:01 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@090a0c3]: Another minor bugfix releaes for ganeti-netbox script (duration: 00m 56s) |
[production] |
| 21:00 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@090a0c3]: Another minor bugfix releaes for ganeti-netbox script |
[production] |
| 20:58 |
<bd808> |
Repooled tools-sgeexec-0941 following reboot |
[tools] |
| 20:57 |
<bd808> |
Hard reboot of tools-sgeexec-0941 via horizon |
[tools] |
| 20:54 |
<bd808> |
Depooled and rebooted tools-sgeexec-0941.tools.eqiad.wmflabs |
[tools] |
| 20:53 |
<bd808> |
Repooled tools-sgeexec-0917 following reboot |
[tools] |