|
2019-02-15
§
|
| 10:35 |
<gtirloni> |
reboot cloudvirt1019 |
[production] |
| 09:44 |
<gehel> |
repool maps100[12] |
[production] |
| 09:33 |
<moritzm> |
imported php-defaults debs to thirdparty/php72 |
[production] |
| 08:42 |
<akosiaris> |
restart gerrit to pick up https://gerrit.wikimedia.org/r/490640 T177868 |
[production] |
| 08:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 46s) |
[production] |
| 08:28 |
<moritzm> |
rolling restart of apertium to pick up Python 3.4 security update |
[production] |
| 07:55 |
<godog> |
bounce prometheus@ops on prometheus2004 to take a snapshot |
[production] |
| 06:40 |
<marostegui> |
Stop puppet on labsdb1005 to leave "max_user_connections" on my.cnf - T216170 T216208 |
[production] |
| 06:39 |
<marostegui> |
Restart labsdb1005 with max_user_connections = 20 T216208 |
[production] |
| 06:20 |
<Joan> |
Restarted CVNBot18 (Last message was received on RCReader 53393.770933 seconds ago) |
[cvn] |
| 06:17 |
<marostegui> |
Deploy schema change on db1109 - T210713 |
[production] |
| 06:16 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 49s) |
[production] |
| 06:13 |
<marostegui> |
Reload haproxy on dbproxy11 to repool labsdb1009 |
[production] |
| 03:50 |
<andrewbogott> |
updated VPS base images for Jessie and Stretch, now featuring Stretch 9.7 |
[admin] |
| 03:43 |
<bd808> |
Restarted webservice |
[tools.admin] |
| 02:25 |
<bd808> |
Restarted webservice; investigating toolsdb query issues |
[tools.wd-depicts] |
| 02:06 |
<bd808> |
Disabled cache refresh cron |
[tools.tool-db-usage] |
| 01:51 |
<bd808> |
KIlled webservice to see if that releases open database connections |
[tools.phragile] |
| 01:28 |
<bd808> |
Re-enabled PAWS vhost on paws-proxy-02 |
[paws] |
| 00:39 |
<mutante> |
puppetmaster1001: sudo puppet node clean bast3003.wikimedia.org ; sudo puppet node deactivate bast3003.wikimedia.org (T216199) |
[production] |
| 00:15 |
<jynus> |
setting labsdb1005 back into read-write |
[production] |
|
2019-02-14
§
|
| 23:47 |
<jynus> |
restarting labsdb1005 mysql in read only mode |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s) |
[production] |
| 23:37 |
<niharika29@deploy1001> |
Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 |
[production] |
| 22:25 |
<gtirloni> |
downtimed PAWS in Icinga |
[paws] |
| 22:16 |
<gtirloni> |
Activated maintenance page on paws-proxy-02 nginx config |
[paws] |
| 22:07 |
<andrewbogott> |
rebuilding labvirt1012 as cloudvirt1012, T216190 |
[production] |
| 21:57 |
<bd808> |
Deleted old tools-proxy-02 instance |
[tools] |
| 21:57 |
<bd808> |
Deleted old tools-proxy-01 instance |
[tools] |
| 21:56 |
<bd808> |
Deleted old tools-package-builder-01 instance |
[tools] |
| 21:28 |
<bd808> |
Updated config to point to tools-elastic-01 and restarted |
[tools.sal] |
| 20:57 |
<andrewbogott> |
rebooting tools-worker-1005 |
[tools] |
| 20:50 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T216102 |
[releng] |
| 20:50 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/490528 / T133646 |
[releng] |
| 20:38 |
<bstorm_> |
Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005 |
[production] |
| 20:34 |
<andrewbogott> |
moving tools-exec-1409, tools-exec-1410, tools-exec-1414, tools-exec-1419 |
[tools] |
| 20:18 |
<thcipriani> |
thcipriani@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:14 |
<thcipriani@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.17 |
[production] |
| 20:09 |
<ejegg> |
updated fundraising CiviCRM from 02ea871b88 to 165fbf5894 |
[production] |
| 19:55 |
<andrewbogott> |
moving tools-webgrid-generic-1401 and tools-webgrid-lighttpd-1419 |
[tools] |
| 19:42 |
<thcipriani@deploy1001> |
Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: [[gerrit:490674|Help Panel: Fix IME broken in help panel search]] T216131 (duration: 00m 54s) |
[production] |
| 19:33 |
<andrewbogott> |
moving tools-checker-01 to labvirt1003 |
[tools] |
| 19:25 |
<andrewbogott> |
moving tools-elastic-02 to labvirt1003 |
[tools] |
| 19:14 |
<thcipriani@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:487007|Stop NavPopups gadget conflict with PagePreviews on Wikivoyage]] T214878 (duration: 00m 54s) |
[production] |
| 19:11 |
<andrewbogott> |
moving tools-k8s-etcd-01 to labvirt1002 |
[tools] |
| 19:01 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:58 |
<bd808> |
Stopped webservice. Implicated in ToolsDB connection overload outage. |
[tools.fountain] |
| 18:52 |
<mutante> |
scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev |
[production] |
| 18:37 |
<andrewbogott> |
moving tools-exec-1418, tools-exec-1424 to labvirt1003 |
[tools] |
| 18:34 |
<andrewbogott> |
moving tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1406, tools-webgrid-lighttpd-1410 to labvirt1002 |
[tools] |