|
2017-09-04
§
|
| 13:37 |
<gehel> |
rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933 |
[production] |
| 13:34 |
<gehel> |
delete leftover grafana-dashboards index on elasticsearch logstash cluster |
[production] |
| 13:22 |
<gehel> |
restart elasticsearch on logstash1006 to test plugin removal - T174933 |
[production] |
| 12:26 |
<moritzm> |
uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org |
[production] |
| 12:18 |
<marostegui> |
Stop MySQL on db1037 as it is going to be decommissioned - T174902 |
[production] |
| 12:10 |
<elukey> |
rolling restart of zookeeper on conf100[123] for jvm security updates |
[production] |
| 11:53 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s) |
[production] |
| 11:52 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s) |
[production] |
| 11:45 |
<godog> |
restart varnish on cp1099 - mailbox lag |
[production] |
| 10:22 |
<ariel@tin> |
Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s) |
[production] |
| 10:22 |
<ariel@tin> |
Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression |
[production] |
| 09:51 |
<moritzm> |
uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org |
[production] |
| 09:41 |
<mobrovac@tin> |
Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update |
[production] |
| 09:36 |
<moritzm> |
uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org |
[production] |
| 08:52 |
<elukey> |
restart zookeeper on conf2003 for jvm security updates |
[production] |
| 08:48 |
<marostegui> |
Stop MySQL on db1045 as it will be decommissioned - T174806 |
[production] |
| 08:47 |
<moritzm> |
installing mercurial security updates |
[production] |
| 08:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s) |
[production] |
| 08:39 |
<moritzm> |
installing libgcrypt20 security updates |
[production] |
| 08:39 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s) |
[production] |
| 08:33 |
<godog> |
upload scap 3.7.0-1 to apt.w.o - T127762 |
[production] |
| 07:39 |
<moritzm> |
installing gnupg security updates |
[production] |
| 07:03 |
<marostegui> |
Deploy alter table on s3 codfw master with replication on kpbwiki - T168661 |
[production] |
| 07:02 |
<_joe_> |
starting additional runJobs instance for htmlcacheupdate on commons T173710 |
[production] |
| 04:52 |
<marostegui> |
Deploy alter table on db1052 - T168661 |
[production] |
| 02:44 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s) |
[production] |
| 02:37 |
<l10nupdate@tin> |
scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s) |
[production] |
|
2017-09-03
§
|
| 17:52 |
<elukey> |
depooled cp4024 (ulsfo upload) due to kernel errors in dmesg |
[production] |
| 15:57 |
<ema> |
restart varnish backend on cp1073 (mailbox lag) |
[production] |
| 09:41 |
<ema@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe |
[production] |
| 09:41 |
<ema@neodymium> |
conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx |
[production] |
| 09:25 |
<ema> |
cp4024 is down, powercycled |
[production] |
| 09:13 |
<_joe_> |
manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart |
[production] |
| 00:59 |
<legoktm> |
legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859 |
[production] |
|
2017-09-01
§
|
| 16:18 |
<bblack> |
restart varnish backend on cp1074 (mailbox lag) |
[production] |
| 15:15 |
<urandom> |
Restarting Cassandra: restbase-dev100[5-6]-{a,b} |
[production] |
| 15:06 |
<urandom> |
Restarting Cassandra: restbase-dev1004-{a,b} |
[production] |
| 15:00 |
<reedy@tin> |
Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s) |
[production] |
| 14:28 |
<marostegui> |
Add 150G to /srv partition on labsdb1001 |
[production] |
| 13:42 |
<marostegui> |
Rename table pr_index on enwiki on db1089 - T174782 |
[production] |
| 13:10 |
<marostegui> |
Stop MySQL on db1026 as it will be decommissioned - T174763 |
[production] |
| 12:32 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s) |
[production] |
| 12:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s) |
[production] |
| 12:17 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s) |
[production] |
| 11:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s) |
[production] |
| 11:23 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s) |
[production] |
| 10:35 |
<elukey> |
stop puppet on thorium and disable root rsyncs - T174756 |
[production] |
| 10:31 |
<marostegui> |
Upgrade MariaDB to 10.0.32 on db1081 - T168661 |
[production] |
| 10:29 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s) |
[production] |
| 09:10 |
<godog> |
bounce graphite-web on graphite1001, problematic query? using all CPU |
[production] |