|
2016-04-18
§
|
| 21:29 |
<papaul> |
conf200[1-3] - signing puppet certs, salt-key, initial run |
[production] |
| 20:58 |
<papaul> |
OS install on conf200[1-3] |
[production] |
| 20:55 |
<mutante> |
furud.codfw - initial puppet install, signing certs |
[production] |
| 18:43 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1005 (duration: 00m 28s) |
[production] |
| 18:42 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 26s) |
[production] |
| 18:37 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool pc2004 (duration: 00m 25s) |
[production] |
| 18:27 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool pc1004 (duration: 00m 36s) |
[production] |
| 17:31 |
<elukey> |
restarted kafkatee on oxygen after module upgrade (was stuck from this morning due to a kafka broker restart) |
[production] |
| 14:21 |
<ottomata> |
upgraded pykafka on hafnium and restarted statsv |
[production] |
| 14:04 |
<andrewbogott> |
puppet agent -tv on labvirt1002 which restarted nrpe and resolved icinga alerts |
[production] |
| 14:02 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool pc1004 (duration: 00m 29s) |
[production] |
| 14:01 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool pc2004 (duration: 00m 27s) |
[production] |
| 11:30 |
<mobrovac> |
restbase restarting eqiad after https://gerrit.wikimedia.org/r/#/c/283958/ |
[production] |
| 10:07 |
<_joe_> |
traffic from eqiad caches switched to codfw for restbase,citoid,cxserver |
[production] |
| 09:43 |
<moritzm> |
installing openssh updates on jessie systems |
[production] |
| 09:20 |
<_joe_> |
upgrading HHVM on terbium |
[production] |
| 09:19 |
<godog> |
depool restbase100[56].eqiad.wmnet, about to get decomissioned |
[production] |
| 09:17 |
<godog> |
repool restbase2006 after raid expansion |
[production] |
| 09:08 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Update master comments in preparation for dc failover (duration: 00m 26s) |
[production] |
| 09:07 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Update master comments in preparation for dc failover (duration: 01m 41s) |
[production] |
| 07:02 |
<jynus> |
changing binlog_format to MIXED on db2018 |
[production] |
|
2016-04-16
§
|
| 14:25 |
<andrewbogott> |
rebooting mw1132 — OOM |
[production] |
| 14:24 |
<andrewbogott> |
rebooting mw1134 — OOM |
[production] |
| 10:34 |
<jynus> |
recreating _counters table on s3 to solve replication issues |
[production] |
| 10:08 |
<jynus> |
recreating hitcounter table on s3 to solve replication issues |
[production] |
| 09:31 |
<volans> |
Skipped on dbstore1002 query from replica: 'DELETE FROM `bawiktionary`.`hitcounter`' |
[production] |
| 09:31 |
<jynus> |
importing sys schema to db1075 |
[production] |
| 06:44 |
<akosiaris> |
restart fermium to apply extra vcpus assignment |
[production] |
| 06:23 |
<volans> |
Stopped and restarted replica on dbstore2002 for s3 to "unstuck" the replica |
[production] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Apr 16 02:31:33 UTC 2016 (duration 9m 11s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.21) (duration: 09m 03s) |
[production] |
| 02:16 |
<urandom> |
Bootstraping restbase1009-a.eqiad.wmnet : T95253 |
[production] |
|
2016-04-15
§
|
| 22:28 |
<mutante> |
rebooting bast4001 - debugging install issue |
[production] |
| 21:04 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Repool new db1075,1077,1078 after TLS upgrade on s3 - T111654 (duration: 00m 36s) |
[production] |
| 16:08 |
<demon@tin> |
Synchronized wmf-config/CommonSettings.php: more liberal initialisesettings invalidation (duration: 00m 42s) |
[production] |
| 15:34 |
<godog> |
remove cassandra metrics for restbase100[1234]* restbase100[789] restbase2004 - T132771 |
[production] |
| 14:40 |
<gehel> |
reindexing all wikis after the switch to codfw (T132762) |
[production] |
| 14:06 |
<urandom> |
start decommission of restbase1009-a.eqiad.wmnet : T95253 |
[production] |
| 12:15 |
<gehel> |
increase max request size for elasticsearch |
[production] |
| 12:14 |
<volans> |
Re-arrange s3 replica topology: making codfw replicate from db1075 (this time for real) - T111654 |
[production] |
| 12:08 |
<bblack> |
experimenting on carbon HTTP config (apt/mirrors/ubuntu.wm.o) - watch out for installer / package-update issues! |
[production] |
| 12:00 |
<gehel> |
remove maintenance from wdqs1002 |
[production] |
| 10:50 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Depool new db1075,1077,1078 to upgrade TLS on s3 - T111654 (duration: 00m 41s) |
[production] |
| 10:37 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1042 after TLS upgrade on s4 - T111654 (duration: 00m 30s) |
[production] |
| 10:21 |
<_joe_> |
restarted zuul, zuul-merger on gallium |
[production] |
| 10:10 |
<TimStarling> |
on helium: scheduled restore of home_pmtpa to bast4001 |
[production] |
| 09:23 |
<volans> |
Re-arrange s3 replica topology: making codfw replicate from db1075 - T111654 |
[production] |