|
2016-02-17
ยง
|
| 19:18 |
<yuvipanda> |
truncate 1.2T php error log file on labstore1003 from cluebot |
[production] |
| 18:35 |
<jynus> |
testing now that alerts still work by stopping db1024 replication (depooled) |
[production] |
| 18:30 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: T127194 (duration: 01m 31s) |
[production] |
| 18:27 |
<jynus> |
no issues found with new mysql, lag monitoring, renabling puppet again on the pending eqiad servers |
[production] |
| 17:49 |
<bblack> |
restarting pybal on eqiad primary LVS ( lvs100[123] ) |
[production] |
| 17:47 |
<bblack> |
restarting pybal on codfw primary LVS ( lvs200[123]) |
[production] |
| 17:42 |
<bblack> |
restarting pybal on ulsfo/esams primary LVS ( lvs[34]00[12]) |
[production] |
| 17:40 |
<bblack> |
restarting pybal on eqiad backup LVS ( lvs100[456] ) |
[production] |
| 17:38 |
<bblack> |
restarting pybal on eqiad inactive LVS clusters ( lvs1007-12 ) |
[production] |
| 17:38 |
<bblack> |
restarting pybal on codfw backup LVS ( lvs200[456] ) |
[production] |
| 17:34 |
<bblack> |
restarting pybal on ulsfo/esams backup LVS ( lvs[34]00[34]) |
[production] |
| 17:13 |
<hoo> |
Updated the sites and site_identifiers table for on all non-Wikipedias (including Wikidata) |
[production] |
| 17:02 |
<ema> |
depooled ulsfo https://phabricator.wikimedia.org/T127094 |
[production] |
| 16:48 |
<ostriches> |
purged ancient boardvote gpg key from mediawiki fleet. unused since forever. |
[production] |
| 16:25 |
<anomie@tin> |
Synchronized wmf-config/: SWAT: Undeploy Extension:ApiSandbox (duration: 01m 30s) |
[production] |
| 16:20 |
<anomie@tin> |
Synchronized wmf-config/CommonSettings.php: SWAT: Remove $wgMWOAuthGrantPermissions (duration: 01m 34s) |
[production] |
| 16:16 |
<urandom> |
restbase deploy (15a6c50) complete, sans restbase1008.eqiad.wmnet (down for maintenance during deploy) |
[production] |
| 16:16 |
<anomie> |
Ran namespaceDupes.php on tawiki |
[production] |
| 16:14 |
<urandom> |
restbase deploy (15a6c50) completet |
[production] |
| 16:14 |
<hoo> |
Re-populating the sites and site_identifiers table for all Wikipedias and testwikidata |
[production] |
| 16:10 |
<urandom> |
restbase deploy restarting at restbase1009 |
[production] |
| 16:09 |
<anomie@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: New userrights and configuration for cswiki (task [[phab:T126931|]]) (duration: 01m 31s) |
[production] |
| 16:09 |
<urandom> |
restbase deploy stalled at restbase1008 (under maintenance) |
[production] |
| 16:05 |
<anomie@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace aliases for tawiki (task [[phab:T126604|]]) (duration: 01m 31s) |
[production] |
| 16:00 |
<urandom> |
continuing production-wide restbase deploy (15a6c50) |
[production] |
| 15:58 |
<godog> |
copy restbase.{v1_*,sys_*,ALL,GET,HEAD,POST,OPTIONS,_robots} to restbase.external on graphite1001 and graphite2001 |
[production] |
| 15:55 |
<godog> |
copy restbase.private to restbase.internal on graphite1001 and graphite2001 |
[production] |
| 15:53 |
<aude@tin> |
Synchronized php-1.27.0-wmf.13/extensions/Wikidata: Fix caching data types bug: T127095 (duration: 01m 44s) |
[production] |
| 15:53 |
<urandom> |
canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50) complete |
[production] |
| 15:53 |
<urandom> |
canary deploy of restbase to restbase1001.eqiad.wmnet (15a6c50) |
[production] |
| 15:51 |
<bblack> |
package upgrades commencing on lvs* |
[production] |
| 15:43 |
<urandom> |
restbase staging deploy (15a6c50) complete |
[production] |
| 15:38 |
<urandom> |
deploying restbase (15a6c50) in staging |
[production] |
| 15:33 |
<jynus> |
stopping puppet on all database hosts (db, dbstore, es, etc.) for lag alert testing |
[production] |
| 14:05 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1022 after maintenance (duration: 01m 34s) |
[production] |
| 14:02 |
<bblack> |
package upgrades on cp* commence |
[production] |
| 13:23 |
<elukey> |
rebooted kafka1013 for maintenance |
[production] |
| 13:06 |
<elukey> |
stopping kafka on kafka1013 |
[production] |
| 12:51 |
<moritzm> |
rolling restart of ms-fe2* |
[production] |
| 12:31 |
<moritzm> |
restarting apache on krypton for glibc update |
[production] |
| 12:00 |
<godog> |
bump stripe_cache_size to 10240 for md2 on restbase1008 |
[production] |
| 11:54 |
<jynus> |
restarting schema change on wikidatawiki (s5) T62539 |
[production] |
| 11:47 |
<moritzm> |
restarting hhvm on mw2* to put glibc update into effect |
[production] |
| 11:29 |
<jynus> |
rolling schema change on wikidatawiki (s5) |
[production] |
| 10:40 |
<jynus> |
refreshed dns server config |
[production] |
| 09:57 |
<godog> |
stop cassandra-a on restbase1008 for raid expansion |
[production] |
| 09:55 |
<godog> |
depool restbase1008 for raid expansion T119935 |
[production] |
| 08:19 |
<akosiaris> |
gnt-instance reboot alsafi.wikimedia.org |
[production] |
| 08:19 |
<akosiaris> |
enabled puppet on sca1001, sca1002 |
[production] |
| 03:15 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Feb 17 03:15:40 UTC 2016 (duration 9m 30s) |
[production] |