|
2016-02-18
§
|
| 12:21 |
<godog> |
expand raid0 on restbase1008 to sdd and sde |
[production] |
| 11:36 |
<paravoid> |
upgrading mr1-ulsfo to its pre-recovery version and rebooting (T127295) |
[production] |
| 11:34 |
<hashar> |
Hard restarting Jenkins T127294 |
[production] |
| 11:32 |
<jynus> |
logical import of db1021 starting for data consistency check and defragmenting purposes |
[production] |
| 11:29 |
<paravoid> |
mr1-ulsfo: "request system snapshot media internal slice alternate" + reboot (T127295) |
[production] |
| 11:27 |
<hashar> |
Jenkins web UI busy with 'jenkins.model.RunIdMigrator doMigrate' while it migrate build records. I did a bunch of cleanup yesterday. Jenkins runs jobs in the background just fine though. T127294 |
[production] |
| 11:12 |
<hashar> |
Jenkins: reloading configuration from disk. Some metadata are corrupted T127294 |
[production] |
| 10:48 |
<elukey> |
rebooted kafka1018 for maintenance |
[production] |
| 10:17 |
<elukey> |
rebooted kafka1014 for maintenance |
[production] |
| 10:10 |
<moritzm> |
restarting hhvm on mw1* to put glibc update into effect |
[production] |
| 09:49 |
<godog> |
remove old restbase metrics under restbase.* from graphite1001 and graphite2001 |
[production] |
| 03:13 |
<twentyafterfour> |
running puppet one last time on iridium. Phabricator upgrade successful with just a few minor issues now resolved. |
[production] |
| 03:01 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Feb 18 03:01:01 UTC 2016 (duration 9m 24s) |
[production] |
| 02:51 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 11m 20s) |
[production] |
| 02:29 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 13m 55s) |
[production] |
| 02:18 |
<twentyafterfour> |
phabricator is back online, sprint extension is broken, I'm investigating |
[production] |
| 01:57 |
<mutante> |
powercycled frozen mw1147 |
[production] |
| 01:51 |
<twentyafterfour> |
phab pre-upgrade: http://pastebin.com/RTmXfDhp |
[production] |
| 01:49 |
<twentyafterfour> |
about to bring down phabricator to do the upgrade |
[production] |
| 01:49 |
<twentyafterfour> |
ran puppet on iridium for testing |
[production] |
| 01:08 |
<twentyafterfour> |
stopped phd and started dumping phabricator's database to /srv/dumps/20160218.phabricator.sql.gz (just in case I need to roll back the update) |
[production] |
| 00:34 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.13/extensions/Flow: Trying again (duration: 01m 50s) |
[production] |
| 00:28 |
<RoanKattouw> |
00:28:25 64 apaches had sync errors , /usr/bin/sync-common missing |
[production] |
| 00:28 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.13/extensions/Flow: SWAT (duration: 02m 06s) |
[production] |
| 00:18 |
<godog> |
restart cassandra-a on restbase1008 after extending /srv |
[production] |
|
2016-02-17
§
|
| 23:53 |
<csteipp> |
redeployed wmf14 patches |
[production] |
| 23:30 |
<csteipp> |
deployed all missing security patches from wmf14 |
[production] |
| 23:10 |
<csteipp@tin> |
Synchronized php-1.27.0-wmf.14/resources/src/mediawiki/page/patrol.ajax.js: add security patches (duration: 01m 28s) |
[production] |
| 23:08 |
<csteipp@tin> |
Synchronized php-1.27.0-wmf.14/includes: add security patches (duration: 01m 35s) |
[production] |
| 23:03 |
<ori@mira> |
Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: live-hacked debug logging for T124356 (duration: 02m 16s) |
[production] |
| 21:42 |
<mobrovac> |
mathoid deploying ed98ffe9d |
[production] |
| 21:35 |
<mobrovac> |
restbase restarted restbase1002 on nodejs v4.3.0 |
[production] |
| 20:40 |
<papaul> |
es201[1-9] - signing puppet certs, salt-key, initial run |
[production] |
| 20:25 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: Re-enable T99096 for mediawiki.org (duration: 01m 29s) |
[production] |
| 20:23 |
<catrope@tin> |
Synchronized docroot/: (no message) (duration: 01m 33s) |
[production] |
| 19:18 |
<yuvipanda> |
truncate 1.2T php error log file on labstore1003 from cluebot |
[production] |
| 18:35 |
<jynus> |
testing now that alerts still work by stopping db1024 replication (depooled) |
[production] |
| 18:30 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: T127194 (duration: 01m 31s) |
[production] |
| 18:27 |
<jynus> |
no issues found with new mysql, lag monitoring, renabling puppet again on the pending eqiad servers |
[production] |
| 17:49 |
<bblack> |
restarting pybal on eqiad primary LVS ( lvs100[123] ) |
[production] |
| 17:47 |
<bblack> |
restarting pybal on codfw primary LVS ( lvs200[123]) |
[production] |
| 17:42 |
<bblack> |
restarting pybal on ulsfo/esams primary LVS ( lvs[34]00[12]) |
[production] |
| 17:40 |
<bblack> |
restarting pybal on eqiad backup LVS ( lvs100[456] ) |
[production] |
| 17:38 |
<bblack> |
restarting pybal on eqiad inactive LVS clusters ( lvs1007-12 ) |
[production] |
| 17:38 |
<bblack> |
restarting pybal on codfw backup LVS ( lvs200[456] ) |
[production] |
| 17:34 |
<bblack> |
restarting pybal on ulsfo/esams backup LVS ( lvs[34]00[34]) |
[production] |
| 17:13 |
<hoo> |
Updated the sites and site_identifiers table for on all non-Wikipedias (including Wikidata) |
[production] |
| 17:02 |
<ema> |
depooled ulsfo https://phabricator.wikimedia.org/T127094 |
[production] |
| 16:48 |
<ostriches> |
purged ancient boardvote gpg key from mediawiki fleet. unused since forever. |
[production] |
| 16:25 |
<anomie@tin> |
Synchronized wmf-config/: SWAT: Undeploy Extension:ApiSandbox (duration: 01m 30s) |
[production] |