|
2016-06-02
ยง
|
| 18:47 |
<jynus> |
restarting replication on db1016 |
[production] |
| 18:43 |
<YuviPanda> |
powercycle labmon1001 again, get into bios |
[production] |
| 18:29 |
<YuviPanda> |
going to try to intentionally trip the NFS check on tools-checker. This will not page |
[production] |
| 18:24 |
<YuviPanda> |
powercycle labmon1001 again |
[production] |
| 18:19 |
<mutante> |
db2007, revoke puppet cert, delete salt key, nuke from stored configs / icinga |
[production] |
| 18:19 |
<bearND> |
mobileapps deployed b2fee30 |
[production] |
| 18:18 |
<mutante> |
db2007 shutdown, schedule eternal downtime |
[production] |
| 18:04 |
<bearND> |
starting mobileapps deploy |
[production] |
| 17:40 |
<subbu> |
finished deploying parsoid version 7188080b |
[production] |
| 17:34 |
<subbu> |
synced new code; restarted parsoid on wtp1001 as a canary |
[production] |
| 17:29 |
<subbu> |
starting deploy of new parsoid code |
[production] |
| 17:21 |
<mutante> |
ran ALTER TABLE character set utf8 .. (https://phabricator.wikimedia.org/T119112#2311402) on RT db |
[production] |
| 17:16 |
<mutante> |
running RT database upgrade from 4.0.4 to 4.2.8 |
[production] |
| 17:13 |
<awight> |
update paymentswiki from d26426c4225080c95f0bd5a6a31c54e4826287b1 to de86eadcd98922ee4207a0c46112585f3ba5c48d |
[production] |
| 17:05 |
<mutante> |
stopped exim on magnesium |
[production] |
| 17:05 |
<jynus> |
stopping replication from db1001 to db1016 (pasive m1 node) before schema change |
[production] |
| 16:52 |
<mutante> |
magnesium (RT), tmp. stopped RT and puppet |
[production] |
| 16:50 |
<YuviPanda> |
begin reinstall of labmon1001 |
[production] |
| 15:19 |
<thcipriani@tin> |
Synchronized php-1.28.0-wmf.4/extensions/Math: SWAT: [[gerrit:292344|Use img instead of meta tags for SVGs]] and [[gerrit:292340|Fix iterator in batchGetMathML]] (duration: 00m 28s) |
[production] |
| 15:12 |
<thcipriani@tin> |
Synchronized portals: [[gerrit:292358|deploying new localized top-links on wikipedia.org]] (duration: 00m 31s) |
[production] |
| 15:11 |
<thcipriani@tin> |
Synchronized portals/prod/wikipedia.org/assets: [[gerrit:292358|deploying new localized top-links on wikipedia.org]] (duration: 00m 32s) |
[production] |
| 14:33 |
<jynus> |
acked ores icinga checks on some scb hosts and pointing to T124201 (it seems the checks arrived before the actual setup) |
[production] |
| 13:52 |
<moritzm> |
installing imagemagick security updates on Ubuntu systems (but affected decoders already neutralised by policy changes) (also Debian systems already addressed) |
[production] |
| 13:34 |
<hashar> |
Downgrading Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb . Paramiko cant acquire ssh connection with Gerrit for some reason... https://phabricator.wikimedia.org/P3204 |
[production] |
| 12:10 |
<hashar> |
Upgraded Zuul upstream code being 66c8e52..30a433b package is 2.1.0-151-g30a433b-wmf1precise1 |
[production] |
| 11:39 |
<jmm@tin> |
Synchronized wmf-config/CommonSettings.php: disable firejail security hardening for image scalers, needs more work for the Score extension (duration: 00m 36s) |
[production] |
| 10:55 |
<hashar> |
Restarted Zuul and reenabled puppet on gallium |
[production] |
| 10:50 |
<hashar> |
gallium: stopped puppet agent |
[production] |
| 10:49 |
<hashar> |
gracefully stopping Zuul, will upgrade / take traces etc over the next half hour or so |
[production] |
| 10:14 |
<jynus> |
archiving again syslog.1 from ms-be2012 on /srv/swift-storage/sdl1/tmp |
[production] |
| 10:08 |
<mobrovac> |
restbase enabling puppet back in production |
[production] |
| 08:40 |
<mobrovac> |
restbase deploy end of 19f25925 |
[production] |
| 08:29 |
<mobrovac> |
restbase deploy start of 19f25925 |
[production] |
| 08:09 |
<mobrovac> |
restbase disabling puppet in production for testing https://gerrit.wikimedia.org/r/#/c/292109/ in staging |
[production] |
| 07:23 |
<moritzm> |
rebooting etherpad1001 (hosting etherpad.wikimedia.org) for upgrade to Linux 4.4 |
[production] |
| 07:02 |
<jynus> |
performing schema change for db1057 |
[production] |
| 03:04 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Jun 2 03:04:44 UTC 2016 (duration 6m 40s) |
[production] |
| 02:58 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.4) (duration: 15m 37s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.3) (duration: 10m 06s) |
[production] |
| 01:52 |
<mutante> |
scb1001/2001 ores - connection refused |
[production] |
| 01:52 |
<mutante> |
mw1136 service hhvm restart |
[production] |
| 01:37 |
<mutante> |
labsdb1001 /etc/init.d/mysql start |
[production] |
| 01:31 |
<YuviPanda> |
service mysql start on labsdb1001 |
[production] |
| 01:25 |
<dereckson@tin> |
Synchronized wmf-config/CommonSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 22s) |
[production] |
| 01:25 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 23s) |
[production] |
| 01:23 |
<YuviPanda> |
reboot labsdb1001 |
[production] |
| 01:21 |
<dereckson@tin> |
Synchronized php-1.28.0-wmf.4/extensions/Flow/handlebars/: HACK: Hide reply form for locked topics (T135848) (duration: 00m 24s) |
[production] |
| 01:19 |
<dereckson@tin> |
Synchronized php-1.28.0-wmf.4/extensions/Echo/includes/special/NotificationPager.php: Fix notification pager (T136759) (duration: 00m 25s) |
[production] |
| 01:18 |
<YuviPanda> |
restart mysql on labsdb1001 |
[production] |
| 01:00 |
<bearND> |
mobileapps reverted to 8d6d648c943074b7d3999baf31d60ad99249cd51 |
[production] |