|
2016-02-16
§
|
| 11:49 |
<hoo@tin> |
Finished scap: Deploy Capiunto (master) to testwiki, test2wiki and testwikidata - T126399 (duration: 24m 46s) |
[production] |
| 11:46 |
<paravoid> |
upgrading pfw-codfw to newer JunOS |
[production] |
| 11:25 |
<hoo@tin> |
Started scap: Deploy Capiunto (master) to testwiki, test2wiki and testwikidata - T126399 |
[production] |
| 11:08 |
<godog> |
repool restbase on restbase1007 |
[production] |
| 11:01 |
<moritzm> |
installing nettle security updates |
[production] |
| 10:50 |
<godog> |
start swiftrepl commons thumbs for top50 popular size T125791 |
[production] |
| 10:14 |
<akosiaris> |
disable puppet, stop salt-minion on sca100{1,2} |
[production] |
| 09:52 |
<hashar> |
will cut the wmf branches this afternoon starting around 14:00 CET |
[production] |
| 09:31 |
<hoo@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 58s) |
[production] |
| 02:34 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Feb 16 02:34:30 UTC 2016 (duration 8m 34s) |
[production] |
| 02:25 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 12m 28s) |
[production] |
|
2016-02-15
§
|
| 21:26 |
<krinkle@tin> |
Synchronized w/static.php: (no message) (duration: 00m 58s) |
[production] |
| 20:55 |
<twentyafterfour> |
moved contents of /srv/phab/dumps into /srv/dumps |
[production] |
| 20:48 |
<twentyafterfour> |
backing up /srv/phab to /srv/phab.bak in case it needs to be restored in a hurry. Note: the backup excludes /srv/phab/repos which is backed up separately |
[production] |
| 20:47 |
<twentyafterfour> |
brought iridium up to date with a current puppet run and checked that all repositories are at the correct tag |
[production] |
| 18:23 |
<volans> |
Upgrading db1022 (MariaDB and kernel) already depooled and put in scheduled downtime |
[production] |
| 16:46 |
<yurik> |
Deployed patch for T126897 |
[production] |
| 16:38 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: depool db1022 (duration: 00m 58s) |
[production] |
| 16:34 |
<bd808> |
Ran sync-common on mw1119 after stuck scap update was killed there |
[production] |
| 16:29 |
<bd808> |
Killed stuck rsync by mw1119 that was keeping l10nudpate running |
[production] |
| 16:28 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 855m 18s) |
[production] |
| 16:27 |
<moritzm> |
installing postgres security updates on maps* |
[production] |
| 15:32 |
<_joe_> |
restarting the salt minions on all deployment targets |
[production] |
| 15:22 |
<apergos> |
disabled puppet again on iridium, had to restore sshd config from filebucket after puppet run |
[production] |
| 15:18 |
<hoo|busy> |
Changed email for global account "Frau pomerenke" |
[production] |
| 14:51 |
<apergos> |
phab role commented out on iridium and puppet re-enabled, no phab changes will be applied there |
[production] |
| 14:24 |
<godog> |
start restbase1007-b cassandra instance, bootstrapping T119935 |
[production] |
| 12:30 |
<_joe_> |
deployment master is now tin |
[production] |
| 11:57 |
<_joe_> |
switching the deployment host back to tin |
[production] |
| 11:48 |
<_joe_> |
restarted hhvm on mw1130, memory exhausted |
[production] |
| 11:28 |
<jynus> |
offlining disk 32:5 on db1063 |
[production] |
| 10:58 |
<hoo@mira> |
Synchronized wmf-config/throttle.php: Throttle exception for [[de:WP:Wikimedia Deutschland/WPFF Berlinale2016]] (duration: 01m 28s) |
[production] |
| 09:38 |
<godog> |
swift codfw-prod: ms-be2020 / ms-be2021 weight to 3500 |
[production] |
| 09:09 |
<_joe_> |
powercycled mw1131, OOM'd |
[production] |
|
2016-02-14
§
|
| 23:18 |
<jynus> |
repaired table and restarted replication on sanitarium (s3) |
[production] |
| 23:06 |
<andrewbogott> |
moved tools dbs off of labsdb1002 via https://gerrit.wikimedia.org/r/#/c/270650/ |
[production] |
| 22:40 |
<_joe_> |
labsdb lvm partition /srv unavailable because /dev/sdc is apparently broken |
[production] |
| 16:55 |
<_joe_> |
powercycling mw1140, stuck in OOM, no ssh no console login |
[production] |
| 13:57 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: vslow, dump to db1054; do not send api to db1054; 1036 only rc (duration: 01m 17s) |
[production] |
| 13:14 |
<hoo@mira> |
Synchronized wmf-config/db-eqiad.php: Depool db1021 (duration: 01m 26s) |
[production] |
| 02:40 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Feb 14 02:40:27 UTC 2016 (duration 9m 8s) |
[production] |
| 02:31 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 12m 01s) |
[production] |