|
2016-02-15
§
|
| 22:54 |
<andrewbogott> |
Andrew encourages future puppet hackers to commit their local changes so that this mess doesn’t reappear |
[phabricator] |
| 22:52 |
<andrewbogott> |
stashed local changes in /var/lib/git/operations/puppet on deploy instance, checked out ‘production’ branch, updated |
[phabricator] |
| 22:32 |
<andrewbogott> |
rebooting instance phab-02 |
[phabricator] |
| 22:29 |
<andrewbogott> |
rebooting instance phab-01 |
[phabricator] |
| 21:26 |
<krinkle@tin> |
Synchronized w/static.php: (no message) (duration: 00m 58s) |
[production] |
| 20:55 |
<twentyafterfour> |
moved contents of /srv/phab/dumps into /srv/dumps |
[production] |
| 20:48 |
<twentyafterfour> |
backing up /srv/phab to /srv/phab.bak in case it needs to be restored in a hurry. Note: the backup excludes /srv/phab/repos which is backed up separately |
[production] |
| 20:47 |
<twentyafterfour> |
brought iridium up to date with a current puppet run and checked that all repositories are at the correct tag |
[production] |
| 20:27 |
<bd808> |
Updated to 3314de8; Guard against %D\d matching Pabricator echo; De-dup Phabricator echos |
[tools.stashbot] |
| 18:23 |
<volans> |
Upgrading db1022 (MariaDB and kernel) already depooled and put in scheduled downtime |
[production] |
| 16:46 |
<yurik> |
Deployed patch for T126897 |
[production] |
| 16:38 |
<volans@tin> |
Synchronized wmf-config/db-eqiad.php: depool db1022 (duration: 00m 58s) |
[production] |
| 16:34 |
<bd808> |
Ran sync-common on mw1119 after stuck scap update was killed there |
[production] |
| 16:29 |
<bd808> |
Killed stuck rsync by mw1119 that was keeping l10nudpate running |
[production] |
| 16:28 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.13) (duration: 855m 18s) |
[production] |
| 16:28 |
<jzerebecki> |
reloading zuul for 2d16ad3..3bb0afa |
[releng] |
| 16:27 |
<moritzm> |
installing postgres security updates on maps* |
[production] |
| 16:26 |
<halfak> |
restarted workers on ores-worker -02 and -03 |
[ores] |
| 16:10 |
<hashar> |
Image ci-jessie-wikimedia-1455552377 in wmflabs-eqiad is ready |
[releng] |
| 15:32 |
<_joe_> |
restarting the salt minions on all deployment targets |
[production] |
| 15:25 |
<jzerebecki> |
reloading zuul for e174335..2d16ad3 |
[releng] |
| 15:23 |
<hashar> |
Image ci-jessie-wikimedia-1455549539 in wmflabs-eqiad is ready |
[releng] |
| 15:22 |
<apergos> |
disabled puppet again on iridium, had to restore sshd config from filebucket after puppet run |
[production] |
| 15:19 |
<hashar> |
Regenerating Nodepool snapshot. Slave scripts have 0 bytes... |
[releng] |
| 15:18 |
<hoo|busy> |
Changed email for global account "Frau pomerenke" |
[production] |
| 15:04 |
<hashar> |
Slave scripts added to Nodepool instances! Image ci-jessie-wikimedia-1455548346 in wmflabs-eqiad is ready |
[releng] |
| 14:51 |
<apergos> |
phab role commented out on iridium and puppet re-enabled, no phab changes will be applied there |
[production] |
| 14:24 |
<godog> |
start restbase1007-b cassandra instance, bootstrapping T119935 |
[production] |
| 13:22 |
<joal> |
Failed deploying refinery: deploy from tin failed, need ottomata help |
[analytics] |
| 13:04 |
<joal> |
Deploying refinery |
[analytics] |
| 12:30 |
<_joe_> |
deployment master is now tin |
[production] |
| 11:57 |
<_joe_> |
switching the deployment host back to tin |
[production] |
| 11:48 |
<_joe_> |
restarted hhvm on mw1130, memory exhausted |
[production] |
| 11:28 |
<jynus> |
offlining disk 32:5 on db1063 |
[production] |
| 11:05 |
<hashar> |
Image ci-jessie-wikimedia-1455534001 in wmflabs-eqiad is ready |
[releng] |
| 10:58 |
<hoo@mira> |
Synchronized wmf-config/throttle.php: Throttle exception for [[de:WP:Wikimedia Deutschland/WPFF Berlinale2016]] (duration: 01m 28s) |
[production] |
| 10:57 |
<hashar> |
bumped Nodepool disk image to fix apt configuration / include jre and package-config. Refreshing snapshot image. Poke T126246 T126230 T119143 |
[releng] |
| 09:38 |
<godog> |
swift codfw-prod: ms-be2020 / ms-be2021 weight to 3500 |
[production] |
| 09:16 |
<gehel> |
re-enabling puppet on elastic05 (https://phabricator.wikimedia.org/T126891) |
[deployment-prep] |
| 09:09 |
<_joe_> |
powercycled mw1131, OOM'd |
[production] |
| 07:52 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270686 |
[releng] |
| 06:52 |
<legoktm> |
legoktm@gallium:/srv/org/wikimedia/doc$ sudo -u jenkins-slave rm -rf EventLogging/ GuidedTour/ MultimediaViewer/ TemplateData/ |
[releng] |
| 06:22 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270677 |
[releng] |
| 06:12 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270675 |
[releng] |
| 06:02 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270674 |
[releng] |
| 05:56 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270673 |
[releng] |
| 05:32 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270670 |
[releng] |
| 04:05 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/270667 |
[releng] |
| 04:01 |
<halfak> |
Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone. |
[ores] |
| 03:59 |
<halfak> |
Restarted the uwsgi-ores-web service on ores-web-02 |
[ores] |