|
2016-08-25
§
|
| 13:09 |
<moritzm> |
puppet re-enabled on all kafka* hosts |
[production] |
| 12:00 |
<moritzm> |
temporarily disabling puppet on kafka* hosts (to enable ferm changes in controlled stages) |
[production] |
| 10:00 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe']) |
[production] |
| 10:00 |
<ema@palladium> |
conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx']) |
[production] |
| 09:51 |
<moritzm> |
temporarily stop irc bot, until puppet has self-healed |
[production] |
| 08:17 |
<moritzm> |
restarted hhvm on mw1191 and mw1216, got stuck |
[production] |
| 07:07 |
<moritzm> |
installing harfbuzz security updates |
[production] |
| 03:28 |
<tgr@tin> |
Synchronized php-1.28.0-wmf.16/includes/specialpage/AuthManagerSpecialPage.php: UBN fix for T143840 (duration: 00m 49s) |
[production] |
| 02:55 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Aug 25 02:55:50 UTC 2016 (duration 6m 30s) |
[production] |
| 02:49 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 09s) |
[production] |
| 02:27 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 41s) |
[production] |
| 01:49 |
<mutante> |
iridium - temp. stopping phd service for home dir change |
[production] |
| 01:08 |
<mutante> |
radium apt-get autoremove; apt-get upgrade (openssh, openssl, passwd, sudo, libpam, libc6 :) |
[production] |
| 00:51 |
<urandom> |
T137474: Stopping dumps in RESTBase staging, and reverting xenon.eqiad.wmnet to Cassandra 2.2.6-wmf1 |
[production] |
| 00:32 |
<ebernhardson@tin> |
Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/Job/CheckerJob.php: Fix CirrusSearch CheckerJob stuck in a loop (duration: 00m 47s) |
[production] |
| 00:08 |
<mutante> |
chromium back in service - both eqiad DNS recursors now on jessie |
[production] |
|
2016-08-24
§
|
| 23:43 |
<dzahn@palladium> |
conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org |
[production] |
| 23:39 |
<mutante> |
chromium - install ntpdate, stop ntp, sync time with hydrogen, start ntp, remove ntpdate |
[production] |
| 23:36 |
<dereckson@tin> |
Synchronized php-1.28.0-wmf.15/extensions/UploadWizard/resources: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 49s) |
[production] |
| 23:35 |
<dereckson@tin> |
Synchronized php-1.28.0-wmf.16/extensions/UploadWizard/resources/: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 47s) |
[production] |
| 23:33 |
<dereckson@tin> |
Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix ProofreadPage::updatePrIndex signature (T143817) (duration: 00m 50s) |
[production] |
| 22:59 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: labs-only change: https://gerrit.wikimedia.org/r/298919 (duration: 00m 46s) |
[production] |
| 22:58 |
<krenair@tin> |
Synchronized wmf-config/LabsServices.php: labs-only change: https://gerrit.wikimedia.org/r/305837 (duration: 00m 48s) |
[production] |
| 22:49 |
<mutante> |
chromium - revoking and re-signing puppet certs, salt keys, initial puppet run.. |
[production] |
| 22:43 |
<yuvipanda> |
force a puppet run on californium |
[production] |
| 22:27 |
<bd808> |
Deployed striker (2fdf103) to californium |
[production] |
| 22:20 |
<mutante> |
rebooting chromium into PXE |
[production] |
| 22:15 |
<papaul> |
installing puppetmaster200[1-2] |
[production] |
| 22:05 |
<mutante> |
stopping puppet and pdns-recursor on chromium |
[production] |
| 21:57 |
<krinkle@tin> |
Synchronized multiversion/MWMultiVersion.php: Ie9c568a87ae - No-op clean-up (duration: 00m 49s) |
[production] |
| 21:54 |
<yuvipanda> |
forcing puppet run on californium |
[production] |
| 21:50 |
<mutante> |
running puppet on lvs servers, removing chromium from resolv.conf for reinstall |
[production] |
| 21:45 |
<yuvipanda> |
re-enabled puppet, ran puppet and verified ORES is ok on scb1001 / 1002 |
[production] |
| 21:40 |
<yuvipanda> |
re-enabled puppet, ran puppet and verified ORES is ok on scb2001 / 2002 |
[production] |
| 21:27 |
<dzahn@palladium> |
conftool action : set/pooled=no; selector: name=chromium.wikimedia.org |
[production] |
| 21:27 |
<mutante> |
depooling chromium for reinstall. scheduled downtime for host and service IPs |
[production] |
| 21:25 |
<yuvipanda> |
disable puppet on scb2001 and 2002 as well |
[production] |
| 21:24 |
<yuvipanda> |
disable puppet on scb1001 and 1002 |
[production] |
| 21:20 |
<yuvipanda> |
forcing puppet run on tin |
[production] |
| 21:19 |
<yuvipanda> |
forcing puppet run on scb1001 |
[production] |
| 21:14 |
<bd808> |
Forced puppet run on logstash1003 but cron had beaten me to it |
[production] |
| 21:12 |
<bd808> |
Forcing puppet run on logstash1002 |
[production] |
| 21:08 |
<hoo> |
Ran DELETE FROM wbs_propertypairs WHERE pid1 = '641' on Wikidata for T132839 |
[production] |
| 21:07 |
<bd808> |
Forcing puppet run on logstash1001 |
[production] |
| 20:59 |
<urandom> |
T137474: Upgrading xenon.eqiad.wmnet to cassandra_2.2.6-wmf2 |
[production] |
| 20:57 |
<bd808> |
Created initial tables for striker in striker db on m5-master |
[production] |
| 20:53 |
<yuvipanda> |
gave bd808 password for striker db on terbium |
[production] |
| 20:39 |
<yurik> |
deployed kartotherian https://gerrit.wikimedia.org/r/#/c/306518/ |
[production] |
| 20:31 |
<yurik> |
deployed tilerator https://gerrit.wikimedia.org/r/#/c/306514/ |
[production] |
| 20:03 |
<urandom> |
T137474 Starting htmldumper in RESTBase Staging |
[production] |