2016-12-21
§
|
13:03 |
<mobrovac@tin> |
Finished deploy [parsoid/deploy@dab1f27]: Bug fix for mwApiServer T153797 (duration: 05m 32s) |
[production] |
12:57 |
<mobrovac@tin> |
Starting deploy [parsoid/deploy@dab1f27]: Bug fix for mwApiServer T153797 |
[production] |
12:46 |
<moritzm> |
install openjdk-6 security update on labsdb1006 |
[production] |
10:43 |
<jynus> |
dropping non-wiki databases from labsdb1001 |
[production] |
10:01 |
<moritzm> |
installing libgme security updates |
[production] |
09:58 |
<jynus> |
extending db1035 /srv partition |
[production] |
08:42 |
<elukey> |
restarted hhvm/jobrunner (and killed ffmpeg processes) on mw116[89] |
[production] |
08:01 |
<marostegui> |
Running optimize table on db1044 for the pagelinks tables as we urgently need some space back on that host - T153826 |
[production] |
07:20 |
<marostegui> |
Running optimize table on db1045 for the revision tables as we urgently need some space back on that host - https://phabricator.wikimedia.org/T153739 |
[production] |
04:36 |
<Niharika> |
commtech Added samwilson as project admin |
[production] |
03:03 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1169.eqiad.wmnet |
[production] |
02:51 |
<mutante> |
relforge1001 has huge /var/log/elastichsearch/relforge-eqiad_feature.log that wrote GBs just today but then stopped |
[production] |
02:23 |
<mutante> |
mw1169 - reinstall done - sign new puppet cert, initial run... |
[production] |
02:20 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 07m 40s) |
[production] |
02:12 |
<mutante> |
mw1169 - delete salt key, revoke puppet cert |
[production] |
02:06 |
<mutante> |
reinstalling mw1169 (carbon DHCP, install1001 TFTP) |
[production] |
02:02 |
<mutante> |
re-enabling DHCP and puppet |
[production] |
01:49 |
<mutante> |
carbon - temp stop DHCP service to test install from install1001 |
[production] |
01:47 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1169.eqiad.wmnet |
[production] |
01:47 |
<mutante> |
mw1169 - schedule 2 hours downtime - boot for reinstall shortly |
[production] |
01:11 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1168.eqiad.wmnet |
[production] |
01:09 |
<mutante> |
mw1168 - remove old salt key, accept new salt key, start minion |
[production] |
2016-12-20
§
|
23:46 |
<mutante> |
mw1168 - re-signing new puppet cert after install, initial puppet run |
[production] |
23:18 |
<mutante> |
rebooting mw1168 into PXE for reinstall |
[production] |
23:09 |
<mutante> |
install1001/2001 - upgrade python-requests python-urllib3 arcconf |
[production] |
21:48 |
<mutante> |
depooled mw1168 videoscaler for reinstall (part of T153488) |
[production] |
21:47 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1168.eqiad.wmnet |
[production] |
21:43 |
<demon@tin> |
Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s) |
[production] |
21:42 |
<demon@tin> |
Synchronized multiversion/submodules.json: no-op scap stuff (duration: 00m 39s) |
[production] |
21:08 |
<eileen1> |
disable Name match only job as we are caught up now |
[production] |
21:05 |
<eileen1> |
disable Major gifts dedupe jobs. The main job has caught up to the end of the queue now so we don't need to target these separately |
[production] |
20:55 |
<mobrovac@tin> |
Finished deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase (duration: 00m 50s) |
[production] |
20:54 |
<mobrovac@tin> |
Starting deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase |
[production] |
20:25 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove dumb comment (duration: 00m 39s) |
[production] |
20:21 |
<demon@tin> |
Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s) |
[production] |
20:21 |
<demon@tin> |
Synchronized tox.ini: no-op scap stuff (duration: 00m 39s) |
[production] |
20:20 |
<demon@tin> |
Synchronized test-requirements.txt: no-op scap stuff (duration: 00m 39s) |
[production] |
20:19 |
<demon@tin> |
Synchronized setup.py: no-op scap stuff (duration: 00m 39s) |
[production] |
20:19 |
<demon@tin> |
Synchronized requirements.txt: no-op scap stuff (duration: 00m 39s) |
[production] |
20:13 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: No-op comment-only (duration: 00m 52s) |
[production] |
17:50 |
<volans> |
rcs1002.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1 |
[production] |
17:41 |
<ema> |
rcs1001.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1 |
[production] |
17:24 |
<reedy@tin> |
Synchronized wmf-config/throttle.php: T153741 (duration: 00m 40s) |
[production] |
16:31 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=restbase,dc=eqiad,name=restbase101[6-8].* |
[production] |
16:30 |
<bblack> |
upgrading tons of outdated OS packages on rcs100x |
[production] |
16:29 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=restbase,dc=codfw,name=restbase201.* |
[production] |
16:26 |
<marostegui> |
Running optimize table on db1045 for the revision tables as we urgently need some space back on that host - T153739 |
[production] |
16:19 |
<mutante> |
notebook1001/1002 - upgraded php packages |
[production] |
16:17 |
<jynus> |
drop repl_records_partition events from db1069 |
[production] |
16:09 |
<mobrovac> |
change-prop restarting everywhere |
[production] |