|
2016-12-21
§
|
| 08:42 |
<elukey> |
restarted hhvm/jobrunner (and killed ffmpeg processes) on mw116[89] |
[production] |
| 08:01 |
<marostegui> |
Running optimize table on db1044 for the pagelinks tables as we urgently need some space back on that host - T153826 |
[production] |
| 07:20 |
<marostegui> |
Running optimize table on db1045 for the revision tables as we urgently need some space back on that host - https://phabricator.wikimedia.org/T153739 |
[production] |
| 04:36 |
<Niharika> |
commtech Added samwilson as project admin |
[production] |
| 03:03 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1169.eqiad.wmnet |
[production] |
| 02:51 |
<mutante> |
relforge1001 has huge /var/log/elastichsearch/relforge-eqiad_feature.log that wrote GBs just today but then stopped |
[production] |
| 02:23 |
<mutante> |
mw1169 - reinstall done - sign new puppet cert, initial run... |
[production] |
| 02:20 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 07m 40s) |
[production] |
| 02:12 |
<mutante> |
mw1169 - delete salt key, revoke puppet cert |
[production] |
| 02:06 |
<mutante> |
reinstalling mw1169 (carbon DHCP, install1001 TFTP) |
[production] |
| 02:02 |
<mutante> |
re-enabling DHCP and puppet |
[production] |
| 01:49 |
<mutante> |
carbon - temp stop DHCP service to test install from install1001 |
[production] |
| 01:47 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1169.eqiad.wmnet |
[production] |
| 01:47 |
<mutante> |
mw1169 - schedule 2 hours downtime - boot for reinstall shortly |
[production] |
| 01:11 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1168.eqiad.wmnet |
[production] |
| 01:09 |
<mutante> |
mw1168 - remove old salt key, accept new salt key, start minion |
[production] |
|
2016-12-20
§
|
| 23:46 |
<mutante> |
mw1168 - re-signing new puppet cert after install, initial puppet run |
[production] |
| 23:18 |
<mutante> |
rebooting mw1168 into PXE for reinstall |
[production] |
| 23:09 |
<mutante> |
install1001/2001 - upgrade python-requests python-urllib3 arcconf |
[production] |
| 21:48 |
<mutante> |
depooled mw1168 videoscaler for reinstall (part of T153488) |
[production] |
| 21:47 |
<dzahn@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1168.eqiad.wmnet |
[production] |
| 21:43 |
<demon@tin> |
Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s) |
[production] |
| 21:42 |
<demon@tin> |
Synchronized multiversion/submodules.json: no-op scap stuff (duration: 00m 39s) |
[production] |
| 21:08 |
<eileen1> |
disable Name match only job as we are caught up now |
[production] |
| 21:05 |
<eileen1> |
disable Major gifts dedupe jobs. The main job has caught up to the end of the queue now so we don't need to target these separately |
[production] |
| 20:55 |
<mobrovac@tin> |
Finished deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase (duration: 00m 50s) |
[production] |
| 20:54 |
<mobrovac@tin> |
Starting deploy [changeprop/deploy@77786a5]: Config change: respect the pages blacklisted by RESTBase |
[production] |
| 20:25 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove dumb comment (duration: 00m 39s) |
[production] |
| 20:21 |
<demon@tin> |
Synchronized scap/plugins: no-op scap stuff (duration: 00m 39s) |
[production] |
| 20:21 |
<demon@tin> |
Synchronized tox.ini: no-op scap stuff (duration: 00m 39s) |
[production] |
| 20:20 |
<demon@tin> |
Synchronized test-requirements.txt: no-op scap stuff (duration: 00m 39s) |
[production] |
| 20:19 |
<demon@tin> |
Synchronized setup.py: no-op scap stuff (duration: 00m 39s) |
[production] |
| 20:19 |
<demon@tin> |
Synchronized requirements.txt: no-op scap stuff (duration: 00m 39s) |
[production] |
| 20:13 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: No-op comment-only (duration: 00m 52s) |
[production] |
| 17:50 |
<volans> |
rcs1002.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1 |
[production] |
| 17:41 |
<ema> |
rcs1001.eqiad.wmnet: python-gevent downgraded from 1.1b6-1~trusty1 to 1.0-1ubuntu1 |
[production] |
| 17:24 |
<reedy@tin> |
Synchronized wmf-config/throttle.php: T153741 (duration: 00m 40s) |
[production] |
| 16:31 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=restbase,dc=eqiad,name=restbase101[6-8].* |
[production] |
| 16:30 |
<bblack> |
upgrading tons of outdated OS packages on rcs100x |
[production] |
| 16:29 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=restbase,dc=codfw,name=restbase201.* |
[production] |
| 16:26 |
<marostegui> |
Running optimize table on db1045 for the revision tables as we urgently need some space back on that host - T153739 |
[production] |
| 16:19 |
<mutante> |
notebook1001/1002 - upgraded php packages |
[production] |
| 16:17 |
<jynus> |
drop repl_records_partition events from db1069 |
[production] |
| 16:09 |
<mobrovac> |
change-prop restarting everywhere |
[production] |
| 15:59 |
<mobrovac> |
restarting change-prop on scb1001 |
[production] |
| 15:48 |
<mobrovac@tin> |
Finished deploy [parsoid/deploy@e8f0865]: Deploy of mwApiServer across the wtp fleet (duration: 05m 59s) |
[production] |
| 15:42 |
<mobrovac@tin> |
Starting deploy [parsoid/deploy@e8f0865]: Deploy of mwApiServer across the wtp fleet |
[production] |
| 15:32 |
<mobrovac@tin> |
Finished deploy [parsoid/deploy@e8f0865]: Canary deploy of mwApiServer to wtp[12]001, take two (duration: 00m 37s) |
[production] |
| 15:31 |
<mobrovac@tin> |
Starting deploy [parsoid/deploy@e8f0865]: Canary deploy of mwApiServer to wtp[12]001, take two |
[production] |
| 14:33 |
<mobrovac@tin> |
Finished deploy [parsoid/deploy@1d75b14]: Canary deploy of mwApiServer to wtp[12]001 (duration: 00m 32s) |
[production] |