|
2017-09-29
§
|
| 11:53 |
<bblack> |
cp4021: reboot for numa mode switch |
[production] |
| 11:44 |
<elukey> |
re-enable job runner daemons on mw130[8,9] |
[production] |
| 11:15 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2053 db2046 - T174509 (duration: 00m 48s) |
[production] |
| 11:05 |
<elukey> |
precautionary stop of jobrunner/jobchron on the new mw130[8,9] - job queue size rapid increase investigation |
[production] |
| 09:55 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw1309.eqiad.wmnet |
[production] |
| 09:33 |
<ema> |
upgrade pybal to 1.14.0 on codfw primaries |
[production] |
| 08:20 |
<ema> |
upgrade pybal to 1.14.0 on codfw secondaries |
[production] |
| 05:53 |
<ema> |
cp4024 repooled T174891 |
[production] |
| 05:20 |
<ema> |
powerup cp4024 T174891 |
[production] |
| 00:26 |
<ejegg> |
updated fundraising tools from 69a96276fa8cd8f68aa1225ea48fff1a7c833101 to 6d4b6f388e64ff4dc9a6a501bc8339fc1294fe9f |
[production] |
| 00:16 |
<ejegg> |
updated CiviCRM 213886998cad62a37aab5b214290b8431ab41199 to 784756b4157517bf9639a67b8d0ddcb1db7c999d |
[production] |
| 00:07 |
<mutante> |
releases1001 - starting Jenkins setup wizard with generated admin pass |
[production] |
| 00:05 |
<mutante> |
releases1001 - stopped puppet, manually fixing --prefix=/ci setting for jenkins process, killing it, removing init.d file, starting with systemd, jenkins now up T164030 |
[production] |
|
2017-09-28
§
|
| 23:46 |
<awight@tin> |
Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 20s) |
[production] |
| 23:45 |
<awight@tin> |
Started deploy [ores/deploy@42c5663]: Cause ORES service restart |
[production] |
| 22:18 |
<MaxSem> |
Deployed fix for T176247 |
[production] |
| 21:30 |
<ejegg> |
updated CiviCRM from 3f48cb03b4a216a34a61f9744a297de4fb99d8a6 to 213886998cad62a37aab5b214290b8431ab41199 |
[production] |
| 21:23 |
<XenoRyet> |
Updated SmashPig from dc6220391036c630c8ad14c0d894d33c1e8d8c2c to d056a131ccbbd4b505da4f99e04f5337fdd9c32e |
[production] |
| 19:36 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1 |
[production] |
| 19:34 |
<legoktm> |
mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798) |
[production] |
| 19:31 |
<bblack> |
cp4026 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 19:28 |
<legoktm> |
mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua"; |
[production] |
| 19:14 |
<ejegg> |
updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export |
[production] |
| 19:14 |
<robh> |
cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891 |
[production] |
| 19:00 |
<bblack> |
cp4025 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 18:54 |
<thcipriani@tin> |
Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: [[gerrit:381259|Revision::newFromTitle may return null]] T176882 (duration: 00m 50s) |
[production] |
| 18:42 |
<bblack> |
cp4023 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 18:37 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:381251|Create rollbacker group at viwiktionary]] T176979 (duration: 00m 50s) |
[production] |
| 18:26 |
<thcipriani@tin> |
Synchronized docroot/noc: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] PART II T104148 (duration: 00m 49s) |
[production] |
| 18:25 |
<thcipriani@tin> |
Synchronized wmf-config: SWAT: [[gerrit:376337|copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging]] T104148 (duration: 00m 51s) |
[production] |
| 17:57 |
<arlolra> |
Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832) |
[production] |
| 17:52 |
<mutante> |
librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera |
[production] |
| 17:46 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s) |
[production] |
| 17:43 |
<bblack> |
cp4028 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 17:40 |
<moritzm> |
installing apache updates on cobalt |
[production] |
| 17:32 |
<bblack> |
cp4022 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 17:30 |
<arlolra@tin> |
Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c |
[production] |
| 17:25 |
<marostegui> |
Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985 |
[production] |
| 17:20 |
<bblack> |
cp4027 - depool->reboot for bios updates, isolcpus fix |
[production] |
| 17:08 |
<bblack> |
cp4021 - depool->reboot for bios update |
[production] |
| 17:05 |
<robh> |
bios firmware uploaded and awaiting reboot for application on cp402[1235678] |
[production] |
| 16:48 |
<bd808> |
Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978) |
[production] |
| 16:26 |
<robh> |
cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891 |
[production] |
| 16:09 |
<robh> |
cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish |
[production] |
| 16:01 |
<marostegui> |
Global rename of CodeCat → Rua - T176985 |
[production] |
| 15:22 |
<hoo> |
Ran scap pull on mwdebug1001 after experiments with T176844 |
[production] |
| 14:54 |
<dcausse> |
restarting elastic on relforge servers to reinstall prod ltr plugin |
[production] |
| 14:51 |
<marostegui> |
Stop MySQL on db1035 as server is going to be decommissioned - T176931 |
[production] |
| 14:45 |
<moritzm> |
installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily) |
[production] |
| 14:42 |
<marostegui> |
Remove db1035 from tendril - T176931 |
[production] |