|
2017-04-18
§
|
| 10:28 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_reduce_ttl(codfw, eqiad) Reduce the TTL of all the MediaWiki discovery records |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) END TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Successfully completed |
[production] |
| 10:26 |
<switchdc> |
(volans@sarin) START TASK - switchdc.stages.t00_disable_puppet(codfw, eqiad) Disabling puppet on selected hosts |
[production] |
| 10:25 |
<volans> |
Final test of switchdc steps in the codfw->eqiad configuration, only idempotent changes, T160178 |
[production] |
| 10:25 |
<moritzm> |
installing wireshark security updates |
[production] |
| 10:20 |
<moritzm> |
uploaded HHVM 3.18.2+wmf2 for jessie-wikimedia/experimental (includes fix for T162354) |
[production] |
| 09:52 |
<oblivian:> |
Setting zotero in codfw UP |
[production] |
| 09:50 |
<_joe_> |
testing switchover script for services, will act on zotero in codfw |
[production] |
| 09:45 |
<_joe_> |
adding 60G to the ocg output partition on ocg1003 |
[production] |
| 09:17 |
<oblivian@neodymium> |
conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw |
[production] |
| 09:03 |
<volans> |
upgrading conftool to v0.4.1 on neodymium/sarin |
[production] |
| 07:48 |
<_joe_> |
uploaded python-conftool 0.4.1 to jessie-wikimedia |
[production] |
| 07:42 |
<_joe_> |
cleaning up orphaned COW images in /var/cache/pbuilder/build/ on copper |
[production] |
| 06:16 |
<marostegui> |
For the record: restarted s7 instance on db1069 - T163183 |
[production] |
| 00:36 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/MobileFrontend/resources/mobile.mainMenu/mainmenu.less: T163059 (duration: 03m 07s) |
[production] |
|
2017-04-17
§
|
| 23:37 |
<mutante> |
runnin rmmod acpi_pad on the 16 R320 via cumin, since blacklisting in puppet does not actively remove, confirmed unloaded. (16/16) success ratio (>= 100.0% threshold) for command: 'lsmod|grep -c acpi_pad ||:' (T162850) |
[production] |
| 23:33 |
<mutante> |
running puppet via cumin on all 16 Dell PowerEdge R320, adding blacklist file for acpi_pad kernel module. 15/16 success, all but tin (T162850) |
[production] |
| 22:46 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 03m 01s) |
[production] |
| 22:42 |
<mutante> |
tin - load average going down, acpi_pad processes gone, cpu usage low again (T163158) |
[production] |
| 22:40 |
<mutante> |
tin - rmmod acpi_pad (T163158) |
[production] |
| 22:08 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 16m 23s) |
[production] |
| 19:16 |
<mutante> |
tegmen test ircecho stop/start service to confirm it's fine on jessie/prod icinga role (that's the passive server) |
[production] |
| 19:02 |
<demon@tin> |
Synchronized wmf-config/: Pruning some old extension message files, co-master sync (duration: 01m 52s) |
[production] |
| 18:58 |
<demon@tin> |
Pruned MediaWiki: 1.29.0-wmf.15 (duration: 00m 14s) |
[production] |
| 18:46 |
<maxsem@tin> |
Finished deploy [tilerator/deploy@001811e]: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only (duration: 00m 19s) |
[production] |
| 18:46 |
<maxsem@tin> |
Started deploy [tilerator/deploy@001811e]: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only |
[production] |
| 18:45 |
<maxsem@tin> |
scap aborted: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only (duration: 00m 19s) |
[production] |
| 18:45 |
<maxsem@tin> |
Started scap: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only |
[production] |
| 15:48 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: Update client caching headers for T161284 (duration: 08m 15s) |
[production] |
| 15:40 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: Update client caching headers for T161284 |
[production] |
| 15:34 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: (no justification provided) (duration: 01m 29s) |
[production] |
| 15:33 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: (no justification provided) |
[production] |
| 15:32 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: (no justification provided) (duration: 01m 42s) |
[production] |
| 15:31 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: (no justification provided) |
[production] |
| 09:33 |
<marostegui> |
Silence alerts for restbase2004 and restbase2009 T160759 |
[production] |
|
2017-04-14
§
|
| 23:14 |
<jynus> |
skipping CREATE DATABASE wbwikimedia on dbstore2001- duplicate declaration due to multi-source |
[production] |
| 22:58 |
<jynus> |
skipping CREATE DATABASE pawikisource on dbstore2001- duplicate declaration due to multi-source |
[production] |
| 22:49 |
<volans> |
restarting parsoid to get the disable linter change T148609 |
[production] |
| 22:17 |
<Reedy> |
created linter tables on wbwikimedia T148609 |
[production] |
| 22:16 |
<Reedy> |
created linter tables on pawikisource T148609 |
[production] |
| 20:53 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable Linter on larger wikis T148609 (duration: 00m 41s) |
[production] |
| 20:26 |
<reedy@tin> |
Synchronized wmf-config/abusefilter.php: abusefilter-modify-restricted for trwiki T161960 (duration: 01m 38s) |
[production] |
| 17:48 |
<mutante> |
mw1297 - restarted hhvm and apache |
[production] |
| 17:07 |
<twentyafterfour> |
deployed phabricator hotfix for T162943 |
[production] |
| 10:29 |
<elukey> |
rollback systctl settings on mw1306 after experiment (stop jobchron/runner, stop hhvm, restore systctl settings, restart hhvm and job* daemons) |
[production] |
| 09:50 |
<elukey> |
temporarily set sysctl -w net.netfilter.nf_conntrack_max=524288 on mw1306 (jobrunner) as test - (rollback: sysctl -w net.netfilter.nf_conntrack_max=262144") |
[production] |
| 09:43 |
<elukey> |
temporarily set sysctl -w net.ipv4.ip_local_port_range="15000 64000" on mw1306 (jobrunner) as test - (rollback: sysctl -w net.ipv4.ip_local_port_range="32768 60999") - T157968 |
[production] |
| 08:32 |
<elukey> |
restored appendfsync to 'everysec' on Redis rdb2005:6380 (end of performance experiment) |
[production] |