|
2017-04-17
§
|
| 23:37 |
<mutante> |
runnin rmmod acpi_pad on the 16 R320 via cumin, since blacklisting in puppet does not actively remove, confirmed unloaded. (16/16) success ratio (>= 100.0% threshold) for command: 'lsmod|grep -c acpi_pad ||:' (T162850) |
[production] |
| 23:33 |
<mutante> |
running puppet via cumin on all 16 Dell PowerEdge R320, adding blacklist file for acpi_pad kernel module. 15/16 success, all but tin (T162850) |
[production] |
| 22:46 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 03m 01s) |
[production] |
| 22:42 |
<mutante> |
tin - load average going down, acpi_pad processes gone, cpu usage low again (T163158) |
[production] |
| 22:40 |
<mutante> |
tin - rmmod acpi_pad (T163158) |
[production] |
| 22:08 |
<catrope@tin> |
Synchronized php-1.29.0-wmf.20/extensions/WikimediaEvents/modules/ext.wikimediaEvents.recentChangesClicks.js: T158458 T163152 (duration: 16m 23s) |
[production] |
| 20:41 |
<bd808> |
Building tools-docker-builder-05 |
[tools] |
| 19:35 |
<chasemp> |
add reedy to sudo all perms so he can admin things |
[tools] |
| 19:16 |
<mutante> |
tegmen test ircecho stop/start service to confirm it's fine on jessie/prod icinga role (that's the passive server) |
[production] |
| 19:02 |
<demon@tin> |
Synchronized wmf-config/: Pruning some old extension message files, co-master sync (duration: 01m 52s) |
[production] |
| 18:58 |
<demon@tin> |
Pruned MediaWiki: 1.29.0-wmf.15 (duration: 00m 14s) |
[production] |
| 18:46 |
<maxsem@tin> |
Finished deploy [tilerator/deploy@001811e]: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only (duration: 00m 19s) |
[production] |
| 18:46 |
<maxsem@tin> |
Started deploy [tilerator/deploy@001811e]: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only |
[production] |
| 18:45 |
<maxsem@tin> |
scap aborted: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only (duration: 00m 19s) |
[production] |
| 18:45 |
<maxsem@tin> |
Started scap: https://gerrit.wikimedia.org/r/#/c/348224/ to test hosts only |
[production] |
| 17:21 |
<andrewbogott> |
adding 8 more exec nodes: tools-exec-1435 through 1442 |
[tools] |
| 15:48 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: Update client caching headers for T161284 (duration: 08m 15s) |
[production] |
| 15:40 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: Update client caching headers for T161284 |
[production] |
| 15:34 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: (no justification provided) (duration: 01m 29s) |
[production] |
| 15:33 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: (no justification provided) |
[production] |
| 15:32 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@6595298]: (no justification provided) (duration: 01m 42s) |
[production] |
| 15:31 |
<mobrovac@tin> |
Started deploy [restbase/deploy@6595298]: (no justification provided) |
[production] |
| 09:33 |
<marostegui> |
Silence alerts for restbase2004 and restbase2009 T160759 |
[production] |
|
2017-04-14
§
|
| 23:14 |
<jynus> |
skipping CREATE DATABASE wbwikimedia on dbstore2001- duplicate declaration due to multi-source |
[production] |
| 22:58 |
<jynus> |
skipping CREATE DATABASE pawikisource on dbstore2001- duplicate declaration due to multi-source |
[production] |
| 22:49 |
<volans> |
restarting parsoid to get the disable linter change T148609 |
[production] |
| 22:17 |
<Reedy> |
created linter tables on wbwikimedia T148609 |
[production] |
| 22:16 |
<Reedy> |
created linter tables on pawikisource T148609 |
[production] |
| 20:53 |
<reedy@tin> |
Synchronized wmf-config/InitialiseSettings.php: Disable Linter on larger wikis T148609 (duration: 00m 41s) |
[production] |
| 20:26 |
<reedy@tin> |
Synchronized wmf-config/abusefilter.php: abusefilter-modify-restricted for trwiki T161960 (duration: 01m 38s) |
[production] |
| 17:48 |
<mutante> |
mw1297 - restarted hhvm and apache |
[production] |
| 17:44 |
<halfak> |
deployed wikilabels-wmflabs-deploy:01bf4bc |
[wikilabels] |
| 17:43 |
<halfak> |
staged wikilabels-wmflabs-deploy:01bf4bc |
[wikilabels] |
| 17:37 |
<madhuvishy> |
Also killed 3912152 - was stuck in Ds on tools-exec-1423 - tools.alaabot 29083 2.0 0.0 5376 1400 ? Ds 14:59:03 00:03:14 yes |
[tools.alaabot] |
| 17:27 |
<madhuvishy> |
Killed job 3912666 cron-tools - seemed to be stuck in some endless wait loop on /usr/bin/yes causing IOWait to spike on tools-exec-1415 |
[tools.alaabot] |
| 17:07 |
<twentyafterfour> |
deployed phabricator hotfix for T162943 |
[production] |
| 12:29 |
<hashar> |
Delete integration-c1 instance (32GB RAM) on labvirt1004. It was used as a workaround for T161006 |
[releng] |
| 10:29 |
<elukey> |
rollback systctl settings on mw1306 after experiment (stop jobchron/runner, stop hhvm, restore systctl settings, restart hhvm and job* daemons) |
[production] |
| 09:50 |
<elukey> |
temporarily set sysctl -w net.netfilter.nf_conntrack_max=524288 on mw1306 (jobrunner) as test - (rollback: sysctl -w net.netfilter.nf_conntrack_max=262144") |
[production] |
| 09:43 |
<elukey> |
temporarily set sysctl -w net.ipv4.ip_local_port_range="15000 64000" on mw1306 (jobrunner) as test - (rollback: sysctl -w net.ipv4.ip_local_port_range="32768 60999") - T157968 |
[production] |
| 08:32 |
<elukey> |
restored appendfsync to 'everysec' on Redis rdb2005:6380 (end of performance experiment) |
[production] |
| 08:17 |
<hashar> |
beta: cherry picking again 348184/4 'service: use gzip for logging in uwsgi' for T161563 |
[releng] |
| 08:03 |
<hashar> |
beta: resetting puppetmaster to last good tag snapshot-20170414T0030 A cherry pick for T161563 end up dropping three patches which broke other parts of the infrastructure |
[releng] |
| 07:52 |
<hashar_> |
Puppet failing on deployment-tin and deployment-mira . Some patches have been dropped from the puppet master :-(( |
[releng] |