|
2018-02-21
§
|
| 16:59 |
<arturo> |
puppet is broken across the cluster due to last change |
[tools] |
| 16:57 |
<arturo> |
deploying https://gerrit.wikimedia.org/r/#/c/410177/ |
[tools] |
| 16:26 |
<bd808> |
Rebooting tools-docker-registry-01, NFS mounts are in a bad state |
[tools] |
| 11:43 |
<arturo> |
package upgrades in tools-webgrid-lightttpd-1401 |
[tools] |
| 11:35 |
<arturo> |
package upgrades in tools-package-builder-01 tools-prometheus-01 tools-static-10 and tools-redis-1001 |
[tools] |
| 11:22 |
<arturo> |
package upgrades in tools-mail, tools-grid-master, tool-logs-02 |
[tools] |
| 10:51 |
<arturo> |
package upgrades in tools-checker-01 tools-clushmaster-01 and tools-docker-builder-05 |
[tools] |
| 09:18 |
<chicocvenancio> |
killed io intensive tool job in bastion |
[tools] |
| 03:32 |
<zhuyifei1999_> |
removed /data/project/.elasticsearch.ini, owned by root and mode 644, leaks the creds of /data/project/strephit/.elasticsearch.ini Might need to cycle it as well... |
[tools] |
|
2018-02-16
§
|
| 18:21 |
<arturo> |
upgrading tools-proxy-01 and tools-paws-master-01, same as others |
[tools] |
| 17:36 |
<arturo> |
upgrading oldstable, jessie-backports, jessie-wikimedia packages in tools-k8s-master-01 (excluding linux*, libpam*, nslcd) |
[tools] |
| 13:00 |
<arturo> |
upgrades in tools-exec-14[01-10].eqiad.wmflabs were fine |
[tools] |
| 12:42 |
<arturo> |
aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'DEBIAN_FRONTEND=noninteractive sudo apt-upgrade -u upgrade trusty-updates -y' |
[tools] |
| 11:58 |
<arturo> |
aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-wikimedia -y |
[tools] |
| 11:57 |
<arturo> |
aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-backports -y |
[tools] |
| 11:53 |
<arturo> |
(10 exec canary nodes) aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'sudo apt-upgrade -u upgrade trusty-wikimedia -y' |
[tools] |
| 11:41 |
<arturo> |
aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade oldstable -y |
[tools] |
|
2018-02-09
§
|
| 10:35 |
<arturo> |
deploy https://gerrit.wikimedia.org/r/#/c/409226/ T179343 T182562 T186846 |
[tools] |
| 06:15 |
<bd808> |
Killed orphan processes owned by iabot, dupdet, and wsexport scattered across the webgrid nodes |
[tools] |
| 05:07 |
<bd808> |
Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1426 |
[tools] |
| 05:06 |
<bd808> |
Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1411 |
[tools] |
| 05:05 |
<bd808> |
Killed 1 orphan php-fcgi process from jembot that were running on tools-webgrid-lighttpd-1409 |
[tools] |
| 05:02 |
<bd808> |
Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1421 and pegging the cpu there |
[tools] |
| 04:56 |
<bd808> |
Rescheduled 30 of the 60 tools running on tools-webgrid-lighttpd-1421 (T186830) |
[tools] |
| 04:39 |
<bd808> |
Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1417 and pegging the cpu there |
[tools] |
|
2018-02-08
§
|
| 18:38 |
<arturo> |
aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1002.tools.eqiad.wmflabs |
[tools] |
| 18:35 |
<arturo> |
aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade jessie-wikimedia -v |
[tools] |
| 18:33 |
<arturo> |
aborrero@tools-worker-1002:~$ sudo apt-upgrade -u upgrade oldstable -v |
[tools] |
| 18:28 |
<arturo> |
cordon & drain tools-worker-1002.tools.eqiad.wmflabs |
[tools] |
| 18:10 |
<arturo> |
uncordon tools-paws-worker-1019. Package upgrades were OK. |
[tools] |
| 18:08 |
<arturo> |
aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stable -v |
[tools] |
| 18:06 |
<arturo> |
aborrero@tools-paws-worker-1019:~$ sudo apt-upgrade upgrade stretch-wikimedia -v |
[tools] |
| 18:02 |
<arturo> |
cordon tools-paws-worker-1019 to do some package upgrades |
[tools] |